My suggestion - leave Long Exposure Noise Reduction on. Other than the delay (as mentioned), there's no downside, all upside.
And contrary to a lot of notes elsewhere -- it DOES affect raw image data. High ISO Noise reduction does not affect raw image data.
The one oddball case where you would not is if you, yourself, are going to take a dark frame and subtract. You might do that (for example) if you were going to make 10 long exposures of the same duration in a row, and didn't want to wait between them. You would then make a single dark frame (lens covered) first, shoot the 10, and subtract in post processing. That's 11 exposures as opposed to the equivalent of 20 (one for each light) the camera would do. But this only works if the one frame is at the same settings, and same temperature, as the rest.
Dark frames are a normal part of astrophotography. They can be VERY effective in removing light haze from thermal noise and hot pixels. There are also bias and flat frames used there, which are not exactly used in DSLR (though a bias frame and the dust-off reference photo are not all that different).