I don't think the pixel density really enters into it much. The issue is the amount of motion of the object as a percentage of the size of the final output image (print). Whether the image was captured on a 16-MP camera or a 12-MP camera, if the object (nail, player) moves far enough while the shutter is open to make a discernible blur on the image, the camera will capture and reproduce that blur. At some really low pixel density the motion blur would get lost in the sensor resolution, but that would be at much lower resolutions than any modern DSLR provides.
If your 3.5 MPH speed calculation is correct, that means the nail in your experiment moves about a tenth of an inch in 1/500 second. (Which looks about right from viewing your images.) Your frame is about 30 inches wide at the subject distance, so that 0.1-inch motion represents about 0.33% of the width of the frame.
If the width of the print is 16 inches, that 0.33% means the size of the blur will be about 1/20 inch. Small, but not invisible. Enough to take away a bit of crispness.
As for sensor resolution, 0.33% of the width of a D700/D3 12-MP sensor is over 9 pixels on the short dimension of the frame (i.e., portrait orientation), so those sensors are easily capable of resolving that motion. (I assume you would be using a lens that gives the same FOV on FX.) 0.33% is over 6 pixels even on a D70 and over 5 pixels on a D2H, still quite enough to be resolved. So your point that 1/500 is marginal for motion-stop basketball shooting with large prints remains true regardless of which body is in use.