I can't speak from specific experience with the D4, but I recently purchased a D3s factory refurb. It arrived packed in a Nikon box labeled as "refurbished". I'd give the cosmetic condition a rating of 9 out of 10. My only disappointment was the shutter count of 121,974. That's a little more than 1/3 the stated 300k lifetime. I expected a much lower count than it has, but very happy overall with the camera. It could almost pass for new. You'd never guess it has over 100k actuations just by looking at it.
Perhaps the lesson here is about expectations. Don't expect a refurb to have a low shutter count and don't expect the retailer to know the count, if asked. It never hurts to ask but when I did, I was told they don't know the shutter counts on refurbs from Nikon.
Whatever path you decide to take, best wishes in acquiring an awesome camera!