As usual, it depends on what you intend to do with the lens. The f/4 has VR III, which, according to Nikon should give you another stop of vibration reduction. My own informal tests tell me the VR in the new 70-200 is better than that in my VR II lenses, though I can't vouch for a one-stop improvement. That kind of claim is advertising. Too much depends on your own steadiness.
But to me the difference is in the weight. I can put the f/4 on my D800 or my D3, and carry it around without having my shoulder collapse, and I can put the combo on a tripod without having to bother with a tripod collar. Beyond that, you might want to check Nasim Mansurav's site. He's made some interesting comparisons between the two lenses, though he's still working on his full review. The only things I do that call for an aperture less than f/4 are street photography, which I normally do with a 50mm f/1.4, and an occasional portrait, which I do with an 85mm f/1.8. I often shoot plays during final dress rehearsal with a 70-200. With the D3 or D800 I simply don't need f/2.8 to do that.
But if you already have the f/2.8 and you don't carry it around, why bother with another lens.