You've got two sets of questions here - part about the D800 and images, and part about post processing software. If we can't quickly resolve the software questions here, it's probably better to move that discussion to the Nikon Software forum.
I'm running Capture NX2 and View on a Windows 7 i5 64 bit system with 8 GB of RAM. You should be fine from a system standpoint.
First things first. Can you use View NX2 to view the images? That's the most neutral viewer and uses the camera settings.
The D800 images should be excellent. Are you shooting NEF + JPEG or just NEF? It is possible to view the JPEG file and a basic JPEG would appear to have low resolution at 100%.
Images should be very sharp. The 70-200 f/2.8 performs very well at f/5 but you can see some softness wide open. Still, I regularly shoot it at f/3.2 or so. Be sure VR is off and shutter speeds are fast enough for a good test. Use the center sensor for AF. Try selecting a subject that is 15-20 feet away. A tripod is a good test, but I have no problems handheld if shutter speeds are 1/500 sec or higher.
Check the Picture Control settings and set it for Standard. Standard is a relatively neutral starting point and should produce good images without undue sharpening or contrast which can cause pixelation and artifacts. Set ISO on a reasonable value - ISO 200 or so for test purposes. Turn off D-Lighting and Active D-Lighting for now while you are testing - both can introduce noise or pixelation.
Nikonians®, NikoScope® and NikoniansAcademy™ are trademarks owned by Nikonians.org.
Nikon®, Nikonos® and Nikkor® are registered trademarks of Nikon Corporation.