Putting the D800 in DX mode using FX lenses doesn't give you anything except a smaller viewfinder and a smaller file size. You are not extending reach but cropping in camera rather than in software.
I don't know what the problem is with the Sigma 50-500mm. It sounds like the stabilization is chattering. I've never used a Sigma telephoto. The only time my Nikon lenses chatter is when I try a 2X extender on an f5.6 lens with VR.