Since your asking about these two lenses and according to your profile your shooting FX, I think the question is really about aperture and bokeh. Your style will drive this. You have a 50 f 1.4 so you may gain some insight looking at your shots using that lens. Check the EXIF data to see what apertures you shot your people with, and how often you shot it wide open. It's really about controlling the background. My own preference is for the 85 f1.4, on full frame (which I don't have anymore).
I use the 105VR and am happy with it. If I were shooting for pay, I'd have the 85 and shoot it wide open with an FX body. It is a wonderful lens, with great attributes for portraits. I do agree with Brian that the 105 let's you get back a bit further and delivers a bit flatter perspective. Since I shoot DX, for me it's a little like shooting the old 135mm, a perspective I like. YMMV.