Your specifications seem somewhat contradictory. The 80-400 is not sufficient - but I'm not sure what failings are the ones that are an issue. Image quality? Focusing speed? Focal length?
You are needing something up to the demands of a D800 sensor, yet you also are on something of a budget. What is that budget?
And you're birding. And you want a fast lens.
These things are fundamentally incompatible, so you'll have to relent on at least some of these things.
First, it seems that one of the problems is lack of focal length, since you've just switched from DX to FX. You had 600mm equivalent, now you have 400mm. Just matching that requires a relatively extreme solution, such as a 300/f4 + TC-20eIII. This solution hardly inexpensive, probably about $2000, but it's still an f/8 combination and that is definitely not what I'd call fast. (I have plenty of experience with f/8 lenses...) Pretty much by definition I think you're looking for a solution that's 500mm or longer and really you are needing to think about 600mm or longer, at which length the solutions are sparse and generally expensive. I don't think that the TC-17 will deliver better results than the TC-20eIII, and both of them are slow enough that I couldn't call that the deciding factor. All of the less-than-megabuck solutions are going to be slow.
The Bigma, otherwise known as the Sigma 50-500/f6.3 HSM, is a potential solution. But it's not that much longer than your 80-400. You apparently have one, and if so it may be an older model with no OS (Sigma's equivalent of VR). It's a good lens, probably a bit better than the 80-400, but I wouldn't call it significant better. It focuses more quickly for sure. The later versions of the 50-500 have OS, and that's more effective than the first-generation VR in the 80-400. A new Bigma runs about $1600. Relatively speaking it's light compared to most other solutions in this space, but it's heavier than the 80-400.
Sigma also offer the 150-500/f6.3 HSM OS, and it is only about $1000. It has most of the attributes of the 50-500 without some of the zoom range and a lot of the price. Personally I think this one is your best bet, but it's still only 500mm with not much prospect of going longer. At f/6.3, it is not a fast lens. But it is a $1000 solution that's a lot closer to the $8000 solutions than one would guess from the price differential, up to and including folks using them on D800 and D800e.
The Sigma 500/f4.5 is a very fine lens, but we're up to $5000 now. Reasonably fast and with excellent optics, it's the lightest and cheapest (!) of the real big guns. It's been argued in the past as to whether or not this one matches the Nikkor 500/f4 AFS, but even if it doesn't it is a top caliber optic. Still, it's less money than the Nikkor 300/f2.8 AFS VR-II and nearly twice the focal length. With a Sigma 1.4x TC this one could get you to 700mm and would actually be longer than what you have been used to. I have a friend who is pretty much a full time birder and this one is his go to lens.
I suppose that another potential solution is the Sigma 120-300/f2.8 HSM OS + Sigma 2x TC, but probably that should wait a bit until the newest model 120-300 hits the streets in a month or two. In the past, I have found this combination not to focus particularly reliably - it often failed to lock on and I had to go back and retry. Generally speaking I find that TCs cause this, but this was worse than usual, especially for a "mere" f/5.6 combination. The D800/D4 AF capability may be better, too, I don't know. The 120-300 is about $3300 and certainly by itself is a great lens, fast and accurate while maintaining zoom capability. It weighs more than twice what the Bigma does, though, at about 7lbs or thereabouts. I have a friend that is a sports pro and he uses this as one of his main tools.
Up to now I haven't mentioned Nikon stuff simply due to price - the 300/f2.8 starts at $5800, and the big guns are all near $10k if bought new. Personally I'm using a 400/f2.8 AFS (non VR) with a TC-20eIII, which if you can find one, would run about $6000 on the used market. A 600/f4 AFS (non-VR) would be in the same vicinity. Finding a Sigma 500/f4.5 on the used market is pretty unlikely although it does happen. People tend to hold onto them. I haven't seen one in a while but I'd guess $4000. The Nikkor 200-400/f4 AFS VR seems pretty short for birding to me, but it's a great lens. It's approaching $7k and you'd still need a TC in all likelihood.
_____ Brian... a bicoastal Nikonian and Team Member
My gallery is online. Comments and critique welcomed any time!