I hate to be the one to dash your plans, but I don't think that switching to the OM-D is going to add anything to your experience in terms of size, weight or quality, but it will definitely help deplete your bank account for no good reason as far as I can see.
Here are the specs on the OM-D:
4.8" x 3.5" x 1.7" Weighs 15oz. - 16.1mp 4/3 sensor
Price with a 12-50 lens $1,300.00.
Here are the specs on the D3100 (and probably the new D3200):
4.9" x 3.8" x 3.0" Weighs 16oz. - 14.2 mp DX sensor (probably more on the D3200)
Price with an 18-55 lens $550.00.
The 18-55 is a spectacular lens for the price - I don't know much about the IQ specs of the Oly lenses, but I would be surprised if they were even close.
So, for the extra inch in depth, I think that you are making a big mistake. On the other hand, you don't have a huge investment in Nikkor lenses, and I don't think that any of them will autofocus on the D3100/3200, so maybe it will work out for you.