Yes, the lens may have moved a bit as I removed the filter, but that's not the point. Look at where the image is most sharp in both cases: still the one without the filter is sharper. Back up a bit and look at the brown area as a whole. Which image do you prefer?
I may run some more tests (I have others already)... and looking at all of them (needless to post here) I see a difference.