Something similar happened to me, too, a few years ago. I compared shots with and without filter, and since then I have not used a protective filter again. The filter quality matters as well - with a high quality filter, the difference is quite small but still visible. And the light matters - any kind of light source hitting the filter can quite degrade the image, especially when using cheap uncoated filters. Finally there is the economic factor. When one owns a number of lenses with different filter threads, buying a set of high quality UV or clear filters can get very expensive. I figure that having a scatched front element replaced, in the unlikely event that this should ever occur, will actually cost less than all those filters.