Yes, you would have to wait for the buffer to empty before switching off if you don't want to lose anything.
And yes, it is annoying. I'm not sure yet how often it is going to happen under normal circumstances - the suggestion I have heard is that it tends only to happen in low light/contrast situations. As I've only just got mine, and the weather has been appalling where I am for the last few days I've only really been trying the lens out indoors. My house is quite dimly lit, so it may be that outside in good daylight it will rarely happen. I'll have to wait and see.