What you are describing is similar to my workflow. I rate every image 1-5 rather than just my selects. For me, my goal is to discard the 1 and 2 rated images, keep the 3's, keep the 4's and use that group to identify selects for editing, and keep the 5's with the same approach. I use the color labels to identify selects, edited selects or other similar decisions.
By setting the filter to 0 I only see unrated images. I can make it through a large group very quickly. The filter also allows me to take a second look at 1 and 2 rated images before discarding them in order to make sure there were no mistakes. Likewise it allows me to look at just 4 and 5 rated images to identify selects for editing and the status of editing.
As you describe, rendering is very fast in View. And it quickly zooms to 100% with a click.
I also use View for quick edits related to event photos.