Unless you are going to take action photos in dim light, large apertures and/or high ISOs are not required. The only other reasons to prefer a larger aperture would be thinner DOF and a brighter viewfinder (up to a point; usually for modern SLRs, anything wider than f/2 makes no difference in viewfinder brightness).
Case in point: Yesterday, I was shooting a hockey game at f/2.8 with the 80-200 lens. I was at ISO 2000 (Hi-0.3) on the D200. I was getting shutter speeds between 1/250th and 1/640th second in aperture priority depending on what part of the rink I was shooting. You really need 1/400th second or faster for hockey (even with a shorter lens). Some of the 1/250th and 1/320th sec frames (even those shot with the 16-50mm lens if I didn't pan) have some motion blur, but I didn't want to push my ISO any higher, so I accepted this as a compromise. If I had an f/2.0 zoom, I would have gotten no motion blurred frames, because I'd have gained a stop of shutter speed. But there are no f/2.0 zooms for FX or DX bodies (though Olympus has several for 4/3 format cameras).
f/4 is NOT a slow lens for a zoom. It's fairly fast. Considering that the very fastest zooms for a 35mm camera are essentially f/2.8, and many zooms are actually f/5.6 or even f/6.3...f/6.3...now, that is slow. f/4.0 zooms are still considered relatively fast/professional spec lenses, to an extent.