Good thought. Carried to the extreme in a common example, the IR spectrum viewed by itself is significantly off the focus of the visible spectrum.
There are lens designs with variations in the extent of spectrum related focus shift. Normally IR is about 10% off the visible spectrum and UV varies in the opposite direction, but with the Coastal Optics 60 f/4, UV, visible, and IR spectrum converge. That is a function of lens design. The Coastal Optics 60 f/4 is a manual focus lens, but it is optically superb. And should be at the bargain price of $4650.