A fully retracted center column probably doesn't seriously impact stability. It will likely marginally effect things but that would need to be tested on a model by model basis.
Raising it does, and progressively higher center columns are progressively more unstable. Center columns also interfere with getting low to the ground (some can be removed but that takes time and effort).
For ultimate stability it is better not to have a center column at all, and lunar imaging puts unusual stress on a tripod because of the high elevations. I don't use them myself, but when I'm doing macro sometimes I would like one, just for tweaking the height an inch or so.
I think the main problem with center columns is when people buy them to make up for a too short base tripod, forcing them to raise it significantly whenever they need an eye level shot.
Although center columns add to the cost, it is unfortunate that most or all budget tripods come with one, eliminating potential choices in the matter.