I agree, and the reason is housing. If housing in this country wasn't fucked up I think it would be possible. As of now, to do this as a janitor you would need to live with roommates or family. You would also need to avoid smoking, alcohol, gambling, and any other money sinks. Then you would need to invest all your savings for years.
Housing cost is the single defining factor in retirability now and if you're a janitor, you seriously need to consider where you are living, because moving from a VHCOL state to a lower one could be all you need.
Sure the pay is lower, but the housing costs are substantially lower.
Can a janitor save enough money to rival Gates? No. Not even close.
But the same janitor could save enough money to have "FU money" later in life, and only continue working as a janitor if he chose to do so.