I had the same realisation a few years ago. I switched from CS to math and physics. It is definitely more challenging but I do not regret the switch. Plus I discovered math can be quite fun.

Most of the most challenging software requires domain knowledge that CS/software devs do not have. Ultimately it is easier to go from a math/physics background to CS/software development than the other way round.