I also held this view for a long time but what you are talking about is basically Amazon EC2. There are, what, 200-250 AWS services, however, and that's where things begin to become more interesting. Can you replace any of them with in house solutions? Certainly. But the costs of doing so might not be favorable.

You could operate an on premise bakery but most companies just order donuts.