Nothing you mentioned, spinning instances up and down, balancing between them, clustering databases, is exclusive to EC2, AWS or "cloud" computing in general.
Scalability is largely a design exercise, not, as much as AWS sales engineers want your CTO/CFO to believe, an infrastructure exercise. At the point where infrastructure becomes an issue, you're building your own AWS.
I'm happy to admit that AWS might make some of this easier, but it's almost certainly going to be more expensive [1], and it's often at the cost of flexibility (lock-in, AWS-specific knowledge).
How are dedicated servers, or even collocated servers, possibly less flexible?
[1] There are exceptions, S3 and Route53 stand out as, at the very least, being cost-competitive to a greater extent than other AWS/Cloud offerings.
I'm unaware of any dedicated provider that will let me take a $20/mo small instance and scale it within minutes to an insanely beef 8 core instance for a day for about $20 and then back down, with less than 15 mins total downtime. Total cost for a month less than $50.
One of the things I feel AWS has succeeded at is putting control of infrastructure in the technology team's hands. In many places dedi or colo requires contract negotiation with the provider and involves some sort of purchasing dept. There are some places I've worked with where getting a dedi/colo could be weeks or months of different teams paperwork. With AWS the tech team can spin up 100 servers with no outside involvement needed after the initial work of getting an aws account with $x-xxx k/mo limit.
It's hard to beat the operational flexibility AWS provides, but I can see a few scenarios where creating your own mini private cloud out of dedi/colo servers could be more cost effective.
> I'm unaware of any dedicated provider that will let me take a $20/mo small instance ...
You are looking in the wrong places. Dedicated server providers offer dedicated servers. What you want is a VPS. There are dozens of VPS providers with a multitude of products and billing by the minute or by the month and everything in between.
> In many places dedi or colo requires contract negotiation with the provider and involves some sort of purchasing dept. There are some places I've worked with where getting a dedi/colo could be weeks or months of different teams paperwork.
This is merely a failure on the part of your employer.
> It's hard to beat the operational flexibility AWS provides, but I can see a few scenarios where creating your own mini private cloud out of dedi/colo servers could be more cost effective.
It's more like the other way around. To refute my point, please give some examples where AWS is cheaper AT SCALE.
AWS might make some things easier or more convenient, but in no way does it come cheaper at scale as their PR flak claims.
Cloud computing was supposed to be synonymous with the concept of utility computing. Where you can plug into any provider with your app and it just works and you pay a markup on the kilowatt hour so to speak. Until there's a uniform cloud platform, we're not there yet. I suspect we're at least a generation away.
Scalability is largely a design exercise, not, as much as AWS sales engineers want your CTO/CFO to believe, an infrastructure exercise. At the point where infrastructure becomes an issue, you're building your own AWS.
I'm happy to admit that AWS might make some of this easier, but it's almost certainly going to be more expensive [1], and it's often at the cost of flexibility (lock-in, AWS-specific knowledge).
How are dedicated servers, or even collocated servers, possibly less flexible?
[1] There are exceptions, S3 and Route53 stand out as, at the very least, being cost-competitive to a greater extent than other AWS/Cloud offerings.