Sometimes. But I don't understand why they couldn't partition to sidestep those orders of magnitude. Since you have actual knowledge of the system, could you give the 30-second version? It looks from the outside like they just didn't make nonblocking load balancing a priority, rather than that feature being harder than the rest of their work.
Same here, I don't understand. The problem is that at a given size they can't use their "intelligent routing" anymore, but afaik no customer of theirs has that size, so why don't they just make as many copies of "old Heroku" as needed? Then they can just use the DNS to select the right shard, and route within shards.
It's like: "Sorry, we're full, go see our competitor. Oh, and by the way, our competitor is us." You're never too large, problem solved.
I don't want to reveal specific (and perhaps confidential) information about Heroku's architecture, but I can tell you a few things about working in AWS in general.
One: routing traffic in AWS typically means ELB. ELB doesn't necessarily give you fine-grained control over per-application routing.
Two: Proxies don't always send a single HTTP request per TCP connection.
Three: In streaming-data startups, I think it's pretty common that a customer does something inefficient, scales rapidly over the course of a week or two, and then realizes too late that their initial design is no longer suitable for their traffic load. Sometimes a single customer makes up a shockingly large proportion of your traffic load--and this load is hard to predict.
Four: The routing infrastructure for any given app still requires multiple load balancers in multiple availability zones. That means shards are more expensive--in terms of nodes and cost.
Five: Big systems come with lots of moving parts. It might be impractical to create an isolated, fault-tolerant copy of two hundred services for each reasonably-sized customer. There are... all sorts of issues with distributing state.
Thank you for the insight. 3, 4 and 5 are probably the reasons why they don't do it. They could, but it would be harder to operate, a bit more expensive and a lot less elastic. And maybe some customers at some points in time are, in fact, too large for that strategy.
As for 1, they do use ELB now, I don't know if they did during the "intelligent routing" era. It is only possible to create up to 10 different ELBs with a standard AWS account, but something as large as Heroku does not have the same limits.
I don't really understand point 2, could you explain what you mean? I can see how keep-alive can be a problem with some configurations but if you use the DNS to select the shard it shouldn't be a problem (you have one balancer per shard, with ELB it means different CNAME targets).
