My startup is heavily invested in and tied to Heroku. We don't have the resources right now to get off of the platform quickly, but plan to as soon as possible.
Like Rapleaf, we have spent an enormous amount of effort optimizing and searching for the causes of latency, and overpay for dynos to reduce but not eliminate the chance of latency issues. Heroku has always told us it's our fault.
While I have no interest in recovering money through a class action lawsuit - which is just grossly unproductive - my confidence in heroku has been shattered and I am embarrassed to have chosen and advocated for them for so long. I look forward to getting off their platform for good, and there is no way I could recommend it to others for similar applications.
Heroku's most recent statement does nothing to resolve the issue for most of their customers, and does not reduce what is clearly a gross overpricing and misrepresentation of what they provide.
It's quite clear that Heroku is not just a bad, but a horrible choice for rails applications that are not carefully designed for concurrency and don't go against Heroku recommendations and use a concurrent application server.
So, uh, what was the thought process when choosing a PAAS provider? I mean, step back for a moment, and ignore the specifics of Heroku; they are a PAAS provider with no completely compatible competitors, is that not so?
Whenever I choose service providers (or choose to outsource services) my first thought is "what happens if I have to switch away from them?" Even if you think my first rule of business is too cynical, everyone has problems, and if you are locked into a provider while they are having problems, well, you have a problem.
I mean, PAAS seems like a great idea for people who want to write apps but not be sysadmins. But personally? I don't understand why anyone would sign up with a PAAS provider that was unique. I mean, if you have to re-write your app to change providers, you are locked in, in a terrifying way.
That's certainly a consideration, but every PAAS is "unique" by definition. Something like Google Apps has more lock-in than others. Some like Heroku are attractive because they have features not available elsewhere that would take a lot of resources to reproduce.
We chose heroku and Rails for the reasons you suggest, and even tried to avoid major tie-ins to the platform. That doesn't mean it's trivial to move to another platform.
That's not entirely true. Cloud Foundry (the software) is an open-source platform that allows for compatibility across providers. You can see a list of compatible providers here, in addition to cloudfoundry.com: http://www.cloudfoundry.com/partners
Full disclosure: I work for AppFog, one of the providers listed.
> That's certainly a consideration, but every PAAS is "unique" by definition.
I don't see how this is true.
There is nothing about PaaS that says there can't be compatible competitors. Hell, there have been PHP shared hosting platforms that were very nearly identical between providers for a decade before anyone started saying PaaS, and really, the "Innovation" that made shared hosting PaaS lies in per-usage billing.
> We don't have the resources right now to get off of the platform quickly,
I hear people say this a lot but I believe it comes more from the emotion of fear than a rational consideration of the possibilities. For example, Amazon Elastic Beanstalk now supports Rails deployments that are just as easy as Heroku, and you retain 100% control over the underlying EC2/ELB/etc resources while having a nice management layer to help you out.
Honestly I have not looked into it so far and hope you're right. We also use a number of addons, though I don't think any would be too difficult to replicate.
Regardless, it's a significant amount of work for a startup with just a few engineers, perhaps a week of two of distraction. I expect a bunch of "how to migrate off of heroku" blog posts will be out there shortly.
Thanks very much for the blog post idea, I think I'll do something with that!
FYI you can keep all your Heroku addons inside your Heroku account while utilizing them from a server outside Heroku. For example, if you use the "Redis To Go" Heroku addon, you can fire up your own EC2 instance (possibly through AWS Elastic Beanstalk) and connect to your Redis To Go instance using the same connection info. The main difference will be that you won't have access to Heroku's environment variables that you're using now, but you can use a config file instead.
Like Rapleaf, we have spent an enormous amount of effort optimizing and searching for the causes of latency, and overpay for dynos to reduce but not eliminate the chance of latency issues. Heroku has always told us it's our fault.
You may well find that running your own ops means you run into exactly the same sort of issues - most companies see multiple major problems with their infrastructure as they scale. Heroku certainly haven't delivered for everyone, and it seems their promises of painless scaling are just not being met when it comes to larger customers, but this stuff is hard, and moving off heroku won't get rid of these problems for you, but it will make them your problems and perhaps give you more of a chance to fix them.
It would be really interesting (for heroku customers and for everyone else) to see a write up of your transition when you do move - it may not be as painful to do so as you anticipate.
Kudos on not recommending a lawsuit, which I can't see helping anyone but lawyers at this point.
EDIT: I don't necessarily support a class action lawsuit against Heroku, but in the interest of conversation...
Scalability is not a magic knob you can twist as far as you want…
Isn't that the problem? That's Heroku's selling point is that scaling is basically a magic knob. Here are some of the things on their homepage right now:
deploy and scale powerful apps
Focus 100% on your code, and never think about servers,
instances, or VMs again.
Run any web or background process with any web framework
or worker type.
Scale distributed processes effortlessly with a single
command. Easily scale to millions of users.
Your comment seems to be defending Heroku, by saying Heroku is lying.
That's interesting terminology to use, "the correct way". Heroku's own devcenter article on deploying Rails apps to Heroku [1] still recommends the thin webserver, which isn't concurrent, and therefore causes all the issues being discussed for any rails app. In other words, if you're using thin, it doesn't matter how you architected your app, it's not going to run in concurrent processes on one dyno. So I'll assume by "the correct way", you're referring to both building your app for concurrency and using unicorn as your webserver. Doing only one of those things will not work, both are necessary.
But their article only mentions unicorn only as an alternative, and up until recently their documentation actually said "We recommend thin" [2] over unicorn and that unicorn was possible but wasn't officially supported (EDIT: actually the most recent version of the article in wayback machine doesn't even mention unicorn on the page).
I wouldn't call using thin "naive", given that it's what Heroku's docs have recommended for a long time. I don't think it's fair to call customers of Heroku naive for not knowing that when Heroku says "easily scale" and "use thin", they are talking about two mutually exclusive possibilites.
As bad as the latency-creating bottlenecks with random-routing and single-threaded apps can get, it is always the case that "twisting the magic knob" of adding more dynos would help. It just wouldn't help as much as people expected it to. Further, the amount of paid-for capacity left idle (while other dynos were backed-up) offends many sensibilities about how a system "ought to" work.
But at least at the margin, the "magic knob" still worked a little. I think that's why the problem festered: there was always a default recommendation -- "more dynos!" -- that helped, which also just happened to fit the mental model Heroku promotes (while also boosting its revenue). In one of the older support logs RapGenius posted, it was evident that while RapGenius was suspicious about so many dynos being necessary, they let the ticket close once that expensive-but-temporarily-suitable 'fix' was in place. They (and Heroku) turned the magic knob and then let their attention move on to other things.
"Scalable way" as defined by the platform Heroku was built and marketed for is single threaded.
I won't argue that the lack of concurrency support in rails and ruby is not also a stupid horrible unacceptable choice - and I never understood why the rails community defended it.
Regardless, Heroku was originally built for and continues to market itself as a platform that can scale Rails by using intelligent routing, but it's quite clear that they can not.
Maybe you understand the above, though it seems you are just trolling rails for being single-threaded. This is not the place.
Ok this is going too far. Did Heroku make a mistake? Yes. Did they mislead and lie to their customers? Yes. But is it worth pursuing a class action lawsuit that could ultimately ruin them - if not financially, but force their mgmt team to focus on this damn suit rather than fixing the damn problems? Hells no.
I hope RapGenius is not the lead plaintiff in this, otherwise I will be very disappointed.
I applaud them for raising this issue, and doing all the research they did. We are all much better off for it.
I applaud them for sticking to their guns to make sure Heroku fixes the issue.
But if they join this class action suit, in my humble opinion, it is one step too far. What is the max they can realistically expect to get back from Heroku? $250K? $500K? Surely not the entire $20K/mo, so I doubt it will be that much.
Given all the money they have raised so far, if I were one of their investors I would start to ask questions about if they are wasting their time frivolously beating a dead horse.
This is getting into 'bad karma' territory. I am sure someone, at some point, will want to sue RapGenius....they may remember how they behave now.
So yes, keep up the pressure, and keep forcing Heroku to fix their faulty systems.
They blatantly lied about how a core part of their service worked for years. It was raised to them previously and they didn't change the docs nor the way their system worked.
This negatively affected a lot of their customers in many ways (overspending on heroku, time wasted, lost customers and revenue).
When Heroku's customers have incurred significant costs and/or losses because of Heroku's false claims, I don't see why this is so inappropriate. We need to make companies accountable for their claims.
For context, I am a Heroku customer and have been significantly affected by all of this. I won't participate in the lawsuit but I can understand why others would.
They didn't "blatantly lie"....they lied. They built their product 1 way and it morphed into another architecture as they grew. They made decisions that optimized for their financial benefit, but I wouldn't go as far as to say they "blatantly lied". Let's not get too carried away here.
As a long time Heroku user, I was pissed when I heard this...but they have done so much good for my own development career and freelance gigs, that this is easily forgiveable with a slap on the wrist and a partial refund to customers that have paid a lot over the years.
More than likely, they just never realized how big of a deal it was - because they were busy expanding. A few customers (few being relative to their total support requests) probably did inquire and some manager probably developed the response that they are aware of the problem, but not dealing with it right now.
It's not as if they built the company on the premise that they are going to say 1 thing and do something else.
This is just something that fell through the cracks as they grew.
Offering, and charging money for, a monitoring tool (New Relic) that doesn't monitor a very important part of the Heroku request cycle is pretty blatant. It was probably pretty profitable too!
I agree completely, it's not like "intelligent routing" is some standardized feature that people have come to expect from Heroku. What people expected was that their app could scale easily on Heroku which as it happens was not strictly true, but to some extent below some threshold it was probably true for most people. People here keep showing a poor understanding of the basics of queueing theory and system administration, as if "intelligent routing" is something they purposely removed just to screw people over.
Think of it like this, if a food restaurant has a vegetarian burger on the menu. After a while they find that as they grow they can't keep producing the burger that way anymore. So they decide to use a beef product in the burger, but they don't tell anyone. They just keep selling it as a vegetarian burger. Do you think that is blatantly lying? I don't see a lot of difference here.
So the size of their purse determines their legal liability? What difference does it make if it is a big multi-billion dollar company that does a few billion in revenue or a startup? If this were Dropbox, would people want to see a clas s action lawsuit too? Doubt it.
Regardless, the point has been made. They need to compensate those customers - but this lawsuit is frivolous, opportunistic and going too far.
Did Heroku make a mistake? Yes. Did they mislead and lie to their customers? Yes. But is it worth pursuing a class action lawsuit that could ultimately ruin them - if not financially, but force their mgmt team to focus on this damn suit rather than fixing the damn problems? Hells no.
Nice template for an emotional appeal you've got there, dawg (seriously, wtf are you going for with the vernacular embellishments)
Your argument justifies malfeasance without penalty for any entity. Swap out 'Heroku' for Monsanto, Microsoft, or banks that launder money for drug cartels and read it again. See if you still buy the reasoning. If misleading and lying to customers (your words) isn't enough for a lawsuit, what is?
(Not saying Heroku is equivalent to any of the above companies, just illustrating the lack of reason in parent post).
Polemic though this site may be, my understanding of the situation leads me to believe that it makes no false claims.
If this is so, Heroku has a problem on its hands: its behaviour is such that it has alienated current customers and, more importantly, dissuaded potential future customers from employing the platform.
Heroku's challenges seem threefold:
One, they need to get their infrastructure in line with their branding, or vice versa.
Two, they need to assuage the doubts of current customers and offer them convincing reasons to stay.
Three, their image and reputation have taken a nosedive. I can't see well-informed potential customers considering Heroku without taking this saga into consideration. This would appear to be the most difficult, and the most important, challenge to solve. Heroku needs image rehabilitation in a big way. The only way I can see them achieving that goal is through increased honesty and engagement with their potential customer base. Unfortunately, I haven't seen anything from them that suggests they're adopting that strategy.
Wow, seriously? This is way over the top and unnecessarily snide ("Heroku...supposedly assists web application developers"). Stop Heroku? Pretty sure nobody's gonna get on board with that.
What is it even trying to accomplish? As far as I can tell it's essentially an ad for that law firm.
While I agree this does seem to just be a law firm trying to make some money, Heroku did directly lie to developers about Cedar stack's routing being "intelligent" when it is indeed random.
New Relic's plugin was pretty quickly updated to include the missing time after the rapgenius article came out. But that doesn't excuse Heroku from lying about what they were selling and so far they've done nothing to make amends for those lies.
This isn't true. Their main "how it works" page claimed intelligent routing with no mention of which stack. This has been changed since the uproar. There are other places the same.
Dear Heroku, slice off a piece of your offering that shows some love to the Ruby community again. Lets call it the Maple stack. Have this stack use the old routing infrastructure.
Edit: To be clear, I do not support this legal action. I do want Heroku to hear the community, but I don't want it to go this far.
The old bamboo stack has the same old intelligent routing, but the intelligence is only within each router. When the number of routers increased, the intelligence stopped mattering because concurrent requests were highly likely to not be handled by the same router, making the routing effectively random instead of intelligent.
So what you suggest would be great, but unfortunately not possible.
I'll just repeat my previous comment: For $60,000 per month they can't create a mode where all your dynos are behind a single HAProxy with "intelligent" least-connections load balancing?
In other terms: "Damn, the hallways in this building are getting too crowded, lets add some more hallways and shit and lets add new rooms... Later: damn, this is getting complicated. Peopke are piling up in doorways, theyre losing patience and leaving (or being forced to leave to ease the congestion)... Wish there was something we could do!"
I'd rather see everyone move away from Heroku than waste everyones time and money on the legal system, personally. The only ones who would win in a business class joint suit case would be the lawyers on both sides.
I think what Heroku did is wrong, but I still think class action suits are usually complete bullshit. The lawyers make off with millions or billions while distributing peanuts to the class action plaintiffs.
Inequitable money distribution aside, I think the bad press Heroku has received over this is probably enough punishment. I don't support how they handled this situation but the core problem isn't exactly that bad in the first place. Nobody died here, some companies simply had slow websites and to some degree that's their own fault, they should have done their own external monitoring if that sort of critical performance was that vital to them.
This isn't some grassroots action lead by startup CTOs across the valley. This is just a lawyer searching for a lead plantiff. I find the site rather misleading, and ultimately trashy. (Disclaimer: I am personally opposed to class action lawsuits.)
Is this in any way affiliated with Rap Genius? In either case, you should make it abundantly clear where you stand, as they're very publicly involved with this spat.
General question: do class action lawsuits need to be sponsored/initiated by an actual plaintiff, or can lawyers just start a bunch of class actions hoping one will eventually stick and they'll get some fees out of them?
The procedure for filing a class action is to file suit with one or several named plaintiffs on behalf of a proposed class. The proposed class must consist of a group of individuals or business entities that have suffered a common injury or injuries. Typically these cases result from an action on the part of a business or a particular product defect or policy that applied to all proposed class members in a typical manner. After the complaint is filed, the plaintiff must file a motion to have the class certified. In some cases class certification may require discovery in order to determine its size and if the proposed class meets the standard for class certification.
Upon the motion to certify the class, the defendants may object to whether the issues are appropriately handled as a class action, to whether the named plaintiffs are sufficiently representative of the class, and to their relationship with the law firm or firms handling the case. The court will also examine the ability of the firm to prosecute the claim for the plaintiffs, and their resources for dealing with class actions.
wtf is this? No one will get money from this. Last time I checked I got a .06 payout from some company that supposedly defrauded me of over $60. Quite a payout. Lawyers took 30% right off the top. Hmm.. maybe someone will make money.
It's to technical... I don't see a reason why you'd win.
Lies happen all the time, i saw someone here mention that Heroku didn't knew it was such a big deal.
They are expanding a lot, the issue (intelligent routing) has been raised and will be solved. They know what they've done wrong now... They didn't see it as a big issue before.
It's up to Heroku to resolve the problem, because at the end of the day it's an issue to be fixed.
And don't we see issues all the time in software development.
As long as the company itselves communicate about the problem and is honest about it when the problem comes up.
That's the only thing i would really care.
PS. I'm not associated (employed or client) by Heroku.
Sorry in advance for being a little off topic here: which VPS hosting companies have the best reputation for automatically restarting your failed VPS (for any reason) from the most recent backup? Many years ago, I was on vacation without an Internet access and my host provider restarted my instance and since I set up my services to start on boot up, all was good.
I would guess that I am a typical customer who does not really need fast scalability. I host several web apps with a modest number of users and what I care most about is reliability and then cost.
Do class action lawsuits actually have any effect on business practices? Just seems like an extortion that is somehow legal. No one benefits but the lawyers. Recently I've been included in a bunch unbeknownst to me that require me to submit written notes within a week of notice to NOT be included. Why can't we just eliminate the middleman in these cases to cause real change?
It is going to be interesting when this gets to discovery, and we find out what Heroku were saying internally this entire time about the issue. There are bound to be internal emails that discuss this issue and what to do (or not to do) about it. For this reason I can see this being settled. Perhaps refunds and an apology.
This is interesting, because I don't remember any similar class action lawsuits/attempts like this in the tech/SAAS sector, though I've often heard of threats. Have there been any similar attempts?
Like Rapleaf, we have spent an enormous amount of effort optimizing and searching for the causes of latency, and overpay for dynos to reduce but not eliminate the chance of latency issues. Heroku has always told us it's our fault.
While I have no interest in recovering money through a class action lawsuit - which is just grossly unproductive - my confidence in heroku has been shattered and I am embarrassed to have chosen and advocated for them for so long. I look forward to getting off their platform for good, and there is no way I could recommend it to others for similar applications.
Heroku's most recent statement does nothing to resolve the issue for most of their customers, and does not reduce what is clearly a gross overpricing and misrepresentation of what they provide.
It's quite clear that Heroku is not just a bad, but a horrible choice for rails applications that are not carefully designed for concurrency and don't go against Heroku recommendations and use a concurrent application server.