>Heroku declined our request for further comment. “Unfortunately, we can’t publicly comment on legal matters,” said spokeswoman Dana Oshiro in an email. “Heroku is committed to our customers’ success and focused on delivering the best possible product and experience. We’ll continue to update our customers via our blog.”
I don't run a company, and I'm not responsible for maximizing return on my shareholders' investment. So maybe I lack the mental schemas that make these sorts of statements seem reasonable, but they always strike me as short-sighted. Yes, admitting wrongdoing may open you up to legal liability. But it seems to me that, in the long run, it's far better to take your licks and own up to your mistakes than to hide behind the shield of pending litigation. My gut instinct is to offer a sincere mea culpa and provide a transparent roadmap to improvement, and if you lose a court case then so be it. I'd like to think that the good will and respect you build from accountability and honesty will ultimately outweigh any short-term losses from court settlements or judgements.
As I said, I don't live in that world. I would be saddened, though not necessarily surprised, to learn that I was wrong.
Seeing that there is a website/lawfirm[1] threatening legal action, I can bet Heroku/Salesforce lawyers jumped on that ASAP and let all their employees that are allowed to talk publicly not to comment on the issue anymore.
The US legal system is setup that if a company admits fault in something like this, it makes defending yourself much more difficult. Meaning if a lawsuit is brought about, it's not about proving innocence anymore, it's about mitigating the damages you will have to pay. Your lawyer is going to want to have as many options available to him/her to deal with your case during it's lifespan, and admitting fault before the lawsuit even begins will limit the lawyers options when it comes to negotiations and what tactics they can use in the courtroom.
In the grand scheme of things, this is why it's always better to defect in the simplified idea game of the prisoner's dilemma. Collaboration is rarely rewarded once lawyers get anywhere near an issue and opting to collaborate (i.e.: admit fault and try to fix it) more or less wins you no points in court, in the U.S.
Given that each possible liability is its own case and stands on its own merits, it pretty much matches the simplest case of the prisoner's dilemma perfectly, IMHO... and so here we are. Sadly.
Collaboration is rarely rewarded once lawyers get anywhere near an issue and opting to collaborate (i.e.: admit fault and try to fix it) more or less wins you no points in court, in the U.S.
Might, in part, have to do with how lawyers get compensated. A short case or a quick settlement will result in very little compensation for them. Over the course of years and decades that matters so we see everything analyzed to death. At $500+ an hour. Per lawyer.
The careful decomposition of problem domain into solution domain followed by recursive search of the solution domain is the lawyer's job. If they don't analyse to death, they are breaching their fiduciary duty.
Don't like the legal method? Don't hire a lawyer. Try doing it yourself. See how far you get when the other guy's lawyers have done the exhaustive analysis and found the deciding element buried in a logical node on a distant subgraph you didn't even know was there to be explored.
"The overlawyered US is the richest and most powerful country on Earth by a country mile."
Thanks to the lawyers, no doubt. I am sure you didn't meant to say that if we pass some tort reform and limit some nonsense lawsuits (IP, malpractice etc) the country will lose it's wealth /power rank.
The rule of law is a key part of the mix of institutions, geography, culture and luck that have made the USA the richest country that has ever been.
And with the rule of law comes lawyers. I know it's fashionable to hate lawyers, but they're a necessary profession.
Law is a field in which small details can have very large consequences once the full line of reasoning is unfolded. Because of the importance of exhaustively covering all lines of argument to the maximal possible depth, lawyers are required to do so.
They don't do it to bill you more. It is their legal duty.
If you don't like that lawyers are required on pain of loss of income, loss of profession and potentially loss of personal liberty to give you the fullest and most complete service that they are able to give you, then you are entitled to represent yourself.
Let's be honest, it isn't this firm's legal duty to set up a marketing website to try to find a plaintiff to launch a lucrative lawsuit. Law firms seek to offer the level of service that maximises their returns, just like any other corporation.
Not all that is a part of the US is a reason for its success. Whilst the US does a lot of things well, there is always room for improvement, and some of the aspects of its legal framework is one of the areas that could benefit. The US spends 2-3 times as much on tort as other developed economies; this is clearly a problem, not an advantage.
Sure, there's room for improvement. But torts law reform is no the original point I was arguing. The root of this thread is a suggestion that lawyers deliberately complicate the law in order to maximise their bill.
I was explaining why lawyers spend so much time minutely examining every tiny bit of a case for their clients. They have to. A lawyer who doesn't could find themselves in breach of fiduciary duty.
Also, read what I've said again. Nowhere have I said that the USA is rich and powerful because of lawyering. But neither has it stopped the USA from being so.
The US legal system isn't really set up any particular way with regard to first party admissions. It's just a statement that can be introduced into evidence like any other. Indeed, usually when they are treated specially it's to make them non-admissible. E.g. in an accident, your offer to pay the injured party's medical bills can't be introduced as evidence of fault.
The reason lawyers tell people to clam up is because jurors put great weight on what people say about their own actions. It's precisely the other edge of the sword of what you're talking about--a positive public statement can have a great effect on customers, but one that comes out wrong can have a very bad effect on jurors when the plaintiff brings it up in court.
Agree. Admitting liability in public before the trial started is very detrimental. After you admit liability, the only dispute left is how much you need to compensate the plaintiffs.
Just imagine being arrested. If you tell the police you are sorry- you are going to be found guilty. That is why all lawyers would tell you to remain silent until he arrives to the police station. The risk of admitting liability when talking is too big.
> Yes, admitting wrongdoing may open you up to legal liability. But it seems to me that, in the long run, it's far better to take your licks and own up to your mistakes than to hide behind the shield of pending litigation.
This is an admirable, well-intentioned perspective, but it is one that won't net you much benefit, not even in the long run.
Filing legal papers -- or, in this case, raising the banner for a class-action -- is serious business. By the time it happens, the time for defusing the situation and calling a truce is long past. The only opinion that matters is that of the judge and/or jury. And let's face it, being the loser in court is going to have far more impact on the public's opinion of the defendant than any honest-from-the-heart statement or outreach.
So with that said, there is nothing of substance that the defendant can say that will help them in court. The opposing attorney is not going to say, "Well, I was going to grill you, but I've had a change of heart ever since reading that impassioned, honest, salt-of-the-earth statement you released pre-trial". The judge won't care either. And the jurors aren't supposed to even be aware of it.
I think a civil trial is not much different than the situation portrayed in the famous "Don't talk to police" video:
The people you're facing in court don't care that you've made an effort, post-lawsuit-filing, to be upfront and friendly. Anything you say will not help you, it can only hurt you. In this case, Heroku's reticence in speaking their mind outside of court is the right one.
What makes this proposed class action suit possible is that Heroku did admit[0] that their service didn't match up with what their documentation and website claimed, that they messed up, and would be fixing it. And it wasn't something like "our documentation and site could have been misinterpreted", they actually admitted the website and documentation were inaccurate with metrics which were incorrect. It's like a class action lawyer's wet dream of culpability on the part of the defendant.
There's a middle ground; it is better to fess up and try to square things with your customers in a situation where you've done wrong, but at the same time you don't want to be exploited by unscrupulous plaintiffs in turn. Most of those customers just want what they paid for or to recover their actual losses (what are called economic damages in law) but a few may see a lawsuit as an opportunity to enrich themselves. The court system is supposed to prevent that, and overall it works moderately well, but once papers are filed it's better for the arguments to take place in court rather than outside.
They have made public statements on their blog, and hopefully that will continue.
In this situation, I think it is the right move to avoid getting involved with this shameless law firm trying to incite drama for their own gain.
This seems like precisely the type of situation class action lawsuits are intended to deal with: a company defrauded a lot of its customers in the same way. It makes more sense for all parties that the case be tried as one than to hear each claimant individually.
In MIT's Mathematics for Computer Scientists course (6.042), I remember learning from Tom Leighton that in certain cases randomly distributing jobs actually IS the optimal solution to load balancing. This technique was somehow associated with Akamai, Professor Leighton's company.
I think the matter is if random distribution is what is documented, no one will raise an eyebrows. I believe the Heroku claimed intelligent routing based on server loads to route traffic to idle servers. That threw people off.
I dont know, but I would assume it has to deal with the fact you can't predict how long a job will take to complete, and a random selection would have more resources-more free-more often, verses FIFO where a group of really large tasks could halt everything.
And what if they turned the whole thing into a request queue and workers get the "messages" as they become available? To ensure there are no timeouts, they could define a max time a request may stay in the queue, and then take it out and handle it with emergency workers or kill it?
IANAL, but my understanding is that threatening a class action lawsuit is easy, but actually bringing an action to court is extremely difficult. And (at least to me) this seems like an awfully weak case -- I don't see any obvious or intentional fraud.
The whole thing seems strange to me. I don't really understand what Rap Genius hopes to gain here. If Heroku wasn't providing enough performance for their money (regardless of technical cause), then why did they stick around so long?
Would Rap Genius still have sued if the documentation had been 100% correct and instead the problem was just plain old slow I/O on Heroku's side?
> If Heroku wasn't providing enough performance for their money (regardless of technical cause), then why did they stick around so long?
We didn't know how bad the performance was because Heroku's tools (logs and New Relic) reported incorrect performance data (i.e., they said requests weren't queuing when they were actually spending a ton of time queuing)
EDIT:
> Would Rap Genius still have sued if the documentation had been 100% correct and instead the problem was just plain old slow I/O on Heroku's side?
To be clear, we have no official association with http://herokuclassaction.com/ or the lawyer behind it – I actually found out about the site from the article's author when he interviewed me for the story.
(But I do think Heroku owes its customers a refund)
I don't think the issues with New Relic was necessarily Heroku's problem. Sure, Heroku did have some outdated info on their website and their logs may not be fully accurate, but I personally don't think that they should provide refunds for that. Solving the problems that have been raised would be a better service to their customers. Providing refunds probably doesn't help Heroku create a better service which is what I think will matter long term.
Agreed, but it's not the one true monitoring system. I happen to be biased and think it's pretty good, but it's not a first-party monitoring system that they swore on a bible was correct.
Maybe they didn't swear on the bible, but when you use a tool from a company with which Heroku has a partnership, you expect that tool to give accurate readings. I would, at least.
This is a civil matter. Probably under torts law, especially if there's anything in California/US torts law that resembles Australia's tort of "misleading and deceptive conduct".
Fraud and torts are different. And to forestall the next obvious point: torts and contracts are different.
> I don't really understand what Rap Genius hopes to gain here.
A refund.
Heroku's selling pitch is linear scalability. Rent 1 dyno, get ~ (1 dyno / Σ dynos) improvement in performance. Rap Genius demonstrated that they were getting exponentially worse increments of added performance per dyno added.
> If Heroku wasn't providing enough performance for their money (regardless of technical cause), then why did they stick around so long?
This is orthogonal to the question of whether Heroku acted in a legal way.
> Would Rap Genius still have sued if the documentation had been 100% correct and instead the problem was just plain old slow I/O on Heroku's side?
Courts don't worry their heads with strawman hypotheticals.
> "I don't really understand what Rap Genius hopes to gain here."
Rap Genius isn't the ones doing the suing. Rap Genius wrote about it, which caused Heroku to come clean. I think Rap Genius already got what they hoped to get (i.e., exposure, official comment, etc).
> "I don't see any obvious or intentional fraud."
You don't need intent to be sued - you only need to have caused damages to the plaintiffs. What Heroku has effectively done was:
1 - misrepresented their product in a substantial way.
2 - "overcharged" their customers substantially by giving them less performance than they paid for.
Both of these are material damages, and there's a more than reasonable case for Heroku's customers to recover this money.
This isn't a criminal case here, there doesn't even have to be fraud (as defined in the criminal law context).
I think after reading the article, it is pretty clear they did not notice the routing issues until they noticed slowed down performance and till their consumers complained. That makes perfect sense, you don't really dig deep into routing until you hit a bottleneck, specially when you are unable to monitor it, and specially when your provider is lying to you about how they route.
A civil suit doesn't mean "you're a bad person." It means "we're not square." If you sold something with certain parameters and delivered less, even if you did it on accident you're not square with your customers, and even if your customer didn't notice you're not square with them.
From the article, Heroku says, "The problem affected an older version of Heroku’s service, he said, and those who use a new version of the service would not have these issues."
As far as I understand that is not really the case. The newer version is the Cedar stack which supports concurrent apps where Bamboo only supports non-concurrent apps. This would reduce the problem with random routing, but not eliminate it.
The problem is not that they provide random routing, the problem is they said in their documentation that they provided intelligent routing, a design choice they backed off of when they switched stacks.
Customers still have a problem, but Heroku doesn't from a legal / class action-y perspective.
At the end of the day Heroku are in the wrong here. Performance issues which Heroku basically wouldn't admit until they were outed about them meant that companies would have invested tonnes of time and effort which in turn equals money on trying to fix issues that were caused by Heroku the whole time.
Heh, nothing will make people go back to taking Heroku's side faster than a class action lawsuit on the other side. This would be an awesome form of submarine marketing if they were actually behind it :)
Um, why? I was one of many plaintiffs in a class action lawsuit against EA (which was settled in our favor). That did not help EA's image at all :)
It looks pretty bad when your customers complain about problems that are hidden by your own monitoring tools, and then you tell them to buy more crap to fix it! It's not a small amount either -- as mentioned the queuing time was larger than the request time. And on top of it you're paying out the nose for this crappy monitoring tool.
Even people who have been somewhat screwed by this issue with Heroku tend to love Heroku; Salesforce could make this right with generous payouts to affected people and 2-6mo of engineering to offer a better load balancer as a service (Actually, I think I could build this as a third party provider, even)
EA is basically a lost cause; their business model and culture is the problem. Having better sacks at Dachau wouldn't have helped the Nazi image, either (to auto-Godwin)
Well that's what they should have done BEFORE the threat of class action lawsuit. It makes them look at a lot worse if it happens after.
I'm pretty surprised they didn't offer refunds already. It was pretty clear they were asked for, and with no update by now, it's safe to say they were refused. That seems unreasonable to me. If you were in the position RapGenius, would you honestly not have expected a refund already?
I wonder what happened to Heroku. They used to be awesome. They still have elements of being awesome, but seem either resource constrained or self destructive. I find it hard to believe Salesforce wouldn't give them what they need to do things correctly -- ranging from "go multi-AWS Region or move off AWS" to building a decent load balancing system to handling this fiasco.
> Actually, I think I could build this as a third party provider, even
I dont use Heroku, but if you read Rap Genius' original paper discussing the queuing problem, they specifically stated that there is nothing that they can do about it.
RG couldn't do anything while using Heroku's load balancing infrastructure (well, other than rewriting their app to be more constant in performance).
A third-party could do a load balancer entirely outside of Heroku or even AWS, feeding sessions in based on feedback from the application ("intelligent routing"). Heroku even offers a way to bill for that kind of thing. You can open outgoing connections from each dyno and then load-balance over that.
Doing it with ELB would be possible I think, but I don't really like AWS ELB. I think it would make the most sense to do near US-East AWS, but arguably a CDN type service could do it from many locations. Being closer to users (as long as your connectivity to US-East/Heroku is still good) helps performance more.
While I agree this sways me to Heroku's side, I don't think this is their marketing ploy.
Rap Genius says they aren't involved either, so this is probably a single-person law firm looking to create some buzz for himself. He should threaten to sue Amazon next for all the times they haven't held their end of the bargain. That'll get him a ton of publicity.
On one hand the Rap Genius expose seemed like they got duped and mislead, but on the other hand something about this lawyer mentioned in the article comes off as ambulance-chase-y to me.
Class action lawyering usually sound ambulance chasey, because any given person doesn't lose that much in a given instance and so the actual people harmed have little incentive to sue.
At the same time, it's an edge case in the economy. If you harm a large number of people a small amount, they won't sue you to recover for their injury.
Really, though, given how much money Heroku charge and how much money Salesforce.com would be able to stump up in a settlement or judgement ... it was inevitable that a fast-moving law firm would jump on this.
If the decisions made by Heroku to misrepresent the product were made prior to the Salesforce acquisition (December 2010) do the founders of Heroku have any liability? Would this sort of thing be a consideration with an acquisition?
I don't believe any decision was made to misrepresent the product. The product just evolved and the product documentation didn't. The end result is the same — the product was advertised with inaccurate claims — but this appears to be more a case of neglect than a deliberate misrepresentation.
It's not a criminal matter. Mens rea -- "guilty conscience", the intention to act unlawfully -- is a component of crime. But it's not necessary for lots of civil legal matters.
For example, if we both enter into a contract, you can hold me to my end. If I fail to hold up my end without intending to fail, then you can still insist that I make right or seek remedies for my failure.
That all sounds right to me, but I think you've mistsken my meaning. The point wasn't that they lack mens rea, but that the decision to misrepresent the product can't have been made before the acquisition because that decision was never made at all. It was misrepresented by reality gradually changing out from under initially truthful statements.
You don't have to decide to misrepresent in order to be misrepresentative. If your material is wrong, it's wrong, and depending on the jurisdiction you may be on the hook.
Ridiculous? Well consider that it gives a strong incentive to ensure that claims made to customers are somewhat accurate. Reducing the cost of caveat emptor makes society richer by reducing transaction costs and so encouraging trade. Sellers have a strong informational advantage over buyers; in general the cost to them of making a correct statement is much lower than having each customer having to find out the truth independently.
Not that this is the purpose of torts law, which evolved mostly to fill glaring, unjust gaps in the remedies of the older forms of common law. But that is the social and economic function that it has come to serve.
It would have been an issue for the acquirer before or during acquisition, if it were present at the time, of course. Some, perhaps many, investors into startups also insist on liability insurance.
But the founders of Heroku have no more liability than the company in control of its assets, which in almost all cases would be some sort of Limited Liability vehicle.
I think the parent was asking if SalesForce could pursue the Heroku founders for some sort of liability (i.e. misrepresenting the company during the purchase).
"[The attorney who set up this site] has spoken with one customer, who has since backed off. “That being said, I am in the process of drafting a complaint, in case their feet warm up,” he told us."
It really disgusts me how quick people are to bite the hand that feeds them. Heroku is not perfect, but they give a whole lot more value than they charge for.
As unlikely as it is, I hope that Heroku terminates the account of anyone that joins the class action against them.
>It really disgusts me how quick people are to bite the hand that feeds them.
Really? Because it really disgusts me when a company:
* Misrepresents their product (saying it's A when it's actually B)
* Privately acknowledges this misrepresentation under the radar
* Makes large and ridiculous amounts of bank on this misrepresentation (since you have to throw more dynos at the app to mitigate the lag caused by the misrouting.. and Heroku may be many things, but economical it isn't)
* And on that note, provides very expensive monitoring and performance analysis tools that further perpetuate the misrepresentation!
* And then force you to resort to litigation to get some portion of this huge mis-spent investment refunded. (Looking at the suit, this is all that's being asked for)
This wasn't a mistake on Heroku's part, this was calculated. And even if you accept that it was a mistake, the fact of the matter is that their negligence in this matter made them a hell of a lot of money.
By misrepresents you mean that the end users didn't exactly understand how underlying architecture worked, didn't like the performance, but kept using the service anyway. If every customer I dealt with while working on the EC2 platform could sue because they didn't understand how every technical detail of AWS worked and they couldn't optimize for it...well there would be more lawyers employed by Jeff Bezos than engineers.
This lawsuit will go nowhere and it is sad that such a supposedly tech savvy crowd wants to bring out the pitchforks so fast. But I guess since Heroku has a lot of cash they are now the enemy since they've made it and they should be taken down.
The problem is that Heroku made claims about their architecture and subsequent performance characteristics that were flatly untrue.
The further problem it looks as if they knew the claims were untrue.
These are not matters of internal consideration. They cut directly to the heart of the Heroku marketing material: "Write for our platform, our expensive secret sauce will make your stuff linearly scalable."
In which case people expect that it will be linearly scalable. Not exponentially anti-scalable.
The same can be said for EC2. "Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change."
If my instances don't launch "quickly" or I've designed my app in such a way that it isn't 100% linery scalable I can now sue Amazon? You can't sue heroku for their marketing speak.
You can sue anyone for anything, success is not assured.
That bit of pedantry aside, Heroku led people on to believe that their queueing backend was intelligently distributing requests to nodes. Rather than randomly.
You are not allowed to argue this, as Heroku themselves acknowledged it!
Furthermore, they blatantly lied about it. From the rap genius article:
Heroku claims that though they received reports of unexplained latency over the past couple of years, they weren’t able to figure out the request queuing issue until they read the Rap Genius article. But there is evidence that Heroku knew about the problem for more than 2 years.
And the conclusion, also from the same article:
Based on Heroku's response so far, it appears their approach to fixing the problem is, “We promised you intelligent routing, we delivered random routing (which is worse than intelligent routing), so we're going to change our documentation to make it clear that we're only promising you random routing.”
This works for future customers, since once Heroku makes these documentation changes, everyone who signs up will understand exactly how routing works. But it does nothing to address the time and money that existing customers have spent over the past few years. What does Heroku owe them?
Who's to say random routing doesn't qualify as "intelligent"? What if random routing is the most efficient routing schema for 99% of their users? Trying to sue because "intelligent" doesn't mean what you think it means isn't going to be very successful.
I don't know how "intelligent routing" can be defined to mean anything. It can be interpreted a million different ways. This is probably one of the reasons why they used it in their marketing because it sounds cool and is a generic term that could be applied to anything.
I don't see much of anything in a lawsuit unless specific metrics related to SLA's that were agreed to were broken.
The documentation used to say that requests were routed to the next idle dyno, but the behaviour was silently changed to be random distribution, which is substantially worse for the customer. There seemed to be previous cases where Heroku were aware of the discrepancy but didn't act to fix it.
>By misrepresents you mean that the end users didn't exactly understand how underlying architecture worked
Uh.. no.
Heroku claimed to use intelligent routing, they used random routing. And then covered it up with their monitoring tools.
How in the name of crap can you blame this on the user?! Did you even read the Rap Genius writeup on this problem? If I were in their position, I would be utterly livid.
I read the article. I wouldn't call it "covering up". It didn't explain everything going on, just like all kinds of things that go on behind the scenes everyday at AWS that users don't understand.
Define "intelligent routing". Maybe it is intelligent to route 85% of your requests to /dev/null. Can you sue because your performance sucks in your application use case? Most normal people eval their provider, try to get answers from support (which rapgenius have done here), and if the answers aren't up to snuff or the performance isn't whay they think it should be for what they are paying they move to another provider. They don't sue their provider for being "slow".
At least Heroku themselves define "intelligent routing" such that the request will only be given to the free Dynos. Their random routing does not do that.
I think the money for this class action lawsuit would be better used if it would go for a sysadmin/devops person to actually deploy the systems involved.
"Bait and switch"? If a couple of Dynos is not enough for your load, you should be moving to self-managed hosts
The first line in the article "Rap Genius founder Tom Lehman says Heroku misled him and cost his company money."
I have to imagine they're involved based on the number of articles I've read about them over the last week. Their own article, tech blogs interviewing them about the article, and now the founder's picture at the top of this article.
Granted, the article is poorly written and seems to imply their involvement. But it explicitly states, near the end, that the law firm involved has yet to find any plaintiffs:
He’s spoken with one customer, who has since backed off. “That being said, I am in the process of drafting a complaint, in case their feet warm up,” he told us.
Furthermore, Tom (the dude pictured) commented in a thread above saying he didn't even hear about the lawsuit until Wired called him to interview him for the article. Consider putting away your "Jump to Conclusions" mat and joining us in the world of facts.
I don't run a company, and I'm not responsible for maximizing return on my shareholders' investment. So maybe I lack the mental schemas that make these sorts of statements seem reasonable, but they always strike me as short-sighted. Yes, admitting wrongdoing may open you up to legal liability. But it seems to me that, in the long run, it's far better to take your licks and own up to your mistakes than to hide behind the shield of pending litigation. My gut instinct is to offer a sincere mea culpa and provide a transparent roadmap to improvement, and if you lose a court case then so be it. I'd like to think that the good will and respect you build from accountability and honesty will ultimately outweigh any short-term losses from court settlements or judgements.
As I said, I don't live in that world. I would be saddened, though not necessarily surprised, to learn that I was wrong.