DDOS attacks are a fact of life, nice to know that Rails Machine will throw you under the bus when one happens. Doesn't seem to fit their homepage description: "You write Rails apps. We deploy, manage, support, monitor, and scale them. Done."
So if someone throws multiple tens of gigabits at your customer, and your upstream threatens to turn off your entire hosting company, you would respond "no way, we're going the extra mile for our customer"?
Rails Machine was, in all likelihood, compelled to act to either (a) preserve its relationship with is upstream or (b) preserve its relationship with its other paying customers that do not attract DoS attacks. Your idealism will fall down quickly when one annoying customer threatens service for every last one of your clients: it becomes a "do I continue to get paid or do I fight for this customer" equation.
Price out mitigation equipment and the multiple high-level engineers you will need to administer it before you respond telling me how wrong I am, by the way. The gear alone is an engineer's salary just to get started.
Any given site might not experience a DDoS attack, but if you run a hosting company, it will happen. The frequency depends on how many customers you have, how popular they are, etc. If a DDoS attack catches you by surprise, then you are very ill-prepared. One possible strategy is to throw the targeted customer under the bus and call it a day. For hosts of a certain size, that may be reasonable, as long as you clearly communicate that to customers.
I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers.
Granted, not everyone knows how to do that in a pinch, but it shouldn't be a big deal for a hosting company. You only have to figure out how to do it once and when you detect a DDoS attack, flip the switch for that customer.
DDoS attacks are too expensive to protect against. Here's a price sheet from Gigenet, starting from a low low price of $750.00/mo with $1000 setup that only protects against 50 megabytes/sec.
95% of dedicated hosts out there will immediately nullroute you if you get a DDoS attack like Softlayer, Ubiquity, etc... Unfortunate as it is, this is a common practice for hosting companies.
At the risk of getting downvoted again, I am going to re-paste a comment I made.
DDoS attacks are way too easy to do now, and something needs to be done about it.
Anyone can go to www.hackforums.net for a free UDP flooder (called shell booters there), or rent one capable of over 20 gb/s for $5.
Or if you want to do it yourself, pick a cheap throwaway vps from www.lowendbox.com, go to www.gametracker.com and grab IPs from COD4, Wolfeinstein ET, Medal of Honor, etc... and send UDP query packets with your IP spoofed as the victim. This will amplify the size of your attack 20x or more and hide your IP address. You can easily get 10-20 gb/sec attacks like this.
> I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers.
That isn't how it works, but I appreciate you trying to explain it to me (I have personal experience as a former employee of a hosting company, in this exact position). A typical DoS attack is designed to flood the pipe, usually with UDP. UDP spam attacks are far simpler than layer 7 attacks that you describe, and layer 7 attacks (if small enough) are easier to handle.
As a point of reference, I don't even have access to a botnet these days but I could knock your home cable connection offline in about five minutes of my time.
In your scenario, if you were to update DNS to point the victim at some EC2 instances, you'd accomplish DoS attacking EC2 instead as they only offer you gigabit connectivity on most EC2 instances. If the DoS attack is multiple gigabit, that isn't going to help you unless Amazon steps in and mitigates about as best they can. A DoS mitigation strategy generally has nothing to do with your application. If an engineer's first answer to a huge DoS attack -- as in, larger than the uplink -- is iptables or "scripts to detect and drop connections", that engineer is uninformed. I'm sorry.
I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's flowing across the wire, and a small TCP SYN flood directed at your server won't gain any attention from your host or their upstream. A multiple-gigabit attack that takes down the entire uplink will.
Depending on your position, there really isn't one. The hosting company can implement one, which they might be able to insert in your path if you are at the receiving end of an attack. There are Cisco products and a bunch of up-and-comers that can do this.
If you're a "typical startup" with an Amazon footprint, you have no mitigation strategy for flooding attacks aside from not attracting them. If someone points multiple gigabit at you, there is just about nothing you can do except hope Amazon can do something.
There's a range[2] of ISPs who will sell DDoS protection to you, either as an addon when you host with them, or as an external service (re-routing your traffic).
E.g. StormOnDemand just recently added it to their portfolio[1], which is note-worthy because they actually list prices right on the website.
Either way, even without "explicit protection" any ISP beyond mom&pop-size deals with these attacks every day and will sort them out for you for free the first couple times. Only when they turn into a habit or become so huge that they have to talk to their upstream they will politely ask you to throw some money their way.
Pulling the plug immediately is definitely not normal. However considering Pastie was apparently a sponsored account it's at least somewhat understandable (albeit a terrible PR move).
For serious accounts (in the 6 digits/year) absolutely not, unless the attack is large enough to affect other customers.
Admittedly RailsMachine looks very small, in all likelihood their pipe was rather easily clogged and they simply didn't have the choices that larger ISPs have.
> For serious accounts (in the 6 digits/year) absolutely not, unless the attack is large enough to affect other customers.
If it doesn't affect other customers, a hosting company won't act or even be aware, in most cases. They'll just send you a bill for the transfer. If someone attacks you and it impacts other customers, you get nulled. I'm aware of 7 digits/year and 8 digits/year accounts through industry anecdotes that have had machines nulled. The engineer operating the null doesn't say, "oh, that's X, maybe I shouldn't fix the network for my other customers".
There's a bit of middle ground between "sending a bill" and nulling.
I've been hit by two larger attacks in the past (GBit/s range) and the respective ISPs were both extremely supportive, switching our IPs while they tightened their filters. Neither billed us a dime despite our ingress spike making quite a bump in their charts and a lot of handholding over 2-3 days.
I understood the original comment about EC2 as a mitigation strategy so the hosting company's infrastructure doesn't receive the brunt of the DDoS gigabits (EC2 will) and it can still service all their other clients. In that case, even if the DDoS'd site will still be flooded and unavailable, its hosting plan doesn't need to be canceled outright. I suppose they could just skip EC2 and null-terminate the DNS until the DDoS stops, but I don't know if this has further implications.
In any case, thank you for a very informative post.
> paying customers that do not attract DoS attacks
A little off topic, but I've always felt a slightly uneasy about the concept of "attracting" DDoS attacks. Sure, if you knowingly piss off a bunch of script kiddies, you're attracting attacks. But it seems that nowadays, any site that hosts user-generated content is at risk of being attacked for any random reason. And yet, a lot of people talk about "customers who attract attacks" as if those customers are to blame. It almost sounds like blaming women who wear certain types of clothes for attracting sex crime.
Of course, the fact that you didn't do anything to provoke an attack might be irrelevant when your upstream faces a choice between cutting you loose and eating hundreds of thousands of dollars. People need to do what they need to do to protect their networks. Nonetheless, I'm curious what you guys think about the concept of "attracting attacks". If you blame the victim even a little bit, does that affect your judgment about what should be done in the case of an attack?
Not all user generated content is the same. Some of it attracts more attention, some of it less. The site operators have a great deal of influence in determining what shows up. You could use HN as a pastie if you wanted. But I imagine if it started causing trouble, pg would enact measures to discourage such use.
I admire the people willing to fight the good fight, but it doesn't seem like the guy running pastie.org has any skin in the game. It's easy to decide you're going to run a laissez faire type site when you don't pay the bills.
> It almost sounds like blaming women who wear certain types of clothes for attracting sex crime.
I completely stopped reading this comment here, when you wrote this, because that was a completely off-base comparison and has absolutely nothing to do with the topic at hand. Worse, you probably know it; I wouldn't assume you to be stupid. And that was a mountainously stupid comment.
In hosting, there are customers that attract DoS attacks. Period. Ask anybody who does hosting. IRC servers are a canonical example and are DoS magnets. Torrent trackers are another. Pastebin sites, like Pastie, are becoming another (look who uses pastebin.com a lot: Anonymous). Hell, Facebook and Google probably takes several DoS attacks a day just by nature of being well-known.
Don't even conflate my argument with an undertone of sexism.
Sorry if my comment came across as suggesting that your argument had an "undertone of sexism". I definitely wasn't trying to say anything of the sort. It was just an analogy that popped into my mind, and I don't think it was a particularly bad analogy.
But I don't think your unwarranted indignation adds anything to the question that I was trying to ask to other HNers. Unfortunately, that question was at the end of my comment, past the point where you stopped reading.
I know this is the norm for the hosting industry, but to anyone else, not being able to host a chat server (IRC) is blaming the victim.
It is as ridiculous as banning airports because it attracts suicide bombers.
The longer hosting companies put off developing a real solution to this problem, the more DDoS attacks are going to happen because they are so effective at getting servers kicked.
I don't remember it being about either DDoS (if anything, I thought they went to Amazon to avoid the DDoS, seemingly somewhat successfully) or "ToS" (which to me has an implication that Amazon decided they didn't like the service, as opposed to caving under the pressure of other people not liking the service).
> The Wikileaks website migrated to Amazon's cloud hosting service yesterday after being hit by a distributed denial of service (DDoS) attack. Amazon decided to discontinue serving the controversial website this morning in response to pressure from critics, including prominent members of Congress. ... Senator Joe Lieberman (I-CT), chairman of the Homeland Security and Governmental Affairs Committee, was among the congressmen who pressured Amazon to stop hosting Wikileaks. He told AFP this morning that he plans to question Amazon about its relationship with Wikileaks.
Well, if you want to believe that public statement (which I see maybe you don't in the second paragraph you edited in), then you also have to retract the DDoS argument, as Amazon expressly and clearly states that that is an incorrect assessment.
> There have also been reports that it was prompted by massive DDOS attacks. That too is inaccurate. There were indeed large-scale DDOS attacks, but they were successfully defended against.
>"Noteworthy: Getting attacked is against ToS at many hosts."
Yes, and the action we take is largely dependent on the magnitude of the attack and how it affects /other/ clients.
