Quite impressive. You know your blog is good when folks will try to take down a CDN to supress what's on it. He's also had heroin mailed to him in combination with a swatting attempt before: http://webcache.googleusercontent.com/search?q=cache:gEjqPfc...
The google-cache page doesn't seem to be loading right for me - It's still trying to pull off extra content like css and images from the regular site. That said, I found the article in way.archive.org, and it has images and formatting intact:
That's an interesting read but I'm really not sure why Krebs posted the real name of "Fly's" wife. She doesn't seem to be involved in her husband's activities at all, so what's up with that?
"fair game"? The Russian hacker did the same with him as he posted copies of Krebs' credit report, directions to Krebs' home and pictures of Krebs' front door.
Plus, threatening to kill Krebs' wife.
But since we're talking about ethics here: Sure, now that the hacker faces 30 years in prison, he's not short of probably sincere apologies. I could really believe that he now has changed his view and accepted his guilt. It makes me ponder the thought of if I were Krebs' to not only feel sorry for the guy but (if it were legally possible) to dismiss the charges against him.
Consider the much more likely outcome of the hackers' plan: That it worked. Would the hacker had the same sense of guilt then? Or the same sense of forgiveness as Krebs or me seems to have? Maybe. We'd never know unless he did. It's more likely, he would have enjoyed Krebs' ruined live. Maybe even continued to threaten his wife and family. Just for the fun of it.
Because doing that in the anonymity of the web makes it easy to misbehave in ways no one ever would in front of the public eye and even less in the eye of his family and friends.
"Fair game" excuses collective punishment, then? I'm not so sure.
Also, a "wife" is a person. How do you justify retaliating against someone by harming someone else? You steal my car, I beat up your wife- hey, "fair game"?
It might destroy their relationship by scaring her into leaving him, which is perfectly fair game. You dont deserve to have a relationship with your wife if you actively try to destroy someone else's.
Relatives of a victim are generally protected by the press, but relatives of a criminal are not. Right or wrong this doesn't seem outside of accepted USA norms. Other countries are another matter, where often criminals identities are protected.
Of course he's not going to get the charges dropped against this guy. Krebs should be pushing for the most harsh penalty possible against this guy so he can continually point to him as an example of what happens when you threaten him or his wife.
I think thirty years is extreme, but I think the hope is that this will serve as an effective deterrent against this kind of crap.
Fortunately in the US justice system victims do not determine sentences. That said, I agree that hackers who maliciously target an individual with disregard to collateral damage should get the maximum sentence.
The people he deals with don't have any rule book they follow. Likewise, it's up to him who he wants to go after. Clearly, he decided that turn around is fair game and went after his wife.
Unfortunately when you deal with criminals, you can't really justify any behavior on either side since they are both operating outside the bounds of the legal system. Outcomes are mooted in the context of the world they are operating in when it's devoid of rules, honor and morality among the participants.
I think this is naive and gullible. This person needs and deserves a rehabilitation attempt, which he's very unlikely to get in prison. The psychopathology of someone who doesn't merely threaten you with death, but an explicitly named loved one, is a very dangerous, damaged, malicious person. Merely being found guilty and in prison does not at all ensure this person accepts responsibility for what they did or that it's wrong.
What someone does while they're being watched isn't a good judgement of their character. What matters more is how they behave when they're not being watched.
I don't know why he did it, but there's a difference between a wife and a wife's name. There's a decent chance that even if the person isn't involved, her name is or may at some point be.
It's not difficult to open a bank account in your spouse's name without his/her knowledge, and use that account to do things he/she wouldn't do.
When I was in middle school and the internet was still fairly new (we had just gotten it) a classmate of mine hatched a terrible plan to get rid of a teacher he hated. He waited until the teacher was out sick one day then during the substitute's typical teaching pattern of having us "read these 3 chapters, answer questions then keep your head down until class is over" he jumped on the teacher's computer. After reassuring the substitute that he was allowed to he tried desperately to find child porn. His plan was to save it into semi hidden folders onto the computer then later on turn the teacher in for having child porn.
Fortunately this classmate wasn't able to find any and eventually gave up. But I've always remembered his plan. It's terrifyingly believable that if someone managed to get into your computer and download child porn, there is likely little recourse or way to prove you did not do it.
There's a further nuance there as well. Say you work for a site that accepts UGC images (like a major social network for example). It's required that you have a team who works with law enforcement to handle investigations / takedown / preservation. Anyone who works for said team unfortunately thus has to see those images and have them, at least temporarily, on their machine, but there is zero legal precedent to protect them. It's a legal grey area and thus I have and continue to recommend anyone in such position to refuse to risk it.
You have the defense of "we are specifically required by law enforcement to do this" which is pretty strong. It's no different from a locksmith being asked by a cop to break into someplace the cop needs to be.
This was my thinking as well. On a huge social networking site I would imagine this comes up an awful lot and to have to see those types of images along with other abuses has got to be difficult. I'm also curious if you see enough of them how it changes your mind and if there have been any studies on it.
Fortunately I know you can integrate with that hashing project (I forget what it's called) where they generate a hash for known images of child porn so those can at least be removed automatically but I don't know how much of a percentage that catches.
Hang on, just to clarify: if someone else puts a file on my PC without my knowledge, without me ever having seen or opened it, he could call the police on me, and it would get me convicted?
The UK is all sorts of fucked up. You're not lawfully allowed to withhold passwords to encrypted data you're in possession of, meaning someone can slip an encrypted USB drive in your bag, you get arrested, and sent to jail for god knows how long because you're unable to give a password to a drive that isn't yours (and maybe doesn't have any password).
I'd never considered that set of circumstances before. The politicians who voted it into law probably didn't consider it either.
And that's exactly the problem: we have layer upon layer of vague and badly-written legislation, which ends up creating terrifying loopholes like the one described above.
My tinfoil hat paranoia is not quite at the level of thinking they're doing it on purpose (although I wonder sometimes).
But it's easy to see that indiscriminate surveillance combined with these vague laws create a situation where anyone could be victimised by the authorities. It's a totalitarian dictator's dream.
It looks that way. Does anyone know what the UK legal position is about forcing someone to unlock their device?
I know you can be jailed here for refusing to hand over an encryption key. I don't know whether your passcode / fingerprint counts as an encryption key in they eyes of the law.
If you refuse to hand over encryption keys in court, after you'd been officially charged that's one thing (though not saying that it's justified either), but if you've been stopped for i.e. mundane traffic check and asked to hand over your phone for inspection, that's totally wrong and oppressive.
I can't believe they both pled guilty! Is it possible in the UK to appeal a conviction if you plead guilty? IANAL, but I think their are some narrow cases where it can be done in the US, for example if you can show that your legal counsel did not provide competent representation.
It is puzzling that they both pleaded guilty. They also both represented themselves in court, so it seems they weren't getting any legal advice at all.
Perhaps they weren't entitled to legal aid? I was under the impression that anyone charged with a criminal offence was entitled but perhaps I'm wrong.
Maybe they just wanted to get the whole thing over with quickly and not face further embarrassment? It's a good example of how these types of charges could easily be used by the authorities to intimidate people. The damage is done whether they are convicted or not.
To answer your question, it seems you can appeal regardless of how you pleaded [0]. But you have to do it within 28 days.
There was a person in Finland mass scraping the alt. newsgroups for porn, and they got some cp on their machine inadvertently. Conviction, publicity and a ruined life promptly followed.
If someone else put 10 kilograms of cocaine in your house without your knowledge, without you ever having seeing or opened it, he could call the police on you and you would get convicted.
The thing that happened to "all that stuff" is called the War on Drugs. It blended in to pretty seamlessly into the War on Terror.
The word "routinely" might be contentious, because we have almost no numbers on it and the law has a way of making things "true" despite reality, but the kind of court you're referring to is just called a court.
You might google the Innocence Project, if you're interested in this sort of thing.
The comments here are overselling the situation. Yes possession of certain child porn images is a crime regardless of intent. However prosecutions are no different from any other crime: they must prove the crime before a jury of your peers beyond a reasonable doubt. There's nothing automatic about it.
This topic always drives hyperbole here on HN and I'm not sure why. Investigators and prosecutors don't waste time trying to entrap innocent web developers. They are kept plenty busy by people who are actually making or distributing child porn. Source for that: I know someone who prosecutes child porn cases. He is kept incredibly busy with obvious scumbag criminals.
Law enforcement's ultimate goal is always to walk the chain of possession back to find the folks who are actually making the imagery--who are actually abusing kids. That is why there is strict liability for possession. It gives investigators a lever to flip distributors to help find the sources.
Possession seems to be a very nebulous concept in English and US law (probably because of their common law systems).
In contrast, German law defines possession as "having effective control" (with some more nuance obviously). Possession is also entirely different from ownership in German law. I can not control an object I have no knowledge of. If you place an object in my house, I only gain possession of it once I discover it.
I think this is one of the cases where the civil law approach of rigorous definitions is clearly superior to the common law approach of establishing precedent.
I won't say that it doesn't happen (bad outliers always exist); however it would be very unlikely you'd be convicted for it.
A few years ago when I worked in computer forensics there was this big myth that if you went on porn websites and one of the images was underage you'd get done for it. However intent is a big part of law and so there would always have to be something along with "just an image" showing some intent to have obtained and viewed it.
> Hang on, just to clarify: if someone else puts a file on my PC without my knowledge, without me ever having seen or opened it, he could call the police on me, and it would get me convicted?
Yes. In such a case, a presumption of innocence would be true in theory but of no value at all in practice. Such an attack could even be carried out remotely -- an attacker could compromise a machine remotely, then plant incriminating evidence on the compromised machine (i.e. child porn, terrorist literature, drug-dealing evidence, etc.), then alert the authorities.
This plausible scenario is another reason to vigorously protect one's computer against external attacks.
Yes. In US law this is called Absolute Liability. Formally, absolute liability means that a conviction does not hinge on the presence of mens rea.
You're likely familiar with this concept in the context of speeding tickets. All that needs to be proven is that you were, in fact, speeding. It doesn't matter if you could not have been aware of your violation due to, say, a speed-limit sign that was blown away by a storm. If you were provably doing 55 in a 45 zone, you have no recourse.
The argument for absolute liability with speeding violations is purely practical, I believe. The reasoning is that it's not a crime, per se, so the trade-off of individual protection vs expediency of trials is deemed worthwhile. Clearly, the same is not true of child pornography convictions.
> Yes. In US law this is called Absolute Liability. Formally, absolute liability means that a conviction does not hinge on the presence of mens rea.
Actually, that's "strict liability".
> All that needs to be proven is that you were, in fact, speeding. It doesn't matter if you could not have been aware of your violation due to, say, a speed-limit sign that was blown away by a storm.
That probably does matter, since exceeding the speed limit properly posted is usually the actus reus of speeding, so even to the extent it is a strict liability offense, the absence of proper signage for any reason (except when the speed limit is either the states maximum highway speed limit or a default limit for some other condition which does not require signage, in which case notice is provided by the law setting the default for the conditions, and the sign is a reminder) makes it so that no offense occurred. [0]
[0] Also, given that states do generally have default speed limits that apply in the absence of signage, one could easily argue that the absence of signage is itself a positive indication that the default speed limit applies, making available a U.S. v. Kantor-style "good faith" defense even under strict-liability principles. [1]
> What's the difference between strict liability and absolute liability
Not a lot, AFAICT; the main differences seem to be:
(1) "strict liability" is the term used in US (and, AFAIK, UK) law (though the latter seems to refer to a criminal offense to which strict liability applies as an "absolute offense"), both criminal and tort, and
(2) "strict liability" can be either an attribute of an offense as a whole or an attribute of an element of (the actus reus of) an offense (that is, there can be a required mens rea for some element of an offense, but if there is an element which does not require any mens rea, the element can be said to have strict liability.) From what I've seen, "absolute liability" is universally a trait of offenses-as-a-whole (though that may be because I've seen less about it, and am less familiar with the systems in which the term applies.)
I don't know about Dropbox, but I do know for a fact Facebook and Tumblr definitely do. It might have evolved since a few years ago, but they used a technology from Microsoft called PhotoDNA [0] to hash all uploaded images and match against known signatures of CP and most definitely involved law enforcement when found.
Is there actually any nation on Earth where powerful and/or connected people can't screw you over if they really want to and if you have no other powerful or connected people on your side? (and no money, etc.)
The thing about this stuff that's really alarming is that any random script kiddie could also do this by coaxing your machine into downloading something. That greatly increases the surface area of people who can screw you. Given the abysmally awful security profile of a lot of consumer software and devices this is very plausible.
> (1)[F1Subject to sections 1A and 1B,] it is an offence for a person—
> (a)to take, or permit to be taken [F2or to make], any indecent photograph [F2or pseudo-photograph] of a child F3. . .; or
> (b)to distribute or show such indecent photographs [F4or pseudo-photographs]; or
> (c)to have in his possession such indecent photographs [F4or pseudo-photographs], with a view to their being distributed or shown by himself or others; or
> (d)to publish or cause to be published any advertisement likely to be understood as conveying that the advertiser distributes or shows such indecent photographs [F4or pseudo-photographs], or intends to do so.
[...]
> 4)Where a person is charged with an offence under subsection (1)(b) or (c), it shall be a defence for him to prove—
> (a)that he had a legitimate reason for distributing or showing the photographs [F6or pseudo-photographs] or (as the case may be) having them in his possession; or
> (b)that he had not himself seen the photographs [F6or pseudo-photographs] and did not know, nor had any cause to suspect, them to be indecent.
There are some amendments in the Sexual Offences Act 2003, but I don't think they turn it into a strict liability offence.
In this case such prove should be possible. Files creation dates would show that the files were downloaded when the teacher was absent. The dates would make it easy to associate this accident with your class and then, after asking a few questions, with your classmate.
The real problem is that the minute it was found the teacher would probably be fired, or at a minimum put on administrative leave, the media would catch word of it and publicly tarnish him, parents would demand swift action, and before the investigation even had a chance to get going the guy's life would be ruined. In the end, even if he was found innocent, the lasting damage would done.
This is basic criminal defense stuff, actually. "What time did you go to the cafeteria? Did anyone else see you there?" Then talk to the people who he says saw him.
Are there any cameras in the school that might have captured his visit to the cafeteria? Review the footage.
How did he pay for his food? Did that create a record that can establish a time and location?
Are there any witnesses who saw the kid sit down at the computer? Like the substitute teacher, for instance?
None of this is unique to child porn cases. Establishing or disputing time and location is basic trial strategy. All a defendant has to do is create reasonable doubt, not conclusively prove innocence.
>This is basic criminal defense stuff, actually. "What time did you go to the cafeteria? Did anyone else see you there?" Then talk to the people who he says saw him.
And they'd remember they saw him after 6 months, and even more so that they saw him leave in 15:20 instead of 15:10, because?
>None of this is unique to child porn cases.
No, but all this make "I didn't change them because filestamps in file are in an hour I wasn't there" difficult.
Heck, the teacher himself will probably not remember where he was at the time the timestamps show...
And easily sold to a jury as proof positive after they were altered. I've feared for a lot time that this type of thing is currently in the government arsenal of threats and weapons.
Honestly, even if you can prove you were framed 100% and are found not guilty, the media has likely already run several stories regarding the teacher who's computer had child porn on it. The school will be forced to fire them. The school board will likely bar them from ever working in another school in that city again. When you Google that person's name the top categories will likely be regarding their trial / arrest.
Once an accusation of child pornography (creation or possession) is put out there if any of it gets exposed to the public by the way of the media (and it will) that person's life is seriously ruined even without prison time.
The problem seems to get worse, too and I certainly don't have any good ideas regarding it. Except maybe re-tooling a new search engine that somehow can take context / validity into account but that's exceptionally difficult to do. And even then if someone gets their news or information from any other source you're still screwed.
that reminds me of randomRambo's story. On stream he was searching for a file on his computer, and a file with a name containing CP. Long story short, police was called, "he molests his kid on stream" (totally untrue), police came, arrested him on stream. took more than half a year for him to get declared innocent.
> took more than half a year for him to get declared innocent.
Even if declared innocent, being charged on suspicion of possession of such porn is a life-long stigma that never wears off. For example, it will be impossible for a foreign citizen to obtain a US 'ESTA' visa waiver after such suspicions.
I get him speaking out for them about the hosting having been free, but Akamai is now the CDN that got bullied into kicking someone of their service against their own will.
Terrible PR, and that mud will stick in tech circles. Akamai folds under pressure.
I know it's a crude comparison, but we don't negotiate with terrorists for a reason.
I would bet money that the attack was truly epic... to the point where it was impacting (or was about to impact) other Akamai customers. Because you are right, this is terrible PR, and Akamai knows it.
DDoS mitigation is fundamentally a problem of "who has more bandwidth?" - if the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over.
The problem is that the economics, right now, are heavily tipped in favor of the attacker.
From the article: "The assault has flooded Krebs' site with more than 620 Gbps per second of traffic — nearly double what Akamai has seen in the past."
If it takes a mere 620 G-bit to screw Akamai, then they're obviously not much of a content distribution platform. I only need a few thousand compromised machines in the right countries to run 5+ T-bit scale attacks. This isn't the 90s any more, and Akamai has no reason to not have improved bandwidth capabilities, unless they're doing like the Telecoms companies by saving that money and never using it to improve their own infrastructure.
I'm going to call shenanigans. Do a quick google for the largest DDoS attacks on record, and this is one of them if not the largest. Pulling from places like Arbor, and their yearly reports, the largest previously seen were ~500Gbps. I seriously doubt you and "a few thousand" machines can magically be 8-10x stronger than the largest attacks on record.
I would love to see some sources that ANYONE can get close to that number. Short of the NSA bringing its full power to target a specific pipe, I don't think we're there yet.
I'm willing to bet you're not including compromised backbone routers and symmetrical gigabit-fiber connections. There's enough of the latter in the USA, to homes, to do that much and then some.
But what you're arguing isn't reality. Show me a source article where someone has compromised a backbone router, and then used it for DDoS. This is almost exactly what I was addressing when I said "Unless you use the power of the NSA to target a single pipe." Even in a hypothetical scenario where you have gotten your hands on one: How long do you think companies are going to let their half million dollar router be consumed for a DDoS before they take notice?
I think its pretty obvious you don't understand how internet traffic really flows, when you think "all I have to do is compromise 600 pc's with a Gb connection and I can launch a 600Gbps DDoS."
"I think its pretty obvious you don't understand how internet traffic really flows, when you think "all I have to do is compromise 600 pc's with a Gb connection and I can launch a 600Gbps DDoS."
I've been doing networking for 26 years. One of my largest jobs was mitigating Slashdot effect for two high-profile sites. I know very well how a DISTRIBUTED denial of service attack works, can work, and have done many of my own in checking security measures for those whom I consult. Compromising backbone routers is actually fairly simple. Too much reliance upon software stacks and not enough reliance upon sound hardware logic design that's proofed against attack in the first place.
>Compromising backbone routers is actually fairly simple.
Yes, the state of security on routers, even some rather large routers is embarrassing, but when routers have business-critical amounts of bandwidth? they are attached to pagers.
Regardless of what you think of us, the folks attached to the pager, when you start messing with big important routers, at least if you mess with them to the point where it interferes with the business needs of the people who are paying money for said routers? you are going to wake us up. You are going to have a really hard time using these routers for much more than an hour before there is someone on-site trying to fix it.
Sure, the state of security for monitoring is also abysmal. if you wanted to put in per-router effort, I'm sure you could take my pager offline when you take my router offline. but customers will notice, customers will complain, and at almost every place where I've been on pager, there have been alternate routes to get to me. Hell, I once woke up to a very excited office manager shouting and pounding on my door because the whole office was down, I was sleeping in, and my pager wasn't charged. It freaked the hell out of my roommates; the office manager had a thick accent, and was built like someone out of a HK action film. They thought for sure I was gonna get messed up because I owed someone money.
But yeah, I mean, sure, with sufficient subtlety, you could use a small amount of the available bandwidth on a poorly-monitored backbone router. And a lot of them are poorly monitored. But my point is just that once you start using them hard enough that it interferes with the business needs of the people paying for them? Regardless of how terrible the monitoring system is, people will notice. Security isn't the only thing that is embarrassing on those routers; businesses are used to this shit failing, and even if most people don't know what to do beyond turning it off and back on, when there are dollars involved, there are procedures for getting someone who does know how to fix it on-site.
Especially since Akamai has placed themselves in the cdn market as "WE cost more, but we are the best there is." Hell, a ton of other "cdns" are merely reselling Akamai with friendlier contracts and features.
I was impressed by that number too, but after reading your comment it seems awfully small. In 2016, what kind of attack do you expect CDN like Akamai to handle?
Don't worry about his comment. He has no idea what he is claiming. These recent attacks are labeled record breaking because they actually are. This one at 620Gbps and the recent ~1Tbps against OVH are the biggest in history, and still 5x less than what he claims.
Of course we will get there sooner than any of us in infosec want, but he is almost an order of magnitude off of what realistic threats look like.
Almost an order of magnitude? You obviously don't know what that means. I'm at half a magnitude of order off by your supposed words (protip: An order of magnitude meas you add a zero to the end of the number you're using,) and you know not much about CDNs if you don't think that multiple terabits of traffic are flowing through Akamai every second already.
Currently, I do the physical networking builds for a mental health company. We're already deploying 100 G-bit in these offices as primary connection trunks, because a lot of these services will be done remotely over video and audio.
I could open up a 20,000 user Camfrog Video Cluster chat room and could saturate a T-bit connection link just like that the second it's half-full. Have you ever used (let alone seen)a T-bit scale program before? Camfrog's been out for over a decade.
Tangent: orders of magnitude are exponential - if you want to say "half an order if magnitude" you have to do it along the exponential curve. 5x is about 0.7 orders of magnitude; half an order of magnitude is a bit over 3x.
Orders of Magnitude, n; a class in a system of classification determined by size, each class being a number of times (usually ten) greater or smaller than the one before."
There are very few disciplines where OOM is done by exponential form (astronomy/star magnitude being one of them.) It's almost always base-ten. When you use electrical conductivity in mineral identification, you're always multiplying a number by ten multiple times over. The effect of that? You either add or remove an equal amount of zeros to the original number being multiplied.
Chattanooga has symmetrical G-bit fiber to the home. 5,000 Chat-town residents got uppity, think of what would happen to Akamai if they couldn't deal with 600-ish G-bit.
It is, as it's a municipal network and not a shitty company-owned one. Designed right from the ground up from day one, much like the fiber service in Sandy, Oregon (300/300 for $40, no limits, caps, throttling, nada.)
> If the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over.
Is there no way to stop such attacks by coordinating with your upstream ISP, or with the sources of the traffic? Why do backbones allow it to be carried? Is the problem that these attacks are too many different individual streams to identify and filter?
It seems like there ought to be some way to hierarchically punt the problem to network operators. "Your network is contributing 10 gigabytes per second to this DDOS attack. Identify the sources and shut them down." - times each identifiable traffic stream.
Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network? Maybe a good-faith collaboration of different ISPs could result in quick shut-downs. Or if networks don't cooperate, then the next-nearest border does it for them (and if they don't like the policies under which their neighbor suspends the traffic, they can sign up to do it themselves). Imagine something like a mini automated DMCA type request. "This IP is DDOSing me", signed by the operator of a reputable network, having the effect of suppressing origin traffic from the IP when received by a reputable network. (Any abuse of the mechanism causes the network to lose its privilege to participate, and DMCA requests related to that network fall upon its neighbors.) Perhaps the suppression would be destination-limited so as not to be vulnerable to too much abuse of the mechanism.
There "is" such a way, see RFC 5575, but it seems to be little known/implemented/deployed/enabled/something.
5575 is a BGP extension that says "for packets from x to y, do z". Assuming a router knows on which if its input ports such packets arrive (and during a DDoS it doesn't have to wait long for the next packet), it can disseminate the flow specification towards the actual source(s) quickly, so the packets can be dropped quite far from the DDoS target, in the ideal case as soon as it reaches an honest ISP.
Egress filtering should kill much of the spoofed-origin traffic and this much of the rest — if deployed.
I'd love to know why 5575 isn't deployed. Memory concerns maybe?
Without net neutrality economics would likely fix this. "We'll limit our 60GBps traffic to this network block because our peering arrangement makes that expensive."
Couldn't an ISP just throttle traffic to a particular network block on receipt of an authorised request.
And that seemingly provides a general solution. Record output to particular network blocks and throttle traffic when it peaks beyond statistically normal bounds? Basically, applying damping.
The problem is a lot of traffic is spiky if you look at it from the view of any given network provider to a site that isn't always getting traffic.
So distribute the DOS traffic enough and it will be very risky for providers to throttle it without appearing to be overall slow for a lot of legitimate traffic.
The other problem is determining what an "authorised request" involves. As it stands it is already a problem that people can - and now and again do - manage to mess up routing tables by sending broken route announcement, re-routing large address ranges to the wrong location.
Too much of internet routing still relies on a large amount of trust. We unfortunately need less of that, not more.
Eventually that might lead us to a situation where we could properly authenticate and authorise requests like what you suggest, but today it is high risk.
One thing perhaps ISPs could do is share data on these attacks, and work to shut off the botnets after the fact- so they can't be used again. So for example if Akamai shared information with ISPs about which orgin IPs were involved in the DDOS attack, the ISPs could reactively reach out to the affected customers. Or the ISPs could even try to examine their own logs after hearing a report of a large scale DDOS attack to determine if any of their own nodes were part of it. There may be also measures ISPs cold take on their CPE routers to detect infected machines on the customer network and warn them more pro-actively.
Governments can already block traffic on their border and can organize attacks again foreign targets (remember China vs Github case).
Oppressive governments can do even worse things to human rights activists, like banning a person from international travel (like Mr. Snowden or Wikileaks founder) and getting them extradicted from other countries.
The Internet is supposed to be a dump series of pipes. However, in practice you can of course try a traceroute and then complain to the ISP, which will or will not cooperate. If the ISP does not cooperate, then you can call their upstream provider and kindly ask them to call their customer and yell at them, but note that money is flowing in the wrong direction for that to work.
Plus you get the problem that you are usually seeing DDOS traffic from innocent bystanders, if the ISP is shutting of a compromised home router, then their customer will not have internet for some length of time.
It probably seems like a bad thing to the customer paying for internet, and since they're the one paying for service they do get some say in the matter.
Which do you think costs Comcast more? Temporarily disabling 20,000 customers' internet connections, or forcing Akamai to drop Brian Krebs as a customer?
I think someone who has equipment engaged in DDoS attacks might be a bigger cost to the telecom than they are paying anyway. Those 20,000 customers are either using disproportionately more bandwidth and resources or they are at least unwittingly involved in illegal activity, so Comcast would have either a financial incentive or social obligation to temporarily suspend their service. The company should be complaining to the customer, and they should be worried about losing their service. If every competitive provider had these very narrow standards, customers should be more worried about not having Internet because of their security illiteracy.
Why not simply block traffic from the specific unknowing customer to the target? I.e. Allow all normal traffic from the customer to go through, except for the traffic that was identified as being part of a DDoS? Is it expensive for an ISP to do that type of intelligent/rule-based routing?
You can capture the IPs, but they aren't going to be the true originating IPs. Usually they are just random IPs generated on the fly. Or sometimes the attacker will pick a company or ISP and spoof the packets with those IPs.
> I would bet money that the attack was truly epic...
Seems like it not only was one of the most expensive attacks so far (and by far the biggest one to have ever hit Prolexic, according to them), it also made little use of reflection of amplification making it much harder to mitigate.
> to the point where it was impacting (or was about to impact) other Akamai customers.
That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers.
>That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers.
I... wonder if anything different would have been done for a paying customer. I mean, if the attack was big enough to take down other customers, and if Akamai had the choice between kicking one customer and all customers being down?
Why's that? The theory is that it was impacting other customers. If the notion is that they kick off the low-revenue person to preserve the larger revenue stream, then it would seem that the only way to get real protection from Akamai is to be their largest customer.
> Why's that? The theory is that it was impacting other customers. If the notion is that they kick off the low-revenue person
Krebs was not a "low-revenue person" he was a "no-revenue person" they provided the service pro-bono. With customers they'd have a contract, and while I don't know Akamai's contracts I assume they either have specific service clauses and/or "use clauses" where protection costs get charged to the customer.
Nope. Contract basically says "here are the service levels Akamai commits to meet" and "Customer's sole remedy is to cancel service if Akamai does not meet these levels." Looking at their contract language does not make me feel any better about whether they'd drop a paying customer in similar circumstances. (Quoting from a version available online at http://contracts.onecle.com/akamai/msa.shtml)
8.1.3 Akamai shall meet or exceed the network availability, capacity and operations levels as set forth in Section 2 above; provided that Customer's sole remedy for the breach of this provision by Akamai shall be the termination rights set forth in Section 10.2 below.
10.2 TERMINATION UPON DEFAULT. Either party may terminate this Agreement in the event that the other party materially defaults in performing any obligation under this Agreement and such default continues unremedied for a period of thirty (30) days following, written notice of default; provided, however, that in the event this Agreement is terminated by Customer due to Akamai's breach of its representations under Section 8.1.3 above and failure to cure, Customer's sole remedy shall be its election to terminate the Agreement without further liability to either party (except for Customer's obligation to pay all accrued and unpaid fees outstanding at the date of termination).
Having not committed a 620 GBps DDos attack on a blogger I don't like lately, how on earth do you generate that much traffic without reflection or amplification?
> There are some indications that this attack was launched with the help of a botnet that has enslaved a large number of hacked so-called “Internet of Things,” (IoT) devices — routers, IP cameras and digital video recorders (DVRs) that are exposed to the Internet and protected with weak or hard-coded passwords.
Pure speculation but you can get and click a VPS/dedicated server with a 1Gbit/s pipe almost everywhere. Most of these machines are not regularly patched and run often vulnerable PHP software.
I guess if you have a few 0-days for common stuff that is often hosted you can easily!? collect a few thousand servers that blast out traffic.
Additionally more machines have faster access to the internet as fiber gets more popular, if you have a decent sized botnet that can pump out traffic on average with 50mbit or so you can also get some serious traffic, not sure about getting 620GBps, through. I guess if you mix all of these and put some amplification attacks on top it's possible.
Oh. My. God. This is in text so it is going to come across as sarcastic but I 100% guarantee you it is not, I am 100% serious. I have been making that mistake for over a decade and yes I do know the difference but I've still been writing it down wrong every single time it's come up. Why did nobody tell me before now?!
I am genuinely grateful you pointed out my error, it's these little things that people judge us on when we write emails and technical documents.
Please excuse me now though, the Englishman in me demands that I hide under the bed out of shame for the next 26 hours.
Yup. Cable subscribers in the USA alone have enough aggregate bandwidth to knock Akamai totally offline if 620 G-bit is something they can't handle. My own uplink is 10 M-bit. 100,000 compromised connections that speed is enough for T-bit scale attacks, and compromising 100,000 machines in the new world of "Idiots of Things" is a no-brainer given almost nobody working in the IoT space even gives a nod towards security.
> Terrible PR, and that mud will stick in tech circles. Akamai folds under pressure.
Definitely. The lesson I'd take from this is that Akamai isn't serious about DDOS protection.
For me, buying DDOS protection is something like buying insurance. I don't expect to need it, but if the worst happens, I expect them to stick with me. The way I measure insurance providers is by asking friends how it was when they had a claim.
It strikes me as especially bad that they're doing it in the moment. It'd be bad enough if they said, "Sorry, Brian, this is too big a distraction; you've got 90 days to find a new home." But that they're dropping him in the middle of an attack? That means I can't trust Akamai.
I had some friends who worked at Akamai. I always got the impression that they were very serious about addressing anything which could disrupt service, including DDoS.
Yup. And it's those people I feel bad for. I'm sure I would have been one of the tech people saying, "We must not give in! Let's use this as incentive to keep upping our game. That's the only way we'll win in the long haul."
You're entirely ignoring the fact that there's no way Krebs could've possibly been paying Akamai enough to tank the attack.
>For me, buying DDOS protection is something like buying insurance. I don't expect to need it, but if the worst happens, I expect them to stick with me. The way I measure insurance providers is by asking friends how it was when they had a claim.
DDoS protection isn't insurance, Krebs gets attacked 24/7. Only an utter moron would be willing to sell Krebs DDoS insurance.
>That means I can't trust Akamai.
Which means nothing at all in a world without alternatives, hosts capable of tanking attacks like that number at two or less. But I get the impression you're not looking to spend hundreds of thousands of dollars a year on DDoS protection anyway.
> You're entirely ignoring the fact that there's no way Krebs could've possibly been paying Akamai enough to tank the attack.
They were hosting it pro bono. He never paid them enough to do anything. And yet...
> Only an utter moron would be willing to sell Krebs DDoS insurance.
But a smart person would cover him for free as a way of proving that they could handle the worst the DDoSsers gave out. To prove that they stick by their customers.
Most people who buy insurance never really use it. So what are they buying? A feeling of safety. Just think about the various insurance company slogans that come to mind.
> But he didn't buy anything. We can't extrapolate what happens to paying customers from the experience of a non-paying customers.
So, if he had paid one cent (thus being a paying customer), you could extrapolate?
I don't see how the price is in any way relevant here. They promised to protect him, and they failed to do so. Claiming afterwards that the premium was too low isn't the way this works.
Also, I doubt that it actually was "for free". He may not have paid in money, but likely in the form of (at the time positive) PR, for example.
You're right the actual amount of money is not relevant. What is relevant is that the contract he signed with them is not the contract paying customers sign when they do business with Akamai.
Since we have no idea what was in the contract this guy signed and it's all speculation, this discussion is totally vacuous and pointless.
> They promised to protect him, and they failed to do so.
How do you know what they promised? They could have promised protection, or they could just as well told the guy "hey, here is some free caching for you, m'kay? No strings attached". Hell, it's possible he didn't even sign anything, and there wasn't a contract at all!
If he were a regular paying customer, I would make the assumption that the contract he signed is likely the same, or similar to the contract I would potentially sign, and this would put Akamai in a very bad light to me.
Since the contract this guy signed is not the contract I would sign, I cannot rationally infer any information from this incident, good or bad.
If Akamai emails me and offers me some free service, then yes, this information would be valuable and relevant. Until that day, I can't make any use of this information.
You are missing the point which is that Akamai can't handle this DDoS.
Do potential customers care if Krebs is a paying customer or not? He went with them as they offer this service which apparently doesn't work as well as advertised.
There is no indication that Akamai can, or can't handle the DDoS. The only information we have is that they are only not willing to do it anymore for this particular customer. There is no indication that they won't do it because they lack the technical capacity to do it. Just as well they might not do it because this thing is financially disadvantageous to them.
As a potential paying customer, what they can and can't do is covered by their SLA, and that's all that matters. If they break their SLA they own the customer compensation. This incident is irrelevant.
Of course I don't actually know what kind of SLA and indemnification Akamai provides. Maybe it's bad. Then after analysing the contracts I would make an informed decision. These things are what I use to make decision, not random stories with no technical or business details on random blogs.
From what I've seen (quoted elsewhere in this thread) there isn't any significant penalty for Akamai if they are unable to mitigate, or choose to not mitigate, a DDoS attack. They might negotiate other terms, but I doubt it. DDoS mitigation is, by its very nature, a best-effort service, and reputation for not giving in to attackers is more important than any contract terms you're reasonably going to be able to get.
The attack was ongoing when Akami gave Brian Krebs 2 hours to find alternate hosting.
The attack showed no signs of waning as the day wore on. Some indications suggest it may have grown stronger. At 4 pm, Akamai gave Krebs two hours' notice that it would no longer assume the considerable cost of defending KrebsOnSecurity.
I mentioned that above, but I think this misses my point, which is:
If Akamai can't provide their service for free, then they shouldn't provide their service for free.
A CDN's whole business is resilience, which in this case makes them the bodyguard, not a bystander.
Whatever your opinion of Cloudflare, it seems clear to me that Matthew Prince keenly understands this, hence him reaching out and offering to step in and get Krebs back online.
tldr; If Akamai can't do the one job they exist to do in the face of an (albeit well armed) assailant, then they're the problem, not Krebs.
"This was flagged to my attention and I've reviewed all the interactions between the author and our team [cloudflare]. The site in question was using the free version of CloudFlare's service. On February 2, 2013, the site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks a site needs at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers and enhances DDoS protection.
The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network."
That was 3.5 years ago. CloudFlare's ability to deal with DDoS has changed substantially in that time and we deal with enormous attacks day in day out automatically.
So how would you feel about 600gbit+? I'm genuinely curious, would it be an interesting challenge or an immediate avoidance? As a business customer myself and a big fan of CF's general attitude to these sorts of things, I know be very happy to see the blog post on it.
We've offered to help Brian Krebs out. We don't see any reason we could not handle an attack of that size. We've already seen gigantic attacks and have a very well developed automatic infrastructure for dealing all manner of attacks at different layers and an experienced 24/7 network team.
How do you feel about Kreb's criticism of CloudFlare for, in his view, sheltering the web presences of various DDoS-for-hire services? Just wondering what your response to his articles such as "Spreading the Disease and Selling the Cure" is. He seems to have taken some pretty strong anti-CloudFlare public positions.
I just want to take this opportunity to say that as a paying customer, this attitude towards your duty of keeping a site online, no matter who's it is, is exactly why we swear by CloudFlare and why almost no one else seems appealing in this field. I've watched your competition snuff out customers for being too controversial, while you guys just get it done. Thanks for this, it's really changed the shape of the internet in some ways.
Honestly I didn't know about this instance, but as another poster has mentioned, I don't believe this reflects where Cloudflare is today, and that's who Akamai is competing with.
On a side note, they really don't appear to be making a concerted effort to get out in front of this which tells me that they either aren't aware of the reaction or don't think it's a big deal.
They gave him service for free and covered many, many smaller attacks, including the 20-100Gbps attacks he reported earlier this month. The fact that they decided not to cover one of the largest attacks ever documented pro bono isn't great but I don't think any of their clients would fail to understand the difference between a favor and a signed contract.
What would have been a bunch of positive earned media is now a flood of negative reaction and lingering questions that they buckled under the pressure.
The biggest DDOS ever, and Akamai dumps the client rather than defend it.
I agree and I could see them reversing their stance now with the press around it. More than likely a middle manager approved the pro bono usage and once it got expensive that manager found him/herself in hot water over the resources being consumed. A few apologetic emails to Krebs from that middle manager, but saying he had to go, and boom we find ourselves here.
In an ideal world, they would not cave... but someone has to pay the giant bill. Who would chip in?
The site must be constantly under attack, and it must cost Akamai a fortune in real $$ all year long. And, when their service is performing well, nobody is talking about it, so there must be very little positive PR.
If someone is ready to foot a 620 Gbps bandwith bill all year long, I am pretty sure Akamai will be more than happy and able to scale up further.
Too bad there is no always a cleaner/smarter solution than pure bandwidth and $$ to fight those attacks.
As someone who actually pays for DDOS protection, this doesn't alter my opinion of Akamai.
I already know that Akamai is expensive and not particularly good at it. They are a CDN who is trying to make some money on the side with unused bandwidth, and will protect their CDN business if it comes down to it.
They don't invest in active defense. In fact, I know folks who actually had to block malware being served from Akamai-owned IPs!
I suspect that is why they gave service to Krebs for free in the first place--they need the marketing.
Isn't this the point at which Cloudflare is supposed to gain a handful of PR points for putting him back online, pro bono, and then doing a write up on how effortlessly they handled the bandwidth with eBPF?
I imagine he is still talking with Akamai(they did not comment after all) and expects to be back after attacks die out. Switching would burn that bridge.
Unfortunately, Krebs has (correctly) repeatedly attacked Cloudflare for sheltering most of the most prolific DDOS attackers. I doubt that's going to happen.
Care to fill in the details for me? Do you mean to say the most prolific DDoS attackers work for Cloudflare? Or that their network somehow (?) shelters them? What do you mean exactly? This sounds interesting.
Cloudflare protects everybody who signs up, including control panels for "booter sites", which are web pages where you can allegedly buy a DDoS attack for an hourly rate.
Kind of like the police protects gangsters from getting shot by other gangsters, but you would really like them not to do that, so that the gangsters can just shoot each other.
In this case, Brian Krebs tried to convince Cloudflare to kick off the booter sites, so they are unprotected, and can DDoS each other. Cloudflare didn't put any effort into that idea, and now he's apparently angry that he didn't get through to them.
That's not called being "angry". That's called being principled. Other things that are considered principled include posting your opinions with your name on them instead of cowardly resorting to a throwaway account like you just did.
Cloudflare is not the police. They're a private organization that makes a profit from offering "protection" for people getting DDoS attacked. They enable the people doing the DDoS attacks by protecting their booter sites (https://www.google.com/search?q=ddos+booter). That's called a racketeering operation (https://en.wikipedia.org/wiki/Racket_(crime)), and that's illegal. There are laws against it. Just because our crappy government is too incompetent to file charges doesn't mean it isn't illegal.
If Cloudflare thinks they can foster criminal activity through their network because they're running a juiced up nginx proxy, they're wrong. The "slippery slope" argument is absolute nonsense. As Krebs himself pointed out, they already remove sites that are hosting phishing attacks and malware.
Cloudflare, it's time to do the right thing here and stop protecting DDoS booters. Your policies are helping to damage the internet and censor people, whether they're illegal (they are) or not.
> they already remove sites that are hosting phishing attacks and malware.
If only...
Let me quote [0]:
> CloudFlare will forward all abuse reports that appear to be legitimate to the
> responsible hosting provider and to the website owner. In response to a legitimate
> abuse report CloudFlare will provide the complainant with the contact information for
> the responsible hosting provider so they can be contacted directly.
So, if I report a scammer CloudFlare will forward my information to that criminal, putting me at risk. Gee, thanks!
and
> Since CloudFlare is not a hosting provider we do not have
> the capability to remove content from a website.
Or to put it in the words that they answer every abuse request with:
> Please be aware CloudFlare is a network provider offering a reverse proxy,
> pass-through security service. We are not a hosting provider.
Which basically translates to "We don't care, we want to pretend that we are not responsible for our actions."
They enable the booter's free speech; they aren't enabling their actual attacks. If comparison to police doesn't suit you since they're special, think of doctors.
Free speech? As an example, criminal conspiracy and solicitation of murder are both crimes of speech, and claiming a "free speech" defense will just make a judge laugh at you.
It is totally a decision for a private company to make. The operative word here being "private". Private individuals have freedom of association; they cannot generally be compelled to associate with people they don't want to.
You're seriously comparing DDoS attack markets to doctors?
It takes an intense contortion of the concept of freedom of speech to apply it to this malicious and illegal activity. I guess under that logic it's also okay for a personnel security company to host (or hide behind their nginx server) a hitman-for-hire marketplace too, as long as they're not the ones doing the actual killing?
DDoS attacks (and their store fronts) are not about freedom of speech. They are, always and everywhere, about the suppression and censoring of speech through violence. Protecting them means that you are protecting violence.
But if you're adamant on this being free speech, fine. Where's the free speech criticism for Cloudflare shutting down those phishing and malware distribution sites? Why are DDoS attack sites magically different and deserving of freedom of speech protection?
Cloudflare will obviously respond to law enforcement requests of what the origin server is. Krebs is not law enforcement, and neither are other DDoSers. What is your problem?
Let's say I'm running a site. It gets DDoSed. Almost all of the booters are behind Cloudflare. How do I proceed here? Call the local police? Email abuse@fbi.gov?
Most people just panic and start paying Cloudflare for protection. Pretty much for the same reason that most people pay into protection rackets instead of reporting them to law enforcement, because they're afraid and their livelihood is at risk.
Cloudflare also has this wonderful policy of forwarding abuse reports (with information on who reported) to the booter site in question. You can imagine the consequences of that from what just happened to Krebs for doing reporting on them.
I'm not saying that Cloudflare or DDoS mitigation shouldn't exist. I'm saying that should not protect sites that are doing the attacks that they profit to defend against.
My point is the traffic isn't coming FROM CloudFlare. When you're attacked, there's no way of knowing who is attacking you. Your recourses are the same even if CloudFlare wasn't protecting the brochure/control panel websites of the services.
If you are being DDoSed. What do you do? Call the local police? Email abuse@fbi.gov?
It's not a "brochure", it's how they meet their customers and take payment from them for their attacks. It's how they make it so anyone in the world can launch a 100Gbps+ attack in 5 minutes for $20.
If you get DDoS attacked, you panic and look for expensive DDoS mitigation, or you go out of business. Legally, enforcement for the specific attacker is almost impossible. Cloudflare both knows this and benefits from protecting it. They realize that customer connection is critical to the system functioning and yet continue to defend it.
> Let's say I'm running a site. It gets DDoSed. Almost all of the booters are behind Cloudflare. How do I proceed here? Call the local police? Email abuse@fbi.gov?
With or without Cloudflare in the mix, how the heck would you go about making the connection between a flood of traffic from a large number of IPs, and any particular booter site? I don't understand how taking Cloudflare out of the mix helps you stop the DDoS.
As an aside, the FBI is indeed interested in investigating large DDoS attacks. Contact your local field office to see if yours qualifies! :-)
Cloudflare realizes that the status quo makes it hard to prove standing to sue, and that's a large part of what allows them to get away with it. But if you go to Google right now and search (https://www.google.com/#q=ddos+booter), you will find that basically all of them are behind Cloudflare. They have spent an enormous amount of time brainwashing Silicon Valley into thinking that this is a free speech argument (as evidenced by some of the absolutely ridiculous comments in here comparing DDoS attackers to unpopular speech protection or making absolutely shameless comparisons to whistleblowers like Aaron Swartz).
DDoS-for-sale sites are not "free speech sites", they are for-profit criminal organizations engaging in the violent censorship of people that are too poor to afford proper DDoS mitigation or that want to control the privacy of their users by managing the SSL certs.
The booters aren't usually as powerful as these 600Gbps+ monsters, but they're quite adequate to wreck almost any network for a long time (most IP transit hookups for racks are 10Gbps or less, these attacks can be well in excess of 100Gbps), requiring you to spend exorbitant amounts of money to protect your site against what are essentially bored high schoolers with a spare $20, your competitors, or whomever. That money is then dumped back into the system, allowing the attackers to build even more sophisticated and powerful infrastructure, leading to worse attacks like the ones we're now starting to see. Krebs was one of the people to document this trend, and now his site has been censored off the net by the same people he was writing about. Why is it so surprising to everyone that he's avoiding Cloudflare?
< . But if you go to Google right now and search (https://www.google.com/#q=ddos+booter), you will find that basically all of them are behind Cloudflare.
Search Google? So should Google be delisting these sites?
If you're getting DDoS'd right now, and you want to sue the booter that is doing it, how would you know which one to sue? Cloudflare obscures the origin IP because it's a reverse proxy. But even if you know the origin IP, that's not the IP the DDoS is going to be coming from. So how does one match up an attack with a specific booter website?
As I just mentioned, Cloudflare realizes that the status quo makes it hard to prove standing to sue them or to go after the attackers, and that's a large part of what allows them to get away with it. How is the FBI supposed to conduct an investigation here? They're not going to be able to get subpoenas for every single DDoS booter behind Cloudflare (one group has documented over 200 of them).
I recognize that it's impossible to eradicate the problem 100%, but by driving it underground, you can dramatically reduce the amount of it by making it harder for them to conduct their business. Cloudflare could do this in a day if they wanted to, instead they sit behind a "free speech" argument waiting for someone to force them to cut it out. Don't say I didn't warn you if the government comes in to change the liability laws to prevent this sort of behavior in the future. Nobody's going to defend DDoS spam packets from criminal botnets as "free speech" when they're preventing all speech from occurring.
At least in the US, "advocacy of the use of force" is not protected by the constitution if it is "directed to inciting or producing imminent lawless action" and is "likely to incite or produce such action". https://en.wikipedia.org/wiki/Brandenburg_v._Ohio
So if someone hosts a site for selling drugs but doesn't sell it himself that is free speech too? It didn't help someone who called himself a Pirate and made a Silk Road.
Not even remotely. If the government steps in with a subpoena for the origin host IP or an injunction to stop protecting the site they'd stop. Someone on the internet asking them has no legal power to do so.
Why send cloudflare the abuse report rather than the police who are paid by the government to investigate crimes. I have little understanding for this current trend of letting suspected criminals go un-investigated while all focus is on private third-parties that is held up to act police, judge and jury.
Because one would think that it would be in CloudFlare's interest not to harbour criminals on their network. This logic seems to work almost everywhere else on the internet, including privacy friendly hosters in iceland. It's mostly just CloudFlare who replies to every abuse report with the same "WE R A REVERSE PROXY", no matter what the actual issue that was raised with them was.
If they were any smaller, their IP ranges would just go into the rogue-isp-blocklist, and that would be the end of that. But because they're mixing in the criminals with their normal customers, that's not really possible.
And since I am unlikely to be in any jurisdiction that CloudFlare is in, nor do I have any chance of finding out who these criminals are because CloudFlare is protecting them, going to the police here wouldn't really achieve much.
Cloudflare has a pretty simple policy. They only censor content when they legally have to, or when it's child porn. That actually opens them up to a lot of heat from people who aren't big fans of the KKK, the Westboro Baptist Church, or botnets. BUT they don't specifically allow botnets as a weird method of promoting them, it's a widely applied policy.
I would bet things would be a fair bit easier for them if they agreed to take things down which most people don't like, but from my position they are taking a very principaled stand for free speech. Are people on hn actually arguing we want more censorship on more places on the web?
Nobody in here is proposing that Cloudflare censor unpopular speech. We are asking that they stop protecting for-profit DDoS attack sites that are destroying the internet and using violence to censor people's ability to speak. That isn't a freedom of speech debate, it's a debate on the ethics and legality of defending and protecting criminal activity that financially benefits them, a timely topic now that this activity is actively threatening the ability of the internet to function for any kind of speech http://www.webhostingtalk.com/showthread.php?t=1599694http://webcache.googleusercontent.com/search?q=cache:0uf9RIu...
I agree that it's an ethics problem, and a non-trivial one at that.
It seems like another problem caused by the fact that code can be data and data can be code. By which I mean, both are information. 'Free speech' implies the intent to be communicated to people, and can be considered 'data'. However a DDoS is a bunch of information with the intent of affecting the behaviour of computer systems, and can be considered 'code'.
The problem lies in discriminating between the two, given that "bits don't have colour", as explained here: http://ansuz.sooke.bc.ca/entry/23
I'm not at all sure what the right answer is, here. I'm also not 100% convinced that Cloudflare has the right approach, but I'm leaning to "yes", considering the alternative.
(by the way, you'd probably be interested in watching the youtube clip jgrahamc posted elsewhere ITT, with someone from Cloudflare saying some words about their perspective on this dilemma: https://news.ycombinator.com/item?id=12564876)
I mostly agree with you but let's not take it too far.
> We are asking that they stop protecting for-profit DDoS attack sites that are destroying the internet and using violence to censor people's ability to speak.
A DoS is not a violent act. I am mostly ignorant of these things but I think attacks if this kind are a service that test our capabilities. My fear is that there might be calls for legislative actions against "DoS attacks" which would then apply to people sitting at home pressing F5.
>would then apply to people sitting at home pressing F5.
How would such a law be different from the current laws? If you sit at home pressing f5 with malicious intent and succeed at bringing a site down, you're committing a crime.
I don't know about you, but criminalizing the act of pressing F5 with any intent seems firmly on the way to Aaron Swartz-like cases to me.
What if you are just fed up of waiting for a site to reload and press F5 a number of times? And what about the (probably majority of) instances where the "attacker" is simply a person who unknowingly downloaded malware onto their computer to get free smileys or whatever?
>I don't know about you, but criminalizing the act of pressing F5 with any intent seems firmly on the way to Aaron Swartz-like cases to me.
What? Why is F5 a special case here and what on earth does any of this have to do with Aaron Swartz.
>What if you are just fed up of waiting for a site to reload and press F5 a number of times?
Did you intend to bring it down? Was it obvious that your activity would bring the site down? If answer to both is "No" then you're fine, this is how most laws work.
>And what about the (probably majority of) instances where the "attacker" is simply a person who unknowingly downloaded malware onto their computer to get free smileys or whatever?
Why are you even asking? If someone else commits a crime you're obviously not at fault...
Also, what was even supposedly wrong with the Swartz case? It was on solid ground both legally and morally, shame he never gave the courts a chance.[1]
[1]: Might as well expand on this a little so I don't get hidden by downvotes. I don't think Swartz deserved to go to prison, but given that he intentionally violated the law it's hard to argue that he shouldn't have been charged.
F5 is a special case here because it is the exact same action that a law-abiding person does. The reason I'm stressing the F5 case is because saying "Hey you pressed F5 with this motive, so you go to jail" is equivalent to thoughtcrime – you're being punished for your thoughts rather than your actions.
Now if someone is using tools specially built for DoS I don't have a a problem with them being prosecuted.
That is also a problematic definition. I recall similar arguments being made against "nmap"; should we ban nmap, or criminalize its use? I also remember when Dan Farmer was fired for simply writing a security scanner (https://en.wikipedia.org/wiki/Security_Administrator_Tool_fo...), using the same reasoning.
If you figure out how to build a 600Gbps DDoS attack with Firefox, you are correct, that still qualifies as a DDoS and you can go to jail for it already. People have been tried in court for using Low Orbit Ion Cannon before, in a few extremely isolated instances. A DDoS is a DDoS, but intent is obviously important, and you do need to actually cause a problem for there to be a crime. I think clicking reload a couple times would be a stretch here for enforcement, perhaps it's possible but AFAICT it's not yet happened.
But we aren't talking about protest with a reload macro here, these are for-profit criminal botnets. And one if them just took down the largest DDoS mitigation network in the world. Which means there aren't many sites on earth left they can't take down. Much smaller attacks have nuked Github for days. Who's next to get "freedom of speeched"?
I think a better solution is to have ISP s work together to warn and cut off access to botnet infected computers. They have the technical ability because they have strikes for copyright. Perhaps it could be a soft ban like an hour long ban or something.
But if two billion people decide to stay at home and continuously press F5, you should get freedom of speeched. I think that's the equivalent of a picket line. Not talking about automated tools other than "refresh page every second".
"I will send DDOS for $xxx, send paypal to ###@example.com"
That's the the extremely unpopular speech that you're proposing to censor. The instant you say "oh but that's different" because of the contents of the speech, you're interjecting your own opinion about that speech.
Which, actually, is fine, but don't play that off as not being speech.
At the level where Cloudflare's network isn't actually being used to send the DDOS attack itself, it's also still speech.
Cloudflare will close accounts when asked, backed by court order. The problem is on today's Internet, that's nigh impossible, which realistically means it falls to Cloudflare to interject an opinion on what's good and bad, but so far they've avoided that as effectively as an ostrich burying it's head in the sand, and so are effectively supporting many bad actors.
While I understand your position, the particular line you quote is not protected as 'free speech' because it's advertising to sell a criminal act for money ... I could be wrong.
Protecting unpopular organizations is taking a principled stand for free speech. Protecting people who profit from breaking people's web services is not.
"We don't take it down unless it's illegal" is a simple policy, but to be a good policy it needs judgment as well.
I have personally witnessed Akamai use strongarm tactics to "prove" you need their, "protection" in such a ridiculously high profile instance I am sure they have no shame.
If this is a CloudFlare Vs Akamai attack Krebs isn't saying, but I would put dollars to doughnuts it is.
While I'm open to believe you if true, you need proof, links to affected people giving details, etc... Not your personal anecdote that may or may not have happened the way you describe it.
Otherwise your post is merely an unsubstantiated personal attack against akamai.
well if you've never offered factual proof to back your "I've seen things" statement, that's not a surprise. If you've ever given proof to back it up, I would love the corresponding link.
I'm surprised that the Azure or Google Cloud teams aren't on top of this. They want tech people to pay attention to their stacks, why not host a high profile site like this to gain the respect of the industry?
You want them to push the idea that isp and other middle men networks should not be dumb pipes, but charge a different pricing depending on traffic type and intent?
Your comment may have the best of intentions, but that's how you take net neutrality out the window.
For both Google and Amazon the cheapest transfer you can have is 0.02 USD/GB, which in case of such attach gives 5.5k USD / hour, or 400k USD for three days of the attack.
Amazon's actual ELB IPs can handle relatively little traffic, "prewarming" is required in order to add more ELB instances--during a DDOS attack, you'll be overwhelmed almost instantly. Route 53 uses DNS round robin, which is trivial to bypass if you're planning on a DDOS attack (by targeting a specific IP). Google actually gives you an anycast IP, so they're a better option.
All that being said: the idea that only ingress traffic matters during a DDOS attack isn't quite right. If the connections are legitimate, you either need to be able to detect the attack attempts (requires expensive coordination and mitigation techniques, especially if the attack is much larger than what a single NIC can handle) or actually serve back the content (which will make your egress skyrocket).
You just have to make sure your infrastructure doesn't try to auto-scale up to actually handle all the traffic or suddenly you have thousands of high-powered instances to pay for...
I think the bigger issue was that he wasn't a customer (they provided the service pro bono), not that Akamai wouldn't keep a customer that got hit with such an attack.
If I remember what cloudflare's article said, that 600gbps attack was a reflection/amplification attack, which for them is likely easier to filter out then just large amounts of direct DDoS traffic, which is what this was evidently.
Shouldn't it be the job of the police to protect his web property. The police, or another government agency, protects citizens offline, why not online? Why do we have to rely on private entities for basic protection online? Time for an online fire department or something similar?
have things changed so much that cloudflare has enough capacity to seriously contemplate handling something that Akamai could not handle?
You've gotta get the bandwidth to your filtering servers before you can filter it. DDoS mitigation, as I understand it, is first and foremost a matter of having more capacity than the attacker.
Here's a "philosophical" question with regards to the internet, and perhaps even it's future. Given that a currently anonymous attacker, and likely not a "state" player (i.e. not a governmental entity with almost unlimited resources) has managed to DDoS a single website, does this portend that unless there are significant changes to the way the internet infrastructure works, we are seeing the demise of the WWW?
Kind of like a reverse wild-wild-west evolution, where the previously carefully cultivated academic and company site presence, gradually degenerates into misclick-hell? And the non-technical, non-IT savvy masses, in a bid to escape this all, end up in a facebook-style future where media is curated and presented for consumption (or perhaps in future, facebook-type entities end up with their own wild-wild-west hell)?
I have a strange feeling that we are seeing the decline of a city/civilisation; once you used to feel safe walking out at night, knew everybody in the neighbourhood, could leave your doors unlocked... and now, you don't dare to go down the lane to the left in case you pick up a nasty virus, and if you hear a knock on the door at night/email from DHL, you don't dare to even look through the peephole/preview the JPG!
You are not the first to come up with this idea. This same thought has been posted every year for the past 20 or so years in mailing lists, forums or Usenet (thought lately, not too often to Usenet).
I think prevention should be emphasized. If there wasn't so much garbage plugged into the Internet, there wouldn't be huge botnets to send DDoSes. There are few groups that scan the Internet for vulnerable systems, and rather than compromise them, send notices to the ISPs. In Canada, the CCIRC does this. But they only check IP blocks assigned to Canadian ISPs and enterprises.
Plus, why do so many ISPs still allow spoofing of IPs? It isn't 1999 anymore.
We should start a grass roots group to talk to everyone they meet, and get people to update their OSes, devices, and get rid of crap.
The quickest way to do achieve that would be to hold ISPs legally responsible for any damage caused by their failure to block spoofed traffic from their own network.
I'm not sure how well this would work outside of the U.S. though. Not everyone is as litigious as Americans are.
The latest attack wasn't using spoofed traffic, from my understanding. Hacked devices were directly sending traffic.
And I ask you this, how is an ISP supposed to know if a device is hacked, or for example, is a webcam uploading a stream to a redistribution site. It can take days to chase down all the IPs, even in the US, and get the ISPs to deal with them.
Should not a router be able to this filtering ? Making Internet Exchange Points do it would be even more quicker. Once big IXPs do it smaller would follow suite for the fear being cut-off and eventually ISPs.
Filtering can only be enforced at boundaries where an operator can say "Link N will only/never have traffic for net range foo/16." And it isn't always possible to make strong blanket statements like that.
Netadmins can make those kinds of statements about traffic originating from with their own networks because they set the rules. But at an interconnection the types of networks connecting, and the purpose of the connection might mean there is little meaningful anti-spoofing protection that can be done.
For example: I send a packet to google, it passes from AS 123 through AS 456 to AS 789. How is AS 789 going to tell the difference between a packet from me, and a forgery originating from AS 456?
It cant. One solution would be blackholing AS 456 from AS 789 at the requrest of its members. Hopefully this will teach 456 to stop misbehaving.
Though we do assume that AS wont itself misbehave and send a spoofed packet to one of its member peer and most of time its true.
We have to worry about misbehaving ISPs for which previously mentioned filtering works.
> Netadmins can make those kinds of statements about traffic originating from with their own networks because they set the rules. But at an interconnection the types of networks connecting, and the purpose of the connection might mean there is little meaningful anti-spoofing protection that can be done.
I dont think so. IXP can force peers to provide their IP Space even if its whole internet. At least they wont be able to spoof IP outside of their space. If they do spoof ddos from their own space the above solution would probably suffice.
EDIT: I just realized peer already has to give destination ip ranges. So IXP dont have to force anyone.
Perhaps its the government? Lots of people think all these NSA programs are evil surveillance, could be it's just the glue that has saved us from just this.
I'm personally amazed that people don't get hacked more often TBH... I can't think of any instances where non-technical people have been pwned in my own life.
I personally have a pa55word that I use for sites I don't trust, but the accounts never seem to fall or even falter. It's amazing really.
"Here's a "philosophical" question with regards to the internet, and perhaps even it's future. Given that a currently anonymous attacker, and likely not a "state" player (i.e. not a governmental entity with almost unlimited resources) has managed to DDoS a single website, does this portend that unless there are significant changes to the way the internet infrastructure works, we are seeing the demise of the WWW?"
I would like to see stats from Tier1/Tier2/IX for that.
Krebs claims it's 665Gbit/s https://twitter.com/briankrebs/status/778404352285405188
Such attack must be visible in many places, however not a single major ISP reported that in mailing list. Previous smaller attacks were reported 'slowing down' some regional ISPs. Perhaps ISPs got better.
Because you have to classify and filter out the spam packets before they reach the intended host and content, which is really hard to do at line rate, especially if you also plan to serve useful traffic.
At line rate, with millions of small packets coming in every second, even counting the number of packets per flow, a prerequisite for some of the simplest mitigation strategies, is really hard (often requires either expensive hardware like ternary content-addressable memory, or exotic data structures like counter braids that are very expensive to decode). A lot of the time people try to get around this by probabilistic sampling, etc. Similarly, MICA's ability to handle key value lookups at line rate on commodity hardware was considered a big success in the database community: https://www.usenix.org/node/179748. Hopefully that should be a good indicator of the challenges inherent in performing any sort of nontrivial computation at line rate, even really, really simple computation.
(This is why most DDOS mitigation strategies involve getting peers to load balance their traffic when it's still manageable, rather than buying bigger and bigger pipes; it's also why ultimately a large part of the responsibility for handling DDOS attacks rests on the shoulders of ISPs).
Linespeed is pretty fast, and there's a lot moving through. Routers are a bit like GNU grep -- they way to be fast is to not touch most of the data.
The more you have to touch, and the deeper you have to touch it, the more expensive it gets.
The real trick is figuring out what's good, what's evil, and downrating the latter whilst allowing the good. Given peering relations, BGP routing, and the sorry state of much of those protocols, tracing problems to their source, quickly, and getting a useful response, is difficult.
It seems to me that he's saying in order to move data at volume, you have to compute on it.
Computation is tricky. Per-bit, if you can handle the network input, you're probably able to fire packets up to the OS layer.
But when you need to run stats on the incoming data, e.g. an ML classifier of "bad/not bad" or "stop/passthrough", you might be O(n^2) or worse. Moore's can't hang.
None, really. It's mostly filters against common types of attacks at L3/L4, then OODA. Variations from normal get looked at and custom filters applied as appropriate.
And of course, there's lots of NOC to NOC back channel comms around this stuff constantly to stay relatively on top of things.
Not sure if this applies to DDoS, but a baseline ML method for security is outlier detection. For example, (1) you get a dataset that is mostly "good" data, with some "bad" data (2) you cluster it using something fast like k-means (3) data points are labelled as outliers if they fall further than some threshold from a cluster center.
I think that's the secret sauce for these quys. I'd be surprised if you can find out a lot about current techniques without signing an NDA and leaving your mobile phone in a box at security.
I still don't get it. Why not just absorb it? Serving static HTML and related data is fast. A single rack in a datacenter could easily serve this with out breaking a sweat.
Sure, this costs Akamai money they don't want to spend, but is such an attack noteworthy? Eh.
You'd still be filling the pipe. And possibly filling many small(er) pipes at interconnection points. The last meters of pipes into a rack isn't where the issues are.
I'm curious, what sort of datacenter racks do you run that can serve 665 Gbps of traffic? For instance, Google's ToR switches (in its own datacenters) as of Jupiter (2012) were 16x40G, which is just 640 Gbps. Obviously, all of Google can serve a lot more than that, and you can get absurd stuff like 640-port Infiniband etc., but you seem to have pretty unrealistic ideas about network capacity. And, as another comment pointed out, that's just the input for this attack... the output would be a lot larger.
The limitation isn't the NIC, it's the processor (and possibly storage). The 10 and 40 Gigabit NICs exist but processing them on a conventional PC is hard.
It is, just about, possible to perform actions on every packet in a 10Gb stream on an x86 machine. You have to use a userspace stack, handle packets across multiple cores, and be VERY careful with what you are doing so you don't do cache misses. At 10Gb/s you're talking only a few hundred clock cycles per packet - anything that doesn't work as planned causes massive backlog.
you need 665 gigabits of idle capacity to... wherever the attacker is coming from. If the attacker can send five gigabits from some town in vermont, and your provider(s) can't get five gigabits plus normal traffic from that town in Vermont to your scrubbers, then legitimate users within those networks will be denied service, even if users on other networks are just fine.
Essentially, it comes down to the fact that getting packets from point a to point b requires a lot of cooperation, and cooperation is difficult. Yes, yes, if you bought me the fiber, I could build you a 665 gigabit network, on the kind of money that a nerd could come up with, (not counting the fiber) but interconnecting that network with other people's networks? yeah, that's gonna cost you. Settlement-free peering is a thing, but it is really difficult to set up and maintain those relationships.
theoretically if it was a well distributed bot net then it could be a few megs from a TON of different sources.. and since akami had their own ASN it could just be jamming up Akami and not other AS
Even if they tried to mitigate and quietly semi failed (like 30% packet loss) the PR would have been better. It could be that such attack takes down their entire network hard. Verisign said a year ago to us they could mitigate 2Tbps for comparison.
Well a single 48 hr mitigation costs 12k/yr contact. Unlimited mitigations probably cost about 130k/yr. Either this negative PR is worth 12-100k in savings for them or they would drop any paying customer if the attack is over a certain threshold?
They might bill you that money, but on average it costs them less to provide that service to you, otherwise they wouldn't make any money.
Because we don't know the operating margins and the distribution of DDoS costs per customer, we can't infer how much this particular attack would cost Akamai.
Maybe because of the scale of this attack, relative to the business value Krebs provides Akamai? Shedding an attack of this magnitude (by dumping Krebs) was probably well worth the PR hit Akamai took.
The first thing a lot of people are thinking (and saying) is "switch to Cloudflare". But there's another name I think needs to be said - OVH. OVH can withstand a Tbps scale attack as far as I know, and it provides this to pretty much anyone. They have a pretty good interface and some of their plans are extremely cheap. They're also great at standing up for free speech, which I really appreciate.
Yeah, been quite happy with their DDoS protection myself for hosting game servers. And short of drawing the massive ire of organized crime and/or nation states, that's about the biggest DDoS bullseye you can paint on your back.
Krebs also hasn't publicly criticized OVH like he has CloudFlare, so I could see that working out well. Would be great press for them, too.
@ 123 IN SOA ns1.prolexic.net. hostmaster.prolexic.com. 2016092204 86400 900 1209600 3600
@ 900 IN NS ns1.prolexic.net.
@ 900 IN NS ns2.prolexic.net.
*@ 300 IN A 127.0.0.1
@ 300 IN MX 10 smtp.krebsonsecurity.com.
@ 300 IN TXT "v=spf1 ip4:... ip4:... ip6:... a mx ?all"
m 300 IN CNAME krebsonsecurity.mobify.me.
smtp 900 IN A 198.251.81.28
*www 300 IN A 127.0.0.1
It might be more useful to return the IP address of whoever made the DNS query.
This could trick the computers that make up the botnet to either attack themselves on the public interface (more resource-intensive than trying to DDoS your own loopback), or even better, their ISP's resolvers (it would force the ISP to do something about it).
With the recursive nature of DNS, I imagine that could get a little hairy as the DDoS'ers would then be targeting whichever DNS servers they were using.
It would be interesting to try out some of these new p2p website technologies like IPFS/WebTorrent with these high profile sites who are frequently attacked.
Hosting static blogs is really easy on IPFS (and if you absolutely can't live without comments: use disqus) but the URL's are cryptic and you either need a public IPFS gateway to access the site - which could get DDoS'ed - or run your own.
Another alternative is ZeroNet but you still need to run the client to access the site.
If the URLs are cryptic, you can use dns to make them look nicer. Take a look at the TXT record for ipfs.io, as well as the TXT record for _dnslink.ipld.io
Both of those websites are hosted through ipfs and have A (or CNAME) records pointing to our gateways. You can also access this locally if you happen to be running an ipfs daemon at http://localhost:8080/ipns/ipfs.io
I tried to get to an article on Krebs' site from a Bruce Schneier blog post, and couldn't, then bumped into this post in HN.
It's a pity Akamai booted him off; on the one hand, I can understand that it would significantly impact on their SLAs to other customers, but on the other hand it's a shame they don't have a lower impact network to re-host him on, and use this as a learning lesson on how to better mitigate such DDoSs...
I'd love to learn more about these botnets. I wonder about things like What's the average time that a compromised computer stays in this net. What is the typical computer (grandmas old PC running XP). Do the ISPs ever get involved to kill bots running on their networks?
Again, the speculation that it is IoT devices is unfortunately just that. However massive compromise of internet connected embedded device is not new: http://internetcensus2012.bitbucket.org/paper.html
no you don't because that's a bad analogy. We do however go to states and could ties to protect the infrastructures they paid the contractor to build.
ISPs are uniquely situated to stop this kind of ddos because the traffic originates from IPs they don't own. The traffic has a spoofed from address. And as a rule, the ISP.should only need to send traffic out of a neighborhood from the block of IPs that is assigned to that neighborhood. You can put a filter on every switch or even every interface allowing only traffic from the IP or IPs on the other side of the link to send traffic. A company like Comcast could make it default part of account setup scripts. If everyone did that, these would disappear over night.
Yes there are. but using your source address means that you are confined by the upload bandwidth of your hosts, instead of some mis configured DNS server with a 1000Mbps up in a datacenter. You'd just have to work a lot harder to get the bandwidth.
to add to that, You'll defiantly get some mis configured servers with 1000Mbps uploads. And those will be really easy to pick out of the lineup. And then you'd probably be able to call the DC and say that they should block that IP at their boarder and they would probably also comply because there's a good chance that customer that was doing 110Mbps and won't want to pay for 1000.
As it is now, because the source is spoofed, you can't really take the source offline, only take the destination down to keep the other hosts in close proximity running.
With a TCP connection you can pick the source and drop the handshake, basically never start the connection. Some of the windowing can be used to make a tcp connection less of an issue as well.
If a bot is using its actual source address, you can block its IP at the edge (and even ask the ISP to investigate). Thanks to IP spoofing, that's totally ineffective, so instead the only way to make the attack stop is to null route the host.
How does IP spoofing even work outside of those DNS reflection attacks mentioned on Krebs' blog? [1]
> "many were garbage Web attack methods that require a legitimate connection between the attacking host and the target, including SYN, GET and POST floods."
I constantly see references relating to DDoS attacks about how IP spoofing is such an obvious trick to use but I've never seen any way to actually do it. Why wouldn't every device on the internet spoof their IP?
> Why wouldn't every device on the internet just spoof their IP if it was this obvious thing?
https://spoofer.caida.org/summary.php - compromise a device in one of the ASes not marked "unspoofable." Those ASes do not consistently perform packet ingress filtering.
That's not to say that DDOS attacks stop being possible, but at least they become traceable.
What do you do when you have 10, 20, 50 million bots using their actual source address? Do you just block all 50 million devices? If so, then that would be a great way to initiate denial of service on those devices.
Actually, that would be a good way to take all those insecure devices off the internet, or at least prompt someone to do something about it (even if temporarily, like hitting the
"factory reset" button) before reconnecting them.
Most countries don't allow cars on the road that are unsafe due to lack of maintenance. Perhaps it's time to do something similar for internet-enabled devices that cause serious harm to others. Hold the user, manufacturer, or network operator responsible for harm caused by their lack of maintenance.
Most ISPs have a clause in their Terms of Service stipulating that you won't use the service for criminal activity. I'm pretty sure a DDoS can be argued as a criminal activity. So, it is in the ISPs right to stop service to those nodes. --Yes, I understand the consumer isn't the criminal, more so they are the victim and the crime took place on their service.
Further, technologists tend to be pretty good at solving problems. I know this isn't the ISPs problem, but it is a flaw in the network, I'm simply wondering if anyone is attempting to solve this problem at the network level rather than simply building bigger caching services to protect those that pay for protection.
Wow, I figured that everyone that had hired vDOS would be irritated but that is pretty impressive. Still it says a lot for how effective he has been at rooting out this stuff, not like the TierN infrastructure folks have managed to track this stuff down with their resources.
Isn't this whole thing a bit silly? I mean what's the point? They just spend time on making him the best marketing, he'll double his audience/readers, no?
The point isn't to cost Krebs readers most likely. It's to show off how awesome this bonnet is.
I'd guess the DDoSer is jumping with joy over this news actually, because now the DDoSer can advertise his service with "I DDoSed Krebs so hard Akamai had to drop him!"
That would harm his business model, which is advertising.
It could work with Facebook Instant Articles, he may even be better off using it since they source the advertising and have been out trying to poach and source quality content.
Something about the platform-centric world we're in now is that this sort of attack doesn't have the blocking power it once did: you can mirror your content on Twitter, FB, G+, etc. and cross-link so people can still read your stuff. This makes the "denial" part pretty watered down; it's a wonder people even bother with these sorts of attacks anymore for non-services (i.e., for regular media material like text, photos, etc.)
Of course, maybe the goal is to deny someone ad revenue, but that seems awfully low-status for such a high-profile attack: "Yeah, we really got 'em! Denied 'em AD REVENUE for a whole week!"
The ddos attacks seem to be getting larger these days.
I've recently seen a ~200 Gbit/s hit us.
Does anyone have good resources around mitigation? I was looking at the BGP flowspec but was hopefully that someone might have come across other tactics?
relevant quote from that:
››Many readers have been asking whether this attack was in retaliation for my recent series on the takedown of the DDoS-for-hire service vDOS, which coincided with the arrests of two young men named in my original report as founders of the service.
I can’t say for sure, but it seems likely related: Some of the POST request attacks that came in last night as part of this 620 Gbps attack included the string “freeapplej4ck,” a reference to the nickname used by one of the vDOS co-owners.‹‹
I'd err on the side of caution and consider that string a Red Herring without more evidence to corroborate the claim. It is a nice coincidence though...
I understand that this is burning bandwidth for Akamai, but seriously, taking into account what is at stake here, I think they need to do their share and continue to support Brian.
Brian Krebs is a hero. Are Akamai executives cowards for dumping him? I'd like to add that law enforcement are heroes.
And it's honorable he wants to meet Fly in person, recognizing him as a human being. I haven't read it yet but I'm assuming the reference to 12-step hints that Fly's having some post alcohol binge regrets.
I'm sure alcohol makes it easier to hurt other human beings, which is why violent people are often drunk. I'd be ashamed of myself if I woke up realizing that I'd spent my life actively trying to harm other human beings for money, feeling no remorse until Karma (here defined as law enforcement officials) finally caught up with me.
I'm wondering if the rising scale of these attacks & the seeming ease with which sites can be taken down will ultimately result in an "authenticated" internet - ie. you can't even connect without identity verification.
We already see publishing through FB Instant Articles etc. moving in that land on top of the current internet, to combat these types of firehose attacks, the only solution may be to take authentication one level deeper into the connection level.
That of course sounds good to security agencies as that's the end of anonymity online.
I doubt it. I think a lot of these attacks (though possibly not this one) would not even be possible if ISPs did egress filtering of packets with spoofed IPs, and yet ISPs don't seem to be implementing even this feature. It seems unlikely that they would leap frog this and start requiring some sort of identity verification, just to solve DDoS problems.
Additionally, in situations that don't make heavy use of amplification (where egress filtering doesn't help much), the way it's usually accomplished is by compromising a bunch of hosts - home computers, routers, etc, and assembling a botnet. In those cases, if your device is compromised, it would authenticate as you anyway, so such a scheme would solve nothing.
>>In those cases, if your device is compromised, it would authenticate as you anyway, so such a scheme would solve nothing.
But if you had authenticated access, you could find exactly which C&C server controlled that botnet node and then who controlled that C&C server right? All these attacks depend on some form if amplification - if only to go from C&C servers to botnet. If just being on the network required authentication, you could trace back network connections and ID the controller even if attack was by a botnet.
I don't think the imperfect mapping between IP and user is the real problem with tracking down botnet controllers. You have a network of compromised hosts, and you can disguise yourself as one of them. There are lots of schemes for this sort of thing when you have a whole bunch of compromised machines under your control. For example, even if you made it illegal to run a Tor node, you could still just have your compromised network of hosts start running their own Tor (are you going to arrest all the people who just got a virus that starts a Tor network?) and you deliver commands from a hidden service. That's a huge effort on the part of society (banning Tor, implementing that, etc) that's easily circumvented.
In any case, like I said, you can't even get ISPs to do egress filtering of spoofed IPs, so even if it were going to solve DDoS, I don't think you'll get them on board for all the complications of implementing the protocols necessary and buying the equipment necessary to log all the traffic necessary to track down botnet controllers (who may be in a country where knowing who they are won't help you much anyway).
It's funny how my mom after reading "record cyberattack" would be wondering how many poor people died but what it means is that somebody was downloading images from website many times.
There are a number of factors that go into play (did the site use custom SSL, what edge locations were they providing caching in, etc), but had Kreb been a normal paying customer, this could easily have been a over a million dollar bill (if it was sustained long enough to alter his 95th percentile bracket) in the cheapest case. If things like custom SSL are in the mix (which Akamai charges absurdly high prices for), or lots of traffic from more expensive POPs, or lack of already having pricing commiserate with high volume traffic commitments, the bill could've been 5-10x that amount or more.
It's kind of stupid to me that the massive and advanced cdn of akamai protect something as non-important as a blog against such a major ddos attack. If they were doing it pro-bono wouldn't the prudent action be to mitigate ddos's until a certain treshold and then actually assess the value of what you are protecting? A good lesson to have learned, I believe.
But no, they'll drop this client which had to have continually given good referrals.
It would be interesting if he started writing on Medium (not saying technically advisable, just interesting). I wonder if he'd ever consider trying that.
The DHT can still be attacked. Here's three methods:
1: Take out the bootstrap nodes. These are several nodes that bootstrap a new client into the DHT system. BitTorrent, Inc. keeps a couple such nodes. On first boot, the client registers it's DHT address and collects a few from the bootstrapping node. The client could then can traverse the network itself. By knocking out these nodes, newly started clients now have to browse the whole IP space for possible DHT clients, which is not feasible.
2: Attack the peers themselves. A malicious program could traverse the network searching for DHT peers in the same way. At first, it would only collect a large number of DHT addresses and their corresponding nodes. Once a sufficient mass is gained, each is targeted with a low level DDOS to knock them offline to further requests. Most of these peers will be homes and local ISPs, which can't effectively deal with DDOS traffic themselves. Others trying to connect to a down client will eventually remove them from their own address space for later queries.
3: Poison DHT peers. This is probably the hardest, but once complete could poison an entire network with a switch. On each of your compromised Bot machines, you make a valid DHT node. Make a LOT of these (like a Botnet). For the most part, participate correctly with the DHT network. Collect as many valid/real DHT user and content addresses as you can and host them in your nodes. When it's time to attack, prevent these valid DHT addresses from resolving on inquiry. Even better, make them go in the wrong direction and infinitely pass around requests to other poisoned bots in your ring to prevent resolution but not hang the process. This is especially useful for content attacks because it attacks the content addresses themselves.
1a. A system could be made for bootstrap links that have the addresses of a few nodes in them.
1b. When you have 10 million nodes like torrents, you can go searching random IPs. As long as many nodes bind to the same ports.
2. Sure, if you have comparable bandwidth to the entire network you can take it down. But that's a lot harder than overwhelming a single target. Nobody can send 20mbps each to millions of IPs.
3. This is the method that takes the least resources, but pretty good countermeasures can be made.
You can say the same about any network. What you are describing is an attack on the WHOLE network (such as what Bruce Shneier recently reported), not a specific file. You can try to bring down the internet, but not a site.
Unbelievable, they enjoyed year of free publicity from association with him, and this is how they repay him. Its bad enough that they couldn't handle the attack, despite all the bragging about their multi-Tbps capacity...
Brian Krebs' wasn't a paying customer right? Akamai provided the service pro-bono. Perfectly acceptable for them to suspend service if it becomes more than trivial in terms of cost or it puts their paying customers at risk.
I've always wondered if your domain is under a http DDoS attack, couldn't you in theory update your DNS A record to another ip and take other servers down (maliciously)?
At this scale it must also cost a ton of money to carry out this attack, I wonder if there's a vulnerability that we don't know about that let them do this so easily?
You're almost never using machines you compromised. Instead, you're using someone else's compromised machines: a botnet-backed DDoS-ing service, which you rented on the open (black) market, at the going rate. Bigger attacks still cost more in such setups—whether or not the botnet nodes were free to acquire, the resulting network is still a scarce resource whose price rises with demand.
(Of course, in the very special case of Krebs, the people he is reporting on frequently are the owners of the botnets, who can of course use their own botnets freely.)
Yes, but they can be flooded as well. Every system is going to have a finite capacity. And if the capacity is exceeded, the system will slow. If the capacity is significantly exceeded, it will become unreachable itself.
So if the capacity of your system is X Gbps, then it will start to have problems if the attacker sends X + 1 Gbps. And will probably be completely unreachable if the attacker sends X * 2 Gbps.
Akamai were providing him hosting free of charge, but the size of the attack will have had not insignificant financial impact on them, and on their customers. It's completely understandable that they've chosen to terminate their hosting agreement at this time.
Well, if your site needs to be online and processing payments so your company can pay salaries and rent, a week (or month) long DDOS would basically end things for you... Everywhere I've worked, a DDOS like this would basically be an emergency.
tl;dr Akamai was hosting his site pro bono. His site was being DDOSed, which cost Akamai a ton of money, so they kicked him off since they were literally only losing money on the deal.
Let us not permit companies to co-opt language for their benefit.
If it was genuinely pro bono ( lit: for the public good ) then they would have taken all steps possible to keep the site online since the public good was served more by having Mr Krebs online than not.
However, in this case they were hosting him free-of-charge because it was good publicity for them. That's a very different scenario.
Besides, pro bono isn't literally "for the public good", it is literally "for good".
Finally - that is a ridiculous standard to hold everything categorized as "pro bono" to. Law firms oftentimes take on cases/clients that can't afford their services, pro bono. Because they call it pro bono, does that necessitate that said law firm should continue to fight all pro bono cases in court until either A. they win or B. they go bankrupt? Of course not.
Well, to be fair to the parent poster (and without going deeper into the merits of this side-argument), I'd note that the expression "pro bono" as generally used in English is actually short for "pro bono publico" which does in fact mean "for the public good".
Law firms (and other professional services firms) call it "pro bono" when they use their specific skill-set to provide their services to those (e.g. the indigent) who couldn't otherwise afford them.
In that example, it's the fact that the indigent can get access to quality legal representation which is itself considered the "public good".
I get that. My point about law firms is that most of them won't take pro bono cases all the way to the supreme court (or equivalent). i.e. there is a limit to how much manpower they are will to expend on a charity case.
In the same way, expecting Akamai to provide free service to Krebs until the end of time because it was referred to as pro bono (even if it was them, which it wasn't) would be silly.
tl;dr - Akamai provided a service that could be seen as publicly beneficial. As long as they were providing free service to Krebs, they were doing something that was arguably pro bono. Them no longer choosing to provide that service does not retroactively detract from its public benefit.
Why would it cost Akamai a ton of money? If they already have the infrastructure and automation in place to mitigate such attacks, it wouldn't be a lot of work and would not require additional resources to mitigate this attack? And as others have said: isn't this bad PR?
I think its time for some serious financial incentives for ISPs to start getting serious about routing (or rather not routing) garbage. Financial fines for every DOS originating from your AS, or blacklisting if you are a repeated offender.
There is an incentive: it is the cost of transit. However, there usually are not a lot of zombies per single ISP for the access level ISP to even see any abnormal traffic.
The best thing is that access ISPs need to implement BCP38 (https://tools.ietf.org/html/bcp38). And shutdown all open recursive DNS servers. It would be great if Microsoft didn't ship such a retarded DNS server too. I would say that most ISPs do not do this.
NTP really should be replaced with something better. There are still large numbers of NTP amplification attacks going on. The big issue with NTP today, is that by default ntpd in daemon mode, is also a NTP server and responds to NTP requests. And so many of the two bit home routers run ntpd.
But the reality is, that no one is even reporting DDoSes right now. I work at an ISP, and I haven't seen a DDoS report in the past year. We pro-actively scan for open DNS and open NTP services. But many DDoS attacks just use regular HTTP/HTTPS, are hard to detect at the individual network connection level. Do you think Akamai sent out a single notice to any ISPs, saying "The following X IPs are sending excessive traffic to site Y, and are suspected to be part of a botnet"?
