As always it’s easy to overlook the insider threat. Grammatically dubious extortion emai? Bitcoin ransom? Did it not occur to you that the cat was the one behind the attack?
We don’t have very many earthquakes in Kansas… but I remember the first/only one I felt.
I was sound asleep when my Siamese woke me up by pawing my face… he then went and sat on the edge of the bed and growled aggressively (very out of character)… Not 30s later, things started shaking.
No idea how he knew, but it was pretty wild. He passed away in 2020, still miss him.
Dang, sorry for your loss. That’s a dope memory of the cat tho. As someone that happened find themselves in Taipei a couple weeks ago for the 7.4, my only thought was getting back to my dog, whom I promised I would get back to. She was sorta freaking out before I left; either could sense me leaving, or sense the earthquake I was heading to…
There are two types of sound in rock. P and S waves. P waves are pressure waves and go faster. S waves go side to side and are a bit slower. So you cat was woken by a hiss from the P waves, which arrive a bit before the earthquake that you can feel.
I have been living in Japan for a while, and in my experience you can feel the earthquake coming a few seconds before the big shaking start. I don't know how to describe it, it is not a sound, more like a very slight movement.
Maybe the phone was silent but still flashing a screen? Mine does that in that mode.
At my first job we had a guy who could spot incidents coming on the monitoring dashboard before they happened. He never managed to explain or even understand what he was looking for and no-one else picked it up, but he would just see something that made him say things were odd, and most of the time we'd get an alert shortly after.
Make or get a human to stare at streams long enough and they’ll attune to the patterns. We’re wired for patterns. It doesn’t even have to be conscious and explainable. The signals just suddenly aren’t right.
My s/o is convinced she has a tell when we play rock / paper / scissors, and maybe she does, but if so I don’t know it. I just know that if I observe her closely and don’t make a conscious decision about what to throw, I win 80% of the time.
Somewhere deep in my brain, there are neurons that developed for some more evolutionarily-relevant purpose and which are now a little disgusted with how they’re being used.
Maybe some signals just bypass the part of the brain which deals with well defined facts. I read somewhere about a construction foreman (HN comment maybe?) who gained the respect of the crew by having an unusually good hit rate in finding piping in the ground or walls. He started to believe in his superpowers but later came to the conclusion that he just subconsciously learned the typical patterns, plus an occasional non-obvious sign. Something like a vent pipe in the wall of the building telling you that sewer piping is probably below ground.
happened to me when playing cyberpunk 2077 on psf ataunch (shudder).
I got to a point where I could reliably tell "the game is about to crash, better save." I save, and 10 seconds after resuming the game, it crashes. I still don't know how I could tell.
The princely sum of $5,000. We got that at my employer back in 2016. We got hit by a ddos, and decided to ignore it, though we did dig up some BTC just in case. We enacted a bunch of DDoS protection as a result, costing way more than $5,000, but not paying money to extortionists is worth every penny.
About 20 years ago I was kinda accidentally the guy who dealt with the DDoS attacks in the sysadmin team. There was a sequence of extortion emails during about 2 week period:
1. $50k or we attack - didn’t register anything
2. $25k or else - a minor overload on the server but nothing serious.
3. $10k or else - a serious attack which affected the service in a major way.
4. $5k or we really pissed - this time they took down a whole Tier2 ISP and Datacenter in London for a day. Other carriers peering on London Internet Exchange had to blackhole traffic to our service provider and finally kept blackholing one of our IPs for a while. I had to scramble to find a DDoS mitigation service, new DC and servers.
We did not respond to any of the emails. The attackers were also quite dumb, they attacked the web servers which were located in a well connected place.
The money making service of the business was in the Caribbean with a 1,5Mbps T1 and a 0,5Mbps satellite backup. They could have saturated those much easier for much longer and the impact then would have been about $1M revenue loss per hour.
The problem with paying extortion or ransoms is that you incentivize the attacker to come back and do it again. It may have been $5k to pay off one attacker and more than that to build the defense, but now you have defenses and are less likely to suffer attackers in the future. And as you say, not paying money to criminals is inherently worthwhile.
From the german chaos computer clubs yearly meeting.
Linus talks about what to do and who ransoms work, how "well" the service is and briefly pros and cons of paying.
On the other hand the attacker may actually have incentive to follow through and hold up their end so as to build a reputation. Making their next victims more likely to just pay.
Somewhere I read that some ransomware had excellent "customer" service for helping you transfer over the payment and promptly restore your files.
This stat includes everyone, some of them may not research what hit them. One of my clients decrypted his database twice, after seeing on the internet that they actually send a decryption key.
Yep, I know it could only invite more trouble. It was fun to fantasize about though, especially after watching the video I linked to in the post: https://www.youtube.com/watch?v=dWzz3NeDz3E
Quite some time ago, someone from my family was alerted by their cat when the dishwasher was leaking. Their conclusion was that the cat was either trying to save them or the cat was trying to kill them.
One of my all-time favorite novels, Anansi Boys by Neil Gaiman, includes an anecdote: a crow's call wakes up someone who's sleeping outdoors, just as a large cat (a tiger, perhaps) is sneaking up on him.
One character suggests the crow was trying to warn the man. Another posits the bird was bringing the sleeper to the tiger's attention so it could enjoy the scraps after the meal.
Odd that the most obvious hypothesis wasn't given: the crow was probably using its alarm call to warn other crows. Crows are social animals, who care about the fates of their "friends and acquaintances" — so they would do that.
But also, on a tangent, there is a bird that does this kind of non-conspecific alarm calling the time as part of its food-gathering strategy: the African fork-tailed drongo.
The drongo gives true alarm calls to food-rival species nearby, to tell them when it has spotted a mutual predator. This leads to these food-rival species coming to rely on these signals. But then, every once in a while, it gives a false alarm, to get the food-rivals to run away for a bit, so it can nab the bugs/berries/etc that the rival would have been eating.
There's also an African bird whose foraging strategy includes alerting large mammals to the presence of food that they can harvest and the bird can scavenge.
In fact, there is a family of such birds, the honeywarblers, who locate beehives, then find humans and lead them to the beehives.
Wikipedia says that the behavior is dying out because there aren't enough human foragers.
There is another Neil Gaiman short story about a cat who goes gallivanting every night and comes back every morning all scuffed up. The owners don't know why since there aren't any other cats around. One night the owner can't sleep and discovers the reason.
Don't want to spoil. It's nice and short and a must-read for cat lovers. "The Price".
On a more serious note, do you think there will ever be a way to stop ddos attacks once and for all?
While all threats are bad, ddos is the most lame type of attacks there is; no special skill or knowledge are needed, just load a script or, heck, pay someone who'll execute it for you as a service.
It's not as simple as "loading a script" - IP addresses (or in the case of IPv6, subnets) are (for the average person) a limited resource, as is bandwidth, and most amplification attacks require IP spoofing which is not possible from most connections.
If it's a volumetric attack, the side with more bandwidth wins (the attacker may be able to amplify here). If it's a load-based/application-level attack, blocking the attacker IPs at the firewall level solves it. This was application level, not (purely) volumetric, since they already had a WAF/Cloudfront.
Identifying attacker IPs to block is a matter of correctly attributing cost to a source IP, correctly attributing benefit (i.e. legit user activity) to a source IP, then blocking the IPs or ranges where the cost significantly exceeds the benefit you see from that IP or range.
That's easier said than done, since cost can come in many forms (e.g. open connections clogging up memory, TLS handshakes, requests that are expensive to parse for your web server, requests that trigger expensive database queries, in/out bandwidth, ...) which is why most just slap Cloudflare (or here, Cloudfront) in front of it and work around with manual rules like in this example.
There's application level DDoS, which you generally stop by not doing expensive work for clients that haven't done expensive work for you. Sometimes, easier said than done.
And then there's volumetric DDoS. You can stop this by having more bandwidth than everyone else... but that's pretty hard and it makes you a potential attacker.
Innovation here is in the form of using BGP to disseminate traffic filters. Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible. But I've seen there's some systems with more precision, like drop udp, drop fragments, drop packets to/from udp/tcp port X.
Most of these systems are designed so that these specialized routes don't propagate beyond immediate peers, but potentially, it might be desirable if they did.
> Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible.
Oh, this poor guy is being DDoS'd, so we're going to make sure that their service remains denied.
Null-routing the target IP helps everybody except the customer who is being attacked: namely, the network operator and their other customers. From the victim's point of view, it's just as frustrating as the attack itself, and gets in the way of troubleshooting.
With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.
> Null-routing the target IP helps everybody except the customer who is being attacked: namely, the network operator and their other customers. From the victim's point of view, it's just as frustrating as the attack itself, and gets in the way of troubleshooting.
If you're running on a single IP, yes. If you're running on multiple IPs, it's not that bad for the one that's being attacked to get its traffic dropped and everything else works. It's not great, but what are you going to do. If you've got enough traffic to overwhelm the inbound on the top of rack switch your box is on, you're not going to be able to really serve any of the good traffic anyway.
> With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.
There's usually way too many source addresses to do that, and anyway, routing infrastructure is geared towards looking at destination addresses, not source addresses. Also, each individual source doesn't look that bad --- if I've got 10,000 sources each sending me 1 Mbps of garbage, nobody is going to accept a block for only 1 mbps of sending, and yet, there's 10 Gbps of garbage arriving at my box; if I've got 10 Gbps or better connectivity, no big deal. But, if I'm only on 1 Gbps, I'm getting less than 1 in 10 of my inbound packets. I'd argue, if everything else has a big enough connection, it's probably still no big deal, it should be able to drop packets headed to me, as long as its upstream connection isn't filling up. But once abuse is causing contention that impacts others on my rack, it's probably time to null route.
If it's one of the big botnets with 100,000+ compromised systems, the individual bandwidth is even less. And if the botnet has significant ability to deliver spoofed traffic, source based filtering is meaningless. If it's reflected DDoS, I dunno --- there's value in hunting down the chargen services and removing them from the internet, but that's usually a lot more work.
OTOH, look on the bright side, if your outbound bandwidth is high and you get a lot of inbound DDoS, you may have roughly balanced your usage, and you may qualify for settlement free peering! (IMHO, this has got to be a major part of Cloudflare's business plan)
So the suggestion here is to implement source routing on a whim, with a table of 100+k entries? At your peering point?
An ISP has to do what it has to do to save their business when a large attack hurts their business. This may be what the attacker wants but that's not an excuse to do nothing.
But what machine learning has to do with this is not clear. Null routing protects you against traffic volume. Machine learning sounds like it maybe can help diagnose more sophisticated low-volume attacks. Maybe. You don't want anything remotely compute intensive when mitigating attacks.
Maybe if the network was much more distributed and lower bandwidth?
If most of your customers are in Mexico, Canada is DDoSing you, and the pipes between you and Canada start filling up as a result that isn’t a big problem, right? As long as consumer routers on you/Mexico’s side of the Canadian clog don’t decide to help out.
Writing this post did make me think that if someone had a well-trained dog, they could hook up a monitoring service to something that makes a particular sound, which tells the dog to alert the person.
Her name was (I sadly lost her to cancer) Bamboo! Because one of the first things she did after I adopted her was to try to eat my bamboo plant.
And as an added bonus, we could get that dog classified as a service dog :D
“Sir you need to leave that mutt outside!”
“He’s a service dog”
“Why? You don’t look like you have any disabilities”
“Wow. First of all – rude! Second, yeah you are right I don’t but you see he’s my DDoS dog and I need him with me at all times to protect the company servers”
Funny, that's how the very first customer realised that the Australian telco Optus was down. The wireless cat feeder relied on the Internet and when no food appeared, the cat decided to complain to management.
I think this would be like a firewall or ingress thing that would drop packets that resulted in excess load before they make it to the application server.
Tangential question that rose up regarding availability vs. quality of life.
For a small startup whose products are only available on the US, does it always make sense to do nightly oncall? This doesn't work for some products, but if, for example, you have a site that sells mattresses in the US, would you wake someone up to fix the site at 3AM?
I guess here the main $$ loss would come from accepting so much traffic. But I wonder if we can better differentiate what's worth waking up for.
> we didn’t have a formal on-call rotation yet. That was a deliberate decision, since being on-call is painful, and the team was good about just collectively keeping an eye out for urgent alerts.
That seems like a terrible solution. Yeah, being on-call is painful, but at least I know beforehand when I'll be on-call and get compensated for it. Always being expected to keep an eye out for urgent alerts just sucks all around.
I know it sounds bad, but in practice, it really did work fine for us for quite a while.
1. We didn't experience that many incidents that couldn't wait until working hours.
2. There was never an explicit expectation to keep an eye out. We did it anyway because we were at an early-stage startup, and we all deeply cared about making our products work for our customers.
I know this from a few startups and it really is not that bad. You really triage what should wake you up and what's ok until morning. It works well as long as the technical founder is ok playing a goalie and essentially being always on call (even though others catch a lot of alarms).
It stops working when the company grows and no one understands the whole system and you need on-calls from several teams. Then the company does some formal on call rotation and it's fine again. It hurts during the transition only.
2 European teams I worked on paid a bonus for on-call duty, and the systems were so stable that enough people volunteered for the few who didn't want it, weren't forced to.
It was pretty great, I took a week shift every month or so except when I was going on holiday, and aside from lugging a backpack with my laptop everywhere, didn't affect my life at all except 1 or 2 minor issues
Not really. If I don't agree to on-call, I do it on best-effort basis. That is: "Oh, I'm camping with kids without a computer. I'll try to help you as much as I can. Did you try Z after X and Y didn't work? Ok, try it and call me back how it went, I'll try to Google something in the meantime." If that would happen too often (more than 1-2x a year), I would try to improve the process or consider switching my job. And my phone is silent during the night.
I wouldn't call this a permanent on-call, just being responsive.
You usually get some extra money for the duty. And if you get woken up, the hours you spend are counted towards your normal working hours - so you aren't expected to show up in the morning after putting out a fire. Or you get some more bonus (like 2x hourly pay for the night work). That's about the balance when people are ok doing it.
But it depends on stability of your service. If it is messed up and people are woken up often, then you won't find many volunteers if they have other choice.
Cats love to hang out in warm areas, even sunbathe, so I doubt they'd do anything to get attention in that situation! Their body temperature is a few degrees warmer than that of humans.
We once detected a DDOS because all our office phones went down. Silly attackers didn’t realise that our (money-making) APIs weren’t colocated with our public website and phone system.
Kinda reads like one, but if he was on GCP and used their ddos shield then it'd read like an ad for their service instead. Would be better if he'd been a bit more abstract and said cloud provider instead of naming AWS.
It's translated by duckduckgo.com's chatgpt interface. don't down vote plz :D
I suspect that I am somewhat sensitive to electromagnetic fields and magnetic fields. There have been times when I have not felt well the next day after sleeping on an electric heating pad, and I have experienced severe discomfort after sleeping on a mattress with magnets.
When I used a CRT monitor, I often had diarrhea if I spent a long time in front of the monitor.
Since using LCD monitors or laptops, those symptoms have disappeared.
When I sleep, there is a wireless router on the right side of my head, and I play youtube videos on my smartphone on the left side. I have strange dreams and wake up early from sleep. However, if I put the smartphone on the right side of my head while sleeping, those symptoms are lessened.
Thus,
Even though there was no sound, wouldn't your cat have sensed that as well?
