This rollercoaster is not over yet. There's a crowdstrike issue causing windows machines/servers to brick globally and this industry is heavily windows dependent. It may or may not be related to the Azure issue but it's suspicious to me.
Almost certainly Azure using Crowdstrike on Windows in one way or another.
Not surprising that AWS and GCP don't seem to be hit as they wouldn't run anything on Windows, unlike Azure, who I'm sure are forced to do so under MS' infamous interdepartmental structure.
I had to physically stand in a queue for about 8 hours for a Ryanair customer support desk in an airport when the airport runway was closed by 1-2cm of snow.
I forget the exact timing and can't be bothered to look up my notes, but it was something like 11pm to 7am at the origin airport for a flight that was supposed to have landed at the destination around 8pm, as we were also stuck on the runway for an hour or so and even getting that far had been delayed.
The replacement flight the next day was also cancelled even though the airport was open.
I ended up taking a ferry and a train, and that was still simultaneously faster than the next available Ryanair replacement flight and cheaper than any other provider on short notice. Fortunately I had an understanding boss who didn't mind me arriving 4 days later than expected, and also a place to crash for free while working out the best route home.
I have never heard of crowdstrike. Is that some kind of antivirus? How is that related to PCs not booting? And why does it affect so many PCs if I've never heard of it? I'm so confused
It's enterprise anti-malware that [in addition to other bits] has a client component installed on all PCs in the corporate network. An update to that client component (called an "endpoint") is causing those Windows machines to BSOD.
It's unlikely you'd have heard of it unless you've worked at a large enterprise that runs primarily Microsoft IT.
Crowdstrike does have Mac/Linux "endpoints" also (IIRC) but I'm unsure if they're affected as well.
Windows complains about some page fault or something in a file name csagent.sys. On my machine this file hasn't changed in several days, but the issue only happened this morning like for everyone else.
This looks suspiciosly a case of "let's download random crap from the web and run it in kernel space. what could possibly go wrong?"
I've never seen a non-Windows machine tbh. But our IT just send out an update that we don't use crowdstrike. Strange that I never heard of it if it's so widespread. But thanks
You'll see this software more in highly regulated areas. Think Government, finance, travel. It exists mainly to check a compliance box.
The Windows claim is a little misleading. We used Linux where I last encountered this. I expect Windows is where problems are manifesting this time; BSOD and kernel panics with this aren't new!
CrowdStrike seemingly came out of nowhere but has existed for a while... I think it's suspicious.
Have we not learned from SolarWinds and company? The vendors become part of your posture. Consolidating far too much
They make malware that steals funds from corporations (willingly!) so these corps can tick a security checkbox for some certification investors have been told is paramount; it's just disguised as security software.
On the wireless they are reporting a bad Crowdstrike update and a major Azure failover in central USA as separate events, are they they the same or different?
A whole lot of people are running Crowdstrike in the cloud and on local PC
An crowdstrike update last night caused a windows kernel panic Azure/Crowdstrike personel have spend rolled back the update in the cloud
Local IT people will have to revert it from local machines manually
Yeah, you're completely out of touch of the industry. As horrendous as it is Windows on Azure (and the entire ecosystem it comes with) is an easy sell for the dinosaur IT leaders that haven't seen a Linux server in their life. You'd be surprised how common that is.
Unless attacks take down infrastructure regularly, we won't go back to decentralized model. The internet itself was created decentralized to withstand a a war .
No, the ‘internet’ was created to allow US defence researchers to access shared computers more easily. It adopted the packet switched model that had been developed theoretically to support command and control applications, but which was never actually implemented by the extant C2 providers (in a dentist’s waiting room now so don’t have links)
I dunno, I have been around and I never seen an on-prem infra being more reliable than your average cloud.
The only difference is that when on-prem goes down you can shout to your infra engineers, when cloud goes down you shout at your enterprise cloud representative. The first is more effective, but even with that it still doesn't achieve the same reliability and disaster-recovery of your average cloud provider.
It’s a general political trend too: de-globalization. Everyone sold the idea of globalisation and off-premise ethereal globalized cloud services. Both good when they work, but a total disaster when they don’t.
There’s a reason that I admire the federal system of the US:
For all the US’s problems, devolving critical functions to layers of differing granularity has proven surprisingly robust to many faults.
I suspect we’ll see economic equivalents, where critical functions are spread around at various scales. We dont need to be either totally globalized or totally domestic.
Do you admire federated system of Germany too? They famously have many department operating semi-autonomous, causing immense friction in adopting any new changes. Do you admire federated system EU? Where countries can run unchecked for years and it is very hard to fix issues in any specific member country.
I'm not criticizing any country or union here, and not praising them. I merely highlighting that maybe federation on its own is not the main cause of success of USA and there are some other more important factors at play.
Except that anything local will be more expensive than anything globally scaled, and the same people complaining about globalization "suddenly" don't want to pay more for the same.
It's not like alternatives for MS services doesn't exist for decades. There smaller and more people friendly hostings, email services, file shares, office software etc. The problem is that people complaining about MS services don't wan't to use them or pay for them.
I wish. Only (mostly small) tech savvy companies might maybe make that move at some point. Herd mentality and short term convenience have already won that battle. AI will only add to that since for 99.9% of people LLMs and cloud are synonyms.
The corporate training always says you are responsible for your chosen cloud vendor's problems, but in reality everyone assumes they can successfully blame the vendor.
Well, the older version is about IBM, and it would probably even be true (today) if we were talking about mainframes, because they are one hell of a stable basket ;)
A side issue, but since we’re on the subject of global tech being generally fscked: I’m currently on holiday in Italy and just discovered the entire archive.ph domain is blocked by the government, apparently due to kiddie porn. Shrug emoji…
There was a law passed that allows ISPs to block sites that host copyrighted content illegally. It's not just Italy, also the Netherlands and many other countries. You can still access it with a vpn or tor. Tor browser on the phone works fine for mobile carriers that block the sites.
Yes that seems to be the case - the blocking page is headed with "Ministero dell'Interno -
Dipartimento della Pubblica Sicurezza - Direzione Centrale per la Polizia Scientifica e la Sicurezza Cibernetica - Servizio Polizia Postale e Sicurezza Cibernetica" along with official looking government logos but text underneath (in Italian and English) talks about a collaboration between govt agencies and ISPs. From mobile cell service it was not blocked.
The text does however mention it's a specific measure against child pornography, not re-hosting copyright content.
I wonder if these airlines were really affected by that Azure problem or if they were affected by the CrowdStrike issue and were just mixing things up.
Waiting in a queue at an airport in Palma, Mallorca, Spain right now and the check in staff are currently flipping through printed sheets of paper to check us all in. It's going to be a very long wait.
The article is pay walled. Seems like this would be the fault of the airlines though. There is a reason to be distributed between different geographic areas.
But if the Azure outage is due to Windows machines crashing because of the currently ongoing CrowdStrike crash/reboot loop issue, then such servers might end up being down in all regions. Looks like there might be some advanced lessons to be learned about blast radius here...
Maybe because Windows Defender Advanced Threat Protection is an enormous resource hog that scans every byte of memory and storage accessed by the Hypervisor and performs a quadratic time computation on the data? I am just guessing because my “fastest” Windows laptop CPU money could buy feels like a hot smelting furnace and a sloth at the same time when I use VMWare Workstation. What the &$@* is it scanning the VMWare guests for?
More likely crash looping of so many VMs overloading some system with insufficient back pressure, possibly combined with unfortunate cluster management scheduler behavior at this scale of crash looping (e.g. too eager to retry scheduling instances, maybe even on new hosts which causes more infrastructure load).
VM storage is probably on Windows Server, plus AD. I'd bet out of band management is all in the impact zone too. Might be back to someone pushing physical switches and hooking up a KVM.
Interesting that I happened to transit 3 different Asian airports today and had zero issues. I haven't seen anything at all related to the outage over here.
I'm so old that I was like "Do that many people really play Crowdstrike?" then I realised that's Counter Strike, then I looked up Counter Strike and it came out 24 years ago.
https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...