> likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid).
Yeah, it isn't as if crowdstrike was specifically advertising certified support for RedHat Linux and related products.
Back in The Good Old Days, an OS vendor would release a beta version and software vendors would test against it and fix problems before the stable OS version was released.
Obviously OS updates come out a lot more often these days than they used to - but we're also better at test automation than ever before, and beta software is easier to get than ever.
It sure would be nice if companies that decide to produce kernel modules and to support certain OSes could test those kernel modules against those OSes at the beta stage.
1.An eBPF probe is not a kernel module. An eBPF probe should never cause kernel panics.
2. RHEL didn't provide beta kernels before very recently, as far as I can tell.
3. Even if you caught an error then, you're still at the mercy of RHEL to fix it. If RHEL breaks a feature, you report it to them, and they decide to ship anyways... well, your product will still kpanic. I'm not talking hypotheticals: I haven't seen RHEL do that, but I've seen other distros do it.
Should, but did. This is the point at which to learn and adapt.
Also, kernels are software just like nearly everything else, and software is buggy. It's a balance obviously, but some basic defensive development can be a real savior for your users.
I don't know the details about this CrowdStrike incident, but I would also be surprised if you couldn't write an automated test (even a "smoke test") to quickly test out these new kernels before they hit your customers. Given what happened, it seems like negligence not to do that.
It's possible CS can do better, of course. But it's just wrong to blame them for the Linux crashes - they're not the ones that introduced buggy code and broke their users. RHEL/Linux did.
1) Those Crowdstrike unit files aren't ebpf probes, so the whole subject of ebpf probes is irrelevant here; or
2) They're obviously able to stop the rest of the kernel from even booting up (as Crowdstrike so convincingly demonstrated millions of times over[1]), so yes, they do indeed have at least as much power as any other bit of the kernel.
Either way, hunting around for nits to pick is a bit pathetic.
denial of service is not the same thing as arbitrary code execution, and that goes double in kernel mode, but yes, it does seem that the linux implementation of ebpf had buggy sandboxing; i don't think allowing clownstrike to prevent booting was part of the intended objective
i wasn't hunting around for nits to pick; i was hunting around to see if you'd ever contributed any useful comments to the site. instead i found you making authoritative pronouncements about ebpf that were so wrong that you had evidently never read so much as a one-line summary of what ebpf was for. do you have a more promising historical comment to offer? perhaps something where people complimented your contribution as being informative?
have you ever made a worthwhile comment on hn?
on thursday, wahern posted this comment https://news.ycombinator.com/item?id=41061179 where they traced through the illumos/opensolaris source code to track down how a peculiar solaris interprocess communication mechanism worked, an investigation i had started but gotten stuck on. why can't you make comments like that instead of harassing me about how i format my comments?
the reason i'm asking is because i'd like to be able to talk to more people like wahern, but most of them avoid this site. a major reason why is that comments here frequently receive vacuous, aggressive responses like the comment you made the day before in https://news.ycombinator.com/item?id=41056718, where you launched a personal attack on me because you didn't like how i was formatting my comments
i'd like you to ⓐ apologize for doing that (this is not the first time you've done that to me personally; so far i haven't looked through your comment history far enough to find out how many other people you have a history of repeatedly harassing) and ⓑ commit to not doing it again
because i'm sure you're capable of making comments that make the site better instead of worse
> do you have a more promising historical comment to offer? perhaps something where people complimented your contribution as being informative?
> have you ever made a worthwhile comment on hn?
I might answer that. If I thought you were owed any justifications from me. Which I don't.
And no, I'm neither “harassing” you nor being “vacuous, aggressive”. This isn't ad hominem, it's ad habitem. You write here for other people to read, and I'd even appreciate many of your comments -- if they weren't so infuriatingly idiosyncratically formatted as to disrupt fluent reading. Have the fucking courtesy to write like a normal person, and you'll be treated like a normal person. To begin with, get the shift key on your keyboard unstuck so you can start your sentences with capitals. And in case your dot / period / full-stop key is totally gone, copy-paste some of these: ........... So you can end them properly too.
Because I'm sure you're capable of making comments without coming off like an illiterate buffoon.
And yes, BTW, you totally were. Careful now, you don't want to end up like chockablock again, do you?
Yes, and? They probably do test their software on RHEL.
But how are they supposed to prevent a bug in a newly released kernel update? You can't test your software on future updates that aren't out yet.
If RHEL breaks some core functionality you depend on, in a newly released update, you can't really do much to prevent breakage, even with the best QA in the world. At best, they could have caught it as soon as RHEL published the new kernel... but by then it's already too late, all your currently-deployed probes now have a ticking time bomb, and need to be updated before the RHEL kernel update is applied, lest you kernel panic.
Maybe by not loading the module into unknown kernels in the first place?
If say you support a distro, you can’t turn around and complain that supporting the newest version is hard, no matter who caused the problem. Plenty of products say ‘this works on $x but it’s not officially supported’.
If you expect software to be future-bug-proof, well, I guess you live in a far better world than I do.
If you advertise your software to be compatible with RHEL, but a glibc bug gets in and causes your sw to crash for a couple of days, before RHEL realises the problem and fixes it, does that mean your software should instantly no longer be advertised as RHEL compatible? That'd make things a lot more confusing, if you ask me.
Not all eBPF programs are compile once, run everywhere.
RHEL 'updates' can mean different things. A patch release won't change kernel ABI. A minor release will. Writing a non-CORE eBPF program for, say RHEL 8.6, might break on RHEL 8.7. It's not advisable to update across minor releases without lots of testing. Most of the time, things 'just work' but RHEL is a very complex product with a specific support cycle, and laziness of users and 3rd party vendors is not their fault.
This was their newer eBPF falcon sensor that was trying to load a bpf program in the kernel and triggered kernel panic. This shouldn’t have happened and was definitely a bug in the kernel.
For the kernel mode, their software will flag an unknown kernel as unsupported and go into a reduced functionality mode (rfm).
The idiots didn’t know that RH E4S was a thing for like 3+ years.. I’m still baffled by how clueless most of the security people and vendors are when it comes to backporting and different streams / channels that are offered by multiple Linux OS vendors.
> Maybe by not loading the module into unknown kernels in the first place?
Then you better tell your customers not to `dnf update` until you've had a chance to whitelist the new kernel and ship it in your own stream. Otherwise everyone who updates before you do ends up broken. If a vendor told me that, I would laugh, realize they were serious, thank them for their time but let them know that we will be going a different direction.
Yeah, it isn't as if crowdstrike was specifically advertising certified support for RedHat Linux and related products.
https://www.crowdstrike.com/partners/falcon-for-red-hat/