Hacker News new | past | comments | ask | show | jobs | submit login

> How Do We Prevent This From Happening Again?

> Software Resiliency and Testing

> * Improve Rapid Response Content testing by using testing types such as:

> * Local developer testing

So no one actually tested the changes before deploying?!




And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.


> This makes them look like absolute amateurs.

This applies also to all Architects and CTO's at all these Fortune 500 companies, who allowed these self updating systems into their critical systems.

I would offer a copy of Antifragile to each of these teams: https://en.wikipedia.org/wiki/Antifragile_(book)

"Every captain goes down with every ship"


Architects likely do not have a choice. These things are driven by auditors and requirements for things like insurance or PCI and it’s expensive to protest those. I know people who’ve gone full serverless just to lop off the branches of the audit tree about general purpose server operating systems, and now I’m wondering whether anyone is thinking about iOS/ChromeOS for the same reason.

The more successful path here is probably demanding proof of a decent SDLC, use of memory-safe languages, etc. in contract language.


> Architects likely do not have a choice.

Architects don't have a choice, CTO are well paid to golf with the CEO and delegate to their teams, Auditors just audit but are not involved with the technical implementations, Developers just develop according to the Spec, and Security team just are a pain in the ass. Nobody owns it...

Everybody get's well paid, and at the end we have to get lessons learned...It's a s*&^&t show...


Some industries are forced by regulation or liability to have something like crowdstrike deployed on their systems. And crowdstrike doesn't have a lot of alternatives that tick as many checkboxes and are as widely recognized.


Please give me an example of that specific regulation.


There's a whole body of regulation around service providers to the U.S. Government making it an effective requirement to use this stuff, starting with the FedRAMP Authorization Act (https://www.congress.gov/117/bills/hr7776/BILLS-117hr7776enr...).

See also Section 4.2.4 of the FedRAMP Moderate Readiness Assessment Report (RAR) which can be found here: https://www.fedramp.gov/documents-templates/ as an example.

You cannot obtain an Authorization To Operate (ATO) unless you've satisfied the Assessor that you're in compliance.


PCI DSS v4.0 Requirements 5 and 6 speaks very broadly for anti-malware controls, which Crowdstrike provides as EDR, and cybersecurity (liability, ransomware, etc) insurance absolutely requires it from the questionnaires I’ve completed and am required to attest to.

> In its first version, PCI DSS included controls for detecting, removing, blocking, and containing malicious code (malware). Until version 3.2.1, these controls were generically referred to as "anti-virus software", which was incorrect technically because they protect not just against viruses, but also against other known malware variants (worms, trojans, ransomware, spyware, rootkits, adware, backdoors, etc.). As a result, the term "antimalware" is now used not only to refer to viruses, but also to all other types of malicious code, more in line with the requirement's objectives.

> To avoid the ambiguities seen in previous versions of the standard about which operating systems should have an anti-malware solution installed and which should not, a more operational approach has been chosen: the entity should perform a periodic assessment to determine which system components should require an anti-malware solution. All other assets that are determined not to be affected by malware should be included in a list (req. 5.2.3).

> Updates of the anti-malware solution must be performed automatically (req. 5.3.1).

> Finally, the term "real-time scanning" is explicitly included for the anti-malware solution (this is a type of persistent, continuous scanning where a scan for security risks is performed every time a file is received, opened, downloaded, copied or modified). Previously, there was a reference to the fact that anti-malware mechanisms should be actively running, which gave rise to different interpretations.

> Continuous behavioral analysis of systems or processes is incorporated as an accepted anti-malware solution scanning method, as an alternative to traditional periodic (scheduled and on-demand) and real-time (on-access) scans (req. 5.3.2).

https://www.advantio.com/blog/analysis-of-pci-dss-v4.0-part-...


Besides things like FedRAMP mentioned in other comments, some large enterprise customers, especially banks, require terms in the contract stating the vendor uses some form of anti-malware software.


Seems like everyone thinks that Execs play golf with another Execs to seal the deal regardless how b0rken the system is.

That CTO's job is on the line if the system can't meet the requirement, more so if the system is fucked.

To think that every CTO is dumbass is like saying "everyone is stupid, except me, of course"


Not all CTO...but you just saw hundreds of companies, who could do better....


That is true, hundred companies have no backup process in place :D


They don't care, CI/CD, like QA, is considered a cost center for some of these companies. The cheapest thing for them is to offload the burden of testing every configuration onto the developer, who is also going to be tasked with shipping as quickly as possible or getting canned.

Claw back executive pay, stock, and bonuses imo and you'll see funded QA and CI teams.


It sure sounds like the "Content Validator" they mention is a form of CI/CD. The problem is that it passed that validation, but was capable of failing in reality.


The content validator is a form of validation done in CI. Their CD pipeline is the bigger problem here: it was extremely reckless given the system it was used in (configuring millions of customer machines in unknown environments). A CD pipeline for a tiny startup's email service can just deploy straight away. Crowdstrike (as they finally realized) need a CD pipeline with much more rigorous validation.


The fact that they even listed "local developer testing" is pretty weird.

That is just part of the basic process and is hardly the thing that ensures a problem like this doesn't happen.


This also becomes a security issue at some point. If these updates can go in untested, what's to stop a rogue employee from deliberately pushing a malicious update?

I know insider threats are very hard to protect against in general but these companies must be the most juicy target for state actors. Imagine what you could do with kernel space code in emergency services, transport infrastructure and banks.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: