There shouldn't be any coming back from this. There are failures on multiple levels here & CircleCI demonstrated no one should keep any sensitive data with them.
> Zuber said that while customer data was encrypted, the cybercriminals also obtained the encryption keys able to decrypt customer data.
...what? why did this engineer have access to everything? Does CircleCI know what minimum access policies are for?
At some point, someone somewhere has to have access to infrastructure, and be able to deploy it (if not, then generate a token of set of credentials that _can_).
Circleci's business model is hosting CI runners for you, so of course they need to be able to decrypt the data, and if you have access to dept new infrastructure, you likely have permission to read encryption keys (or deploy new infrastructure that can read said keys and then use those machines to get the keys).
What CircleCI _have_ done is set up enough logging and auditing that they were able to figure out who was compromised, how they were compromised, the time frame and the resources they accessed, which IMO is about as much as you can ask for.
> At some point, someone somewhere has to have access to infrastructure
They could run everything in Nitro Enclaves or similar, that require multiple people to deterministically compile and sign new software for, and release secrets into.
I design quorum controlled infrastructure for a living, mostly in fintech where no single human can ever be trusted. You 100% can run infrastructure that, barring a platform 0day, can prevent any single human from having access to the memory of customer workloads and secrets. Customers likewise would encrypt any secrets or code directly to keys that only exist in the enclaves.
CircleCI had negligent security design, but all its competitors are just as bad, to be fair.
Building with security in mind makes you last to market, which is unforgivable in our industry. Getting hacked however is just considered a cost of doing business.
There is no evidence GitHub has any multi-party accountability for sysadmins or enclaves for secret management. You enter secrets into the GitHub Web UI in plaintext, which means at least some employees can access them in plaintext.
GitHub/NPM have historically failed to support supply chain integrity practices in their public offerings such as hardware anchored code signing, signed code reviews, reproducible builds, multi-party approvals, etc. It is reasonable to expect they are not doing any of that internally either.
Assume any secret you give GitHub will become public knowledge and act accordingly.
The good news is there is never a reason to trust a VCS or CI system with high value secrets. They should never ever need any power beyond running tests, accessing a test environment, or sending notifications.
CircleCI have sufficient logging, however, failed to do fraud analysis to detect irregular access. Fraud analysis can be basic and just have a threshold on the number of encryption keys accessed each day. For this particular employee, there will have been a spike.
They also point their finger at anti-virus for not detecting the malware. That is a lame excuses. Professional malware developers will check if their product goes undetected by the major anti-virus providers. Anti-virus should not be relied upon to protect against sophisticated attacks
They didn't blame the anti malware. In the blog post[0] all they say is
> This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software
That's not blaming that's telling people that their antivirus didn't detect it. If that wasn't there, people would be talking about why they didn't use X antivirus which would probably detect it.
> CircleCI have sufficient logging, however, failed to do fraud analysis to detect irregular access.
This is a significant move of the goalposts. Of course CircleCI messed up, but to go back to the OP's point of "nobody should have that much access to production", well that's just not true.
I can't judge if their employees need that much access to production.
However, if they need this, you need a detection mechanism. It doesn't have to be advanced; a threshold on the number of production keys accessed per day is sufficient. It should raise the alarm to the manager in question, who can then confirm if this is expected behaviour given the tasks assigned to the employee in question.
Not necessarily. AWS engineers cannot access your encryption keys unless you give them explicit permissions. They even offer Nitro enclaves where AWS Engineers can never access your keys.
Giving access to customer data to your staff by default is a design decision and can be avoided. However, in the end, it is a cost-benefit analysis where you have to decide how much you care about your customer's security. I have worked for enough start-ups/growth companies to know the value put on customers' security is sometimes shockingly low.
Not all attacks are targeted and AV can detect abnormal behaviour. Regardless, having the operating system do proper sandboxing is much more valuable than trying to fix it after the fact with AV.
They can use hardware wallets for digital signing the operations, and keep encryption keys offline, split the keys to make sure that there's no 1 person that can access it.
There are lots of tools for key management, but lots of companies don't care about it.
Unpopular opinion but I think whenever these 'hack' incidents happen, there should be a full disclosure and the company should be made to tell us what exactly was hacked right down to the nuts and blots, sql queries and server processes level.
There should be full transparency on this and all the open source eyeballs should be able to study and scrutinize this. If not, anyone will be able to get away with data theft tomorrow saying, "my server got hacked". What will cause them to not do it except some sense of personal ethics which is rapidly degrading these days?
That is not an unpopular opinion, people have been arguing on this forum for proper legislation around these types of security incidents and breaches for years.
Bridges tend to be owned by the government which in democratic counties gives an accounting to the public.
Private bridges, have owners who give an accounting to no one when they collapse. At best, if you have standing you can sue them and they will defend themselves by giving an accounting for why it’s not their fault or they did their best.
Those things need to be supervisioned.
FlyTAP has gotten their entire rewards DB (if not all their DB) hacked and if it were not for IHBP I would never know because the company never told me.
As far as I know FlyTAP never got a fine for ignoring its obligations.
GDPR is a good idea but there must be supervision for it being applied as defined.
Stuff is so complex nowadays that there's a ton of vectors an employee can use to eventually access almost anything on production systems. Unless you kneecap their ability to reach anything at all. Even if say you use Vault for secrets, extensive ACLs, on-demand access, there's an almost infinite amount of ways to squeeze out data in ways it hasn't been anticipated. Kuberneters, containers, etc doesn't always help, often sometimes it makes things worse.
Defending against internal threat seems like a losing battle that can only be mitigated, slowing down more than preventing an attack.
Stopping employee from accessing anything and everything in production impacts productivity and engineer often do not want to work in that kind of environment.
Implementing a zero-trust architecture with a trust-score system for users and a dynamic policy for accessing resources can help to limit potential damage in the event of a security incident. But I agree that the balance between protecting against attacks and maintaining productivity can be delicate.
It was easy to take an existing open source terraform module [0], modify it a little for our purposes, and deploy runners that provision job executors in our tightly controlled VPC. It is very simple and everything is open source so you can really understand what it is doing. In our setup, all secrets are issued from our private vault instances. Most of them are short lived or one time use when possible. We are looking at moving our stuff to EKS now as well.
Overall, it was pretty easy to get going but we have the resources to do this. I could see why a small startup would outsource this to someone like Circle CI.
I second GitLab runner, used it a lot on bare metal CI servers, because we have some really exotic hardware requirements. Happy customer for 3 years now, and the CI part costs us nothing.
IMO, CI as the product companies are done for, to some extent they built a business on GitHub not having CI built in, and now it does. (GitLab already did, and when have you seen a GitLab-hosted project using CircleCI or similar?)
The more Actions matures, the harder it is for them to continue - their market will be legacy projects already on the platform, and perhaps the new projects from companies already using it for older projects (the one reason I can think of to prefer Circle over Actions for a new GH-hosted project).
Buildkite[1]. You never give them any secrets. It is essentially a frontend for agents that live on your infrastructure in the cloud, e.g. AWS. The company I work for does the CI runs on spot instances which is perfect for this kind of interruptible work.
We're on Google Cloud and pretty okay with Google Cloud Build - not as good and powerful as Buildkite but does not require us to run/maintain our own agents and one less vendor to manage.
Honestly, while there are some difference, the reduced integration overhead of using the offering from either your main cloud platform or your source code host probably wins.
Disclaimer: I work for AWS in Professional Services. All opinions are my own.
The beauty about CodeBuild is that there is no “lock-in”. All it is fundamentally is a Linux or Windows Docker container with popular language runtimes and a shell script that processes a yaml file or you can supply your own Docker container.
You just put a bunch of bash commands or PowerShell commands in the yaml file and it runs anything.
The Docker container and the shell scripts are all open source and you can quite easily run them locally.
I could see outside of AWS keeping your Docker containers for your specific build environments in a local repository and doing all of your builds inside them using Jenkins using a self hosted CodeBuild like environment.
For AWS, if you use CodeCommit (AWS git service), all access is via IAM and granular permissions. If you integrate with Azure DevOps, the AWS credentials do have to be stored in a separate MS hosted credential storage.
CodeBuild also supports at least Github natively.
I’m not shilling for AWS. I have an MS development background (.Net) and only have “DevOps” experience using AWS and Microsoft tooling.
> There shouldn't be any coming back from this. There are failures on multiple levels here & CircleCI demonstrated no one should keep any sensitive data with them.
Are other companies any better? It seems like now it’s assumed that all of these companies will eventually be hacked, so if you use them you need to have systems in place to mitigate damage. And if you don’t use them them then you need to have mitigation strats anyway. Basically either way you’re screwed.
Not necessarily. Yes, it can be hard to fight off dedicated hackers, but it shouldn’t be that easy. I know it’s easy to judge them when we’re not in their shoes, but one compromised employee shouldn’t be able to produce these results. There wasn’t even proper key management protocols in place here. I can set up a production-ready Vault (HashiCorp) cluster (with decentralized key shares) in a weekend and exercise better OPSEC than them. That’s actually disappointing.
Using Vault isn't going to make this impossible. The secrets still end up somewhere on a host in an env var. People still get permission at some point to read from Vault. Combine enough parts of the puzzle and you can find cracks in the system.
From the outside, it looks like many key infrastructure operations are running things like they are still startups, moving fast and breaking things. No access controls, engineers with machines they can take anywhere and install anything on, with full access to everything. Can't get in the way of moving fast, after all.
Developer laptops are the next logical vector from zero trust networks backed by SSO. Reading between the lines it seems like the session was an SSO token. I think you're right about giving too broad of access. A lot of companies I've been at assign you to LDAP groups over time and if you've been there a while you'll belong to a lot of LDAP groups. There is no ritual to minifying access that I've ever been made aware of. We definitely need some developer oriented antivirus software, because moving to a life where every bit of software needs to be approved would be a literal nightmare.
"some" means "all" from Australia's recent intrusions (Optus, Telstra). They can say "some" when the people at the top responsible for reporting only want a sample. It's PR.
I honestly can't think of a worse result of a hack of a CI/CD service than source code of companies being stolen. In my mind, this is akin to the Okta breach a while back, a ton of companies being hit hard through no fault of their own all at once.
I can appreciate the want to diversify services so that secrets/env are separate from code, but I think I would honestly trust the behemoth that is Github with both.
That being said, my company still uses Okta, so freebies and mulligans are certainly still tolerated when it comes to data breaches.
I mean, it was preventable, you just had to not use the massive single point of failure that is a cloud CI app. But everything is "so much easier" and "so much cheaper" with the cloud I guess!
Having experienced cloud CI on several platforms as well as in-house installs of both Travis and Jenkins, I think the ratio of downtime between cloud and local makes the cloud a far, far better pick.
That said, I am currently very glad I'm not running anything on circle ci...
Hindsight is 20/20 for sure, but for this use case confidentiality seems more important than availability.It is something of an impossible choice, but I've come to think that every service big or small gets hacked sooner or later. I'm thinking being a smaller target is better in that case.
I’m fairly confident that in the same breach scenario (laptop with admin permissions taken over), most small orgs would fare worse. The CI system would likely be behind a VPN, but the laptop would likely have those credentials, so it would not stop an attacker.
A small, 20 person org has maybe 2 people assigned to ops, so monitoring and breach detection is likely worse.
Now, a small org may be a less attractive target and some orgs can have top notch security people, but on average, the trade-off is likely not in favor of hosting your own.
I had the opposite experience with a large UK corp when moving from self-hosted Gitlab to gitlab.com.
Runners were still self hosted, but if the thing controlling them is just giving 500s all day and you’ve no influence on fixing it, then your jobs aren’t being run and your developers are sitting somewhat idle.
Github Actions has been better, but not perfect… but you can still fully self-host this if you think it’s worth it !
The difference, hopefully, is that private Jenkins installation is behind corporate network. So for attackers to reach it, either they need to breach the network first or it’s an internal employee.
Personally i’m a big proponent of both cloud services and privately managed services. CI is one of those i think are better kept private due to its sensitivity.
> So for attackers to reach it, either they need to breach the network first or it’s an internal employee
So for instance an employee getting their laptop infected with malware, or through phishing, or through one of the many vulnerabilities discovered regularly on enterprise VPN software?
There is a reason that many organisations are going away with VPNs and "corporate networks" all together - it gives a false sense of security that stuff behind it is protected by the VPN and leads to poor security practices inside like obsolete Jenkins installs.
Disclaimer: I work at a company that sells Zero Trust as a concept and associated software, but I've had this opinion since before joining (I've seen enough of 'there's a VPN and then lots of apps/services/servers with very poor auth practices because it's behind the VPN, why bother?')
As someone from the security domain you should know there is not one single tool/service that will solve all your security challenges. I did not claim VPN will.
Your example of employee laptop getting infected with malware should be remediated by proper device management. Which should include policies like forced updates, prevents software install, uses corporate firewall, up to date antivirus, etc.
Since you mentioned you work for a company that sells zero trust, your response sounds like your solution would replace VPN. Don’t get me wrong, but feels like you are wearing the sales hat now.
Yes, old fashion VPN appliances will be probably seize to exist. Replaced by modern equivalent like tailscale. Layered security is the answer.
But if you look at the recent hacks of auto companies, it was all driven by bad web apps, badly implemented SSO, bad REST APIs etc and they could get control of the cars because it was all public on the internet. A properly implemented VPN would have killed their ability to do those hacks dead. A VPN concentrates an encryption and auth system in one place, which is hopefully somewhat competent. The zero trust concept assumes every app is a web app, and every web app is secure enough to be exposed to anyone. Big assumptions to make.
I've never seen a Jenkins instance that did not need to be open to the public internet in order to support GitHub webhooks. Granted they were using IP based restrictions but it's still not behind a corporate network by any stretch in that setup.
Though I believe the almighty cloud is overrated, not all cloud providers are equal. As much as I dislike Google, they would not have messed up this badly. Ditto Microsoft with GitHub.
I think you’re over-estimating the amount of resources and expertise maintaining your own CI infrastructure requires.
The places I typically work for don’t have the operations capacity necessary. Nor do the small development teams have anyone other than me familiar enough with system administration to set things up in a reasonable amount of time.
A lot of junior developers have little to no Linux server experience. And I don’t really count following a step-by-step tutorial as experience. I have that experience and server problems can easily eat days of time the team can’t afford. :(
Laptop security aside (this is a hard problem and good solutions can often be detrimental in other ways) there should have been way, way more auditing around access to customer repos. The fact that it took so long to both mitigate further access and to understand the rough scope of the hack is concerning.
More broadly... it shouldn't be that easy to get encryption keys to everyone's secret env variables used for CI jobs.
Seems like everyone gets hacked eventually. Like, I'm sure CircleCI had security experts they hired. I don't doubt that they took things seriously and made sure they followed best practices. But that's not good enough. You will still get hacked. What do we do about this?
You are absolutely right. The sophistication of the hack implies, you will never know how the attackers will get you. What an organisation can do is, firstly do better disclosures. Secondly, let's the end users know what is risk they have when they add keys, credentials etc. To what extent certain systems have compromised and what steps have been take to mitigate this. Thirdly, organisations should run drills, better access management systems, better auditing systems, have an XDR system in place. You will get hacked, security will be compromised, it is the ability to reduce the contagion risk is what the organisation should measure. Protecting their customers.
You cannot shutdown access to resources as developer need them for productivity. Secondly, your competitor is most likely going to take risk of hack and move faster in bringing features while you reduce productivity.
Would hardware security keys protect from this? If you already have a session token on a site (and that site doesn't somehow restrict the session token to only being used on the machine that generated it? Which afaik isn't possible) then it's too late, yes?
Correct, the hardware token adds nothing over TOTP in this case. You can use a platform authenticator instead of a roaming authenticator, but that only stops you from authenticating on a different machine; it doesn't stop a stolen session from being used on a different machine.
The step-up authentication CircleCI mentions is probably more of a sudo model, with a long-lived baseline level of privilege that can only be elevated for short bursts. This is orthogonal to MFA, but it's at least less of a nuisance with a hardware token than with other options.
> and that site doesn't somehow restrict the session token to only being used on the machine that generated it
One simple practice for security-critical systems is to bind sessions to the client's IP address during authentication. It's not bulletproof given the assumption of malware an attacker can still tunnel through, but neither is locking the session to the device. You could, but the malware is also on the device and can do anything the user can (this is why it's preferable to test user presence outside the device by e.g. tapping a USB key).
With WebAuthN it should be possible to require that credentials are frequently refreshed, which can only be done on the original device. mTLS has similar properties.
You can also require an authentication loop for each major service, so only services the developer is authenticated to would be vulnerable to credential stealing.
And for critical infrastructure network security would help (e.g. a vpn). With hardware tokens required to initiate the connection, the attacker wouldn’t be able to spin up their own connection.
Finally, access to user secrets shouldn’t be a default level of access for engineers. Instead, that should require a break glass escalation of privileges, reducing the risk of pilfering.
> Would hardware security keys protect from this? If you already have a session token on a site (and that site doesn't somehow restrict the session token to only being used on the machine that generated it? Which afaik isn't possible) then it's too late, yes?
Sort of.
There are ways to verify both the device being used to access a resource, and the account. This is a good use for TPM. One of the features is attestation, which can be used to do this sort of a thing.
For things like crypto keys, this is why HSM (hardware security module) exist. - they make it very difficult to get access to private key (basically, private key doesn't leave the HSM itself, but other cryptographic artifacts based on that key do).
It's interesting cuz they're saying it's a stolen SSO cookie. Chrome on macOS stores cookies encrypted under a key that's stored in the keychain. So does Safari. To get to the key one of three things must have happened:
1. The engineer was prompted to give the malware (PTX app?) access to the browser key, and agreed to this. Big mistake if so.
2. The malware has an exploit for macOS security.
3. The malware has a way to take over the browser cross-process.
Hardware security keys don't help in this case. What helps is either users not granting malware access to critical secrets, OR, operating system security being enhanced.
The sad part is, the damage this does to companies won't be felt for years (which is how long it'll take someone to take the stolen source code, analyze it, and make a convincingly-distinct clone), so companies will think that nothing came of this, and they'll keep using CircleCI (and other similar platforms which put everyone's eggs in the same basket, how appealing to hackers that must be).
I suspect the intruders would use the code to find vulnerabilities in other companies. Which they can use for easy ransomware access or straight up stealing customer data. Or both. Much less trouble than analysing and running someone else’s software.
Web software is incredibly simple. Having someone's ruby code doesn't really help you much. Hackers just try a few SQL injections, stuff script tags into all of the form fields, and call it a day. More advanced hacks on webapps are virtually unheard of. Certainly custom-designing a hack for some company isn't going to happen. It's a waste of time, because chances are you won't find some complicated hack that wasn't already exposed through the spamming techniques I mentioned above. And most larger, "worthwhile" companies tend to run their own in-house CI and not use Circle.
In this incident, the unauthorized actor exfiltrated customer information on December 22, 2022, which included environment variables, keys, and tokens for third-party systems.
A lot of production AWS/GCP keys are likely stolen for those that deploy from CI/CD.
How could a stolen session token even be useful. I have to log into tools every day, if I change IPs at all I have to re-authenticate, and all prod access needs to be approved and has a finite lifespan.
How could a CI company be that negligent. They should be leading this stuff from a best practice point of view.
No need for a VPN if you have access to the laptop, you just use that as a proxy. Look at any c2, open source or commercial, and they'll have that as a feature.
Most places I’ve worked, has the concepts from above.
Security is lax at most companies but even having worked as an engineer on support rotation, customer anc production access was the last thing I could do and I needed to have exhausted all other options. I don’t feel retaining production access to generate tokens should be a thing.
It would be so interesting to get more details about the initial compromise. What was the engineer trying to do that ended up with downloading PTX-Player.dmg and (probably) the PTX.app installed in /Applications? Was it targeted directly at CircleCI or is this some generic info stealer? What AV / endpoint security solution were they using? Did it pass the built-in macOS protections (gatekeeper, xprotect, etc)? Public VirusTotal seems to know nothing about that hash.
I worked for an anti-virus company. There are tools that check if your malware can avoid detection by the major virus scanners. As such, the recommendation is never to rely on a virus scanners alone to protect critical assets.
I think the CTO has played it pretty well, his recent blog post is kind of "transparent" enough to sound like they care, but very quick to rush everyone back to normality with a kind of "nothing to see here" attitude.
"Thanks customers for the support" is almost a patronizing thing to say IMO. They should at least offer compensation financially for this and as others have said, his recent update has left more questions unanswered for me.
The way I see it, I'm done as a customer, just need the time to migrate away.
What is interesting here is CircleCI is SOC2 Type 2 compliant. The whole narrative changes if CircleCI was only a self hosted solution and the hack would have happened by one of the customer employees. I'm sure no one would have blamed CircleCI. I dont know if this employee had remote access to all the self hosted enterprise customers too, then that's true lapse on CircleCIs part.
SOC Type 2 compliance only suggests they have a sit in audit once a year. Basically the produced a lot of paperwork and it doesn't mean that they evidenced stuff honestly to the auditors.
Auditors audit for one year. They make sure all the processes and controls are hit and used as expected. I am wondering why the audit firm is not made accountable when these hacks happen.
SOC2 is relatively minimal. You describe your policies and controls, the auditor verifies that they meet some baseline requirements of generic "best practices" appropriate to your risk profile, then they collect data during the observation window to verify that you actually adhere to them.
In this case, I imagine that CircleCI's controls involved developer workstations having anti-malware software, logs and audit trails for access to production systems, and some kind of intrusion detection system around those, and some SLAs and policies on how they react to alerts from those systems. It sounds like they had all of those things in place but the malware wasn't detected and their IDS didn't pick up the external access. Unfortunate, but both of those are entirely possible in many environments. SOC2 can't guarantee that your anti-malware systems or IDS are flawless and can't be bypassed by a clever attacker. Otherwise, since they've been able to identify what the attackers gained access to, it sounds like they did have logs and audit trails in place.
SOC2 auditors typically only have a limited understanding of security and technology themselves (the higher end ones might have more resources available to them) and are only able to sample a small amount of data during the observation window.
If you're trusting your sensitive data and credentials to a 3rd party, you should definitely require that they have SOC2 or better, but you still have to own your own security and consider how you need to protect yourself if/when they are compromised.
The post says "If you stored secrets on our platform during this time period, assume they have been accessed" so I'm guess self-hosted customers weren't impacted.
The method of attack sounds like CircleCI's production cloud (probably AWS) was impacted - "the targeted employee had privileges to generate production access tokens as part of the employee’s regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys."
But I am surprised that their SOC2 auditors didn't raise exceptions about their lack of controls. Sounds like a pretty immature program, they only talk about 2FA, MDM and SSO which is basic stuff. Where is the SIEM? Or CSPM? Or any alerting!? Yes there are SOC2 automation platforms out there that rubber stamp stuff, but at CircleCI's scale I'd expect more scrutiny.
I hate to be that guy but this news highlights some blatantly incompetent security protocols (especially key management) by a company that we should expect better from. Even something as simple as a Vault (HashiCorp) cluster with decentralized key shares would’ve prevented this. I’m really disappointed in CircleCI. There’s no way I’d trust them after this.
> On December 29, 2022, we were alerted to suspicious GitHub OAuth activity by one of our customers. This notification kicked off a deeper review by CircleCI’s security team with GitHub.
> ...
> Review GitHub audit log files for unexpected commands such as ... repo.download_zip
I don't know what you'd get from GitHub other than source code but the CircleCI blog post explicitly describes the attacker downloading entire repos as a .zip
GitHub OAuth access credentials could be used to compromise OAuth-authenticated apps. And GitHub personal access tokens with too broad of permissions could download pretty much anything, including any secrets stored in GitHub Actions.
But getting the source code is pretty bad by itself. Not from an intellectual property standpoint, but because I've never seen a company whose developers didn't commit live credentials into their source code.
If they had access to GitHub Actions creds, I'm assuming the attackers could also push releases. I wonder how much malware is now in the wild because of this breach
> I've never seen a company whose developers didn't commit live credentials into their source code.
I don’t do that, but I’m pretty pathological about stuff like that.
I learned it from the company I used to work for, who were paranoid to the point of lunacy, about Chinese hackers (they were breached once, and took the lesson to extremes).
I don’t think they are an outlier. I’ll bet lots of companies are just as tinfoil.
It’s a downright unbearable development environment, though.
Github secrets are encrypted, right? I didn't think there was any way to access them without running a github actions job that deliberately uploads them somewhere else (or echos them into the log with rot13/base64 or something else).
The article is referring to how the attacker could use the stolen github tokens to download someone's source code. The source code isn't coming from CircleCI
CircleCI has listed literally every kind of credential used by its users as vulnerable. This includes deploy keys that are used to download source code. Anyone who has access to customer data, and a deploy key, can just check out the source code, instantly.
The extent to which CircleCI has gone to eliminate all threats is... scary. They've gotten GitHub to invalidate any GitHub access tokens used by a CircleCI customer. They've gotten AWS to e-mail AWS customers if one of their access keys was stored in CircleCI. It's a complete and total compromise of literally every customer secret in CircleCI. I expect this will be the biggest hack of 2023... and it's still January.
So, yeah, I'm pretty sure customer source code is up for grabs.
From the circleCI blogpost[1]:
"Our investigation indicates that the malware was able to execute session cookie theft, enabling them to impersonate the targeted employee in a remote location"
I haven't seen much discussion on how this specific attacker entrypoint can be mitigated. So I'm going to make a naive attempt in this comment.
How about storing the client's IP address in the session cookie. Then whenever the server recieves the cookie, it compares the client's IP address against the one stored in the session cookie. The server denies the login if there's a mismatch.
The cookie would of-course have to be signed(hmac etc) so that it is tamper proof.
One problem with this is that client IP addresses are easily spoofed[2].
So, instead of storing the client's IP address; how about we instead store the clients' SSL fingerprints[3][4]. I haven't looked much into the literature, but I think those fingerprints are hard to spoof.
> How about storing the client's IP address in the session cookie. Then whenever the server recieves the cookie, it compares the client's IP address against the one stored in the session cookie. The server denies the login if there's a mismatch.
That doesn't work in environments with multiple NAT origin IPs in place, or when they're using crap like Netskope/some other "security"/"privacy"/"VPN" software, as IPs tend to randomly change with these. It would generate way too many false-positive reports.
> One problem with this is that client IP addresses are easily spoofed[2].
Only if the backend servers are badly set up. For me, I always run haproxy as the frontend and forcibly delete incoming headers (X-Forwarded-*, Forwarded), and as an added precaution the backend software is configured to only trust the haproxy origin IPs - so even in the case an attacker manages somehow to directly access the backend servers directly, they cannot get a spoofed IP past the system.
> So, instead of storing the client's IP address; how about we instead store the clients' SSL fingerprints
That requires client-side SSL authentication, which is theoretically supported by all major browsers, but very rarely used and the UI support is... clunky at best.
That fingerprints a piece of software (and, if SSL library versions or configurations change, the version), but not a specific client. It's virtually worthless as a security measure if the endpoint is a common browser.
Customers got an email from Circle immediately after they were able to "resolve" the issue. The full post-mortem was released recently. We got the email on Jan 4 or Jan 5.
Some customers got such an email, but not all of them (also see https://news.ycombinator.com/item?id=34255721). The only email received by people in my org was the email pointing to the post-mortem, which went out on 2022-01-13.
This is a super critical hack, looks really bad for CircleCI. But I don't see any negative news much elsewhere. They will move on.
Also, maybe that localised build system running on an old server for each team seem to be not a bad idea to reduce the blast radius when eventually a hack happens. These providers are supposed to be the gatekeepers and experts who one leaves the tedious and critical work to. If they are just being a leaky cauldron, maybe not bad to cook in my old pot at home.
A hack of a CI/CD (Continuous Integration/Continuous Deployment) service can have a significant impact on the companies that use it. In this scenario, an attacker would gain unauthorized access to the CI/CD service's servers, potentially stealing sensitive information such as source code for various companies' software projects. This type of incident is similar to the Okta breach
> Zuber said that while customer data was encrypted, the cybercriminals also obtained the encryption keys able to decrypt customer data.
...what? why did this engineer have access to everything? Does CircleCI know what minimum access policies are for?