Not getting root on your own machine as a developer? That's ridiculous. I left a company where I was the lead developer because I got tired of waiting two weeks for the outside networking consultants to get resources provisioned on AWS. Our own internal network people had to wait on them.
Now I work for a much smaller company and they gave me full admin access to the AWS console from day one and I am "just" a senior developer.
It's unfortunate but it's true, I've seen it 100x.
We let our devs have local admin at my company but please don't misunderstand the serious risks involved in that decision. We take great measures to ensure that we can do that while still keeping our customers safe.
We would never allow admin AWS access, that's absurd. An attacker on your box would be able to own prod. Sorry that it's an inconvenience! When you manage to solve infosec, hit your ops/sec team up, they'll be happy to hand you the keys after that point.
I think the GP is talking about things like provisioning AWS resources, not having root on the production box.
Anyone should be able to provision new resources without harming production. Most of the time you don't see this, because management has still not gotten it through their thick heads that in order for the business to move fast, devs need the agency to implement solutions fast.
Cost-cutting of AWS leads to one (or several!) groups that are the gatekeepers to AWS, purportedly in order to make it more "efficient" by everyone doing everything one way, or having one group of experts handle it. But that's just one group of plumbers handling repairs for an entire city. Leaks spring up and nobody is there to fix them, and meanwhile, the developer is sitting there staring at the leak, going "will you please just let me duct tape this leak so I can get back to work!?"
Exactly. When the last company I worked for decided to "go to the cloud", it took weeks to get simple EC2 instances setup because we had to go through my manager, say why we needed it, then wait on an outside vendor. We had layers of consultants getting in the way.
I still ended up doing stuff suboptimally like using all of the stuff I would use on prem like Consul (instead of Parameter store),Fabio (instead of load balancers), Nomad (instead of ECS), VSTS (CI/CD - when I should have use the Codepipeline, Code Build, code deploy ) and even a local copy of Memcached and RabbitMQ just so I wouldn't have to go through all of the red tape just to provision resources so we could get our work done. Our projects ended up costing more because it took so long to get anything approved, I would overprovision "just in case".
Anytime that I wanted to stand up VMs for testing, I would send a Skype message to our local net ops person for an on prem VM and have it within an hour.
I was the "architect" for the company - I couldn't afford to make excuses and it's never helpful at my level of responsibility to blame management when things don't go right. I had to get it done.
So, I learned what I could, studied AWS services on a high level inside and out, got a certification and left for greener pastures.
And people wonder why I never do large organizations.
Trust but verify. AWS has plenty of logging capabilities. I'm not saying that all developers should have unfettered access. But someone has to be trusted.
Logging doesn't help you when you business has to shut down because someone took over you account and deleted everything.
Separation of access is important and _required_. Developers don't need access to prod, admins maintaining the infrastructure don't need access to the directory, IDM doesn't need access to either QA or prod.
Developers do need full access in an environment to properly test - but that environment should be basically hermetically sealed from the rest of the companies infrastructure. So even if they do screw up the whole business won't be affected.
If someone took over your account and deleted everything, and you couldn't get any of it back, you weren't taking care of the "availability" third of security. I agree that developers don't need access to everything, but I completely disagree that they should have no access to prod.
The games of phone tag and "try typing this" that happens during prod issues is a waste of everybody's time, and I fully believe that the people who write the code should be the ones with both the responsibility of the pagers and the ability to fix the code they've deployed. Everybody is happier, and the job gets done more quickly, when the job gets done by the people most qualified to do it (because they wrote it), and when they bear the consequences of writing bad code.
The environment needs to be set up to be forgiving of mistakes, yes, but that's easily done these days and should never result in loss of data if the infrastructure is properly automated. If giving production access means your developers can screw something up, then your admins can just as easily screw something up. Create environments that forgive these failures because they'll happen one way or another.
There are already examples companies which have folded overnight after losing creds and having everything deleted.
Removing root is not a trust issue - it’s a security surface area issue. You increase the number of audit points and attack options by at least an order of magnitude (1 admin : 10 devs).
In a small shop this might be acceptable, however in a large org it’s plain old insane.
If you believe that devs require root then that’s an indicator that your build/test/deploy/monitor pipeline is not operating correctly.
> If you believe that devs require root then that’s an indicator that your build/test/deploy/monitor pipeline is not operating correctly.
For one, I never said anything about root. I'm not sure anybody should have root in production, depending on the threat model. What I am saying is that the people who wrote the proprietary software being operated should be the ones on the hook for supporting it, and should be given the tools to do so, since they're the most aware of its quirks, design trade-offs, etc.
That means not just CI/CD and monitoring output, but machine access, network access, anything that would be necessary to diagnose and rapidly respond to incidents. That almost never requires root.
> Not getting root on your own machine as a developer?
was the origin of this thread, and there are tons of places where developers are not permitted root access to their own dev machines. We are not all talking about prod instances.
I have this conversation with my own counterparts in network / platform / infosec / application teams (I am an app dev), and in some cases the issue is conflated because dev environments are based on a copy of prod, and the compromise of such prod-esque data sources would be almost equally as catastrophic as an actual prod compromise.
If this is your environment, then don't be that guy and make it worse by changing the subject from dev to prod. Don't conflate the issue. Dev is not prod and it should not have a copy of sensitive prod data in it. If your environment won't permit you to have a (structural-only) copy of prod that you can use to do your development work unfettered, with full access, then you should complain about it, or tell your devs to complain if it affects them in their work and not such a big deal for yours.
Developers write factories, mocks, and stubs all the time to isolate tests from confounding variables such as a shared dev instance that is temporarily out of commission for some reason, and so they don't have to put prod data samples into their test cases, and in general for portability of the build. Then someone comes along and says "it would be too expensive to make a proper dev environment with realistic fake data in it, just give them a copy of Prod" and they're all stuck with it forever henceforth.
It's absolute madness, sure, but it's not misrepresented. This is a real problem for plenty of folks.
You're assuming that a small company has a separate person solely dedicated to infrastructure.
Yes I have an AWS certification and on paper I am qualified to be an "AWS Architect". But I would be twiddling my thumbs all day with not enough work to do and would die a thousands deaths if I didn't do hands on coding.
Yes that sounds like someone who doesn't want to have to wait two weeks to get approvals to create resources in a Dev environment.
But as the team lead, I already had the final say into what code went into production and could do all kind of nefarious acts if I desired. Yes we had a CI/CD process in place with sign offs. But there was nothing stopping me from only doing certain actions based on which environment the program was running in.
I've seen what happens to people who are "just developers" that spend all their life working in large companies where they never learn anything about database administration, Dev ops, Net ops, or in the modern era - cloud administration. They aren't as highly valued as someone who really is full stack - from the browser all the way down to architecting the infrastructure.
Why wouldn't I choose a company if given that option that lets me increase my marketability, and gives me hands on experience in an enterprise environment instead of just being a "paper tiger" who has certifications but no experience at scale?
That's what made things more infuriating at the company I left. I came in as the lead developer knowing that if I wanted to get things done, I would have to ingratiate myself to the net ops people. I could fire off a Skype, ask for what I needed on prem (VMs and hard drive space mostly) and by the time I sent the ticket request as a formality, it was already done.
But then they decided to "go the cloud" and instead of training their internal network ops people and having them work with the vendor who was creating the AWS infrastructure, the vendor took everything over and even our internal folks couldn't get anything done without layers of approvals.
So I ended up setting up my own AWS VPC at home, doing proof of concepts just so I could learn how to talk the talk, studied for the system administrator cert (even though I was a developer) and then got so frustrated it was easier to change my environment than to try to change my environment.
So now they are spending more money on AWS than they would have in their colo because no developer wants to go through the hassle of going through the red tape of trying to get AWS services and are just throwing things on EC2 instances.
In today's world, an EC2 instance for custom developed code is almost always sub optimal when you have things like AWS Lambda for serverless functions, Fargate for serverless Docker containers and dozens of other services that allows you to use AWS to do the "undifferentiated heavy lifting".
A developer wouldn’t have to install malware. A developer could create malware. Even if you have all of your deployments automated, any developer worth anything could sneak malicious code into the process.
A developer doesn't need admin access to the AWS console to install malware, bitcoin miners, etc. He just needs to have his code installed. The person who is deploying the developer's code installed is rarely going to code review the code before its installed. If my code has access to production when you deploy it, I can make it do anything I want and you would never know.
What developers are you talking about? I want to know what developer would risk their career and prison time. And if a developer has no problem with going to prison, surely they have no problem finding some 0day privilege escalation exploit.
> It's unfortunate but it's true, I've seen it 100x.
That's not really an argument, how many developers did you have and how many of them risked prison time to install malware on dev machines?
> We take great measures to ensure that we can do that while still keeping our customers safe.
> We would never allow admin AWS access, that's absurd. An attacker on your box would be able to own prod.
You aren't doing a great job then, because your production stuff should be on a separate AWS account altogether.
Even if your production stuff is in a separate account, that just helps prevent someone from accidentally screwing up production. To think that not giving your developers - the people who are creating code that you are putting on your servers and know the infrastructure as well as anyone - will prevent them from being malicious is just security theatre. It may help you check the box about being compliant with some type of standard but it really doesn't help you. If the developers program has access to production resources, they can gain access to those resources.
I think of bitcoin mining attacks as pentesting with a contingent fee. Probably the cheapest and least destructive way to learn you have security weaknesses.
This isn't what this is about (nanny state not allowing you root on yer laptop). Rather it is about the idea that software should require the necessary privileges to do its job, and no more.
Singularity in general requires root, it's just in the form of setuid helpers. Now, you can force it to not require root but from memory there are a lot of caveats with using it that way. Unprivileged LXC is much more full-featured. And obviously rootless runc works great as well (though I'm a bit biased of course).
Singularity is capable of using setuid helpers, but by default it uses user namespaces (the USERNS kernel feature) and does not need the setuid helpers.
Lots of things are 'more full-featured' and none of them work well in an HPC context, where individual user jobs may need to be staged carefully.
> Lots of things are 'more full-featured' and none of them work well in an HPC context, where individual user jobs may need to be staged carefully.
After long discussions with the Singularity folks I've come to the conclusion that the only special features that Singularity has are:
* Their distribution format is a binary containing a loopback block device, allowing you to have "a single thing to distribute" without concern for having to extract archives (because in theory archive extraction can be lossy or buggy). The downside is it requires (real) root to mount or a setuid helper, because mounting loopback is privileged because it opens the door to very worrying root escalation exploits. When running without setuid helpers I'm not sure how they have worked around this problem, but it probably involves extraction of some sort (invalidating the point of having a more complicated loopback device over just a tar archive).
* It has integration with common tools that people use in HPC. I can't quite remember the names of those tools at the moment, but some folks I know wrote some simple wrappers around runc to make it work with those tools -- so it's just a matter of implementing wrappers and not anything more fundamental than that.
Aside from those two things, there's nothing particularly different about Singularity from any other low-level container runtime (aside from the way they market themselves).
A lot of people quote that Singularity lets you use your host user inside the container, but this is just a fairly simple trick where the /etc/passwd of your host is bind-mounted into the container and then everything "looks" like you're still on your host. People think this is a security feature, it isn't (in fact you could potentially argue it's slightly less secure than just running as an unprivileged user without providing a view into the host's configuration). If you really wanted this feature, you could implement it with runc's or LXC's hooks in a few minutes.
I'm working on [1] and the plan is to make this work much easier for people, so they don't need to hit the "docker is root" problem anymore -- this was a pain for me a few years ago when I was a university student trying to run some code on a shared computing cluster.
There's still lots of stuff left to do (like unprivileged bridge networking with TAP and similarly fun things like that).
Ever worked in government? You won't get (full) admin access there, either.
I thought it was good practice to have strong separation between Dev and Production, and I'm pretty sure you're meant to create AWS keys+accounts with less-than-root access for day-to-day work.
Yes. I create separate roles for different ec2 instances, Lambda expressions, etc. based on least privilege.
With AWS databases - except for DynamoDB - you still use traditional user names/passwords most of the time. Those are stored in ParameterStore and encrypted with keys that not every service has access to. Of course key access is logged.
There is a difference between the root account and an administrator account.
Day to day work on the console is configuring resources.
Even if you do have strong separation -in our case separate VPCs, someone has to have access to administer it. We don't have a separate "network operations" department.
I don't want to run services as root, except for services that grant login rights (It's probably better to have one ssh daemon, than allow/force every user to run their own...). And even then, keep the code that can change user I'd to a minimum.
I certainly don't want run big, rapid-changing code as root. Not the countinous integration pipeline, not the build pipeline.
It's a misfeature of docker that it needs more privileges than a traditional chroot/fakeroot build (probably a good reason to build on those tools to build docker images, rather than rely on Dockerfile/docker build. Build images and reserve Dockerfiles for pulling in ready images and setting some parameters for configuration).
Another major use-case of rootless containers (though image building is not as useful in that case) is being able to run things unprivileged on computing clusters. I implemented rootless containers in runc and quite a few other tools (like umoci) in order to be able to handle cases where you don't get root on a box.
There is also the security benefit of there being no privileged codepath that can be exploited. So the only thing you need to worry about is kernel security (which, to be fair, has had issues in the past when it comes to user namespaces -- but you can restrict the damage with seccomp and similar tools).
As CTO at a small healthcare company, one of my primary goals is to prevent even our employees from getting unaudited access to the db in our production system. In dev, no problem.
Having AWS admin privileges doesn't give you access to the database data. RDS/Aurora/Redshift instances are still controlled by your typical database privileges.
I don't think this is right. With RDS, for instance, you can reset the 'master' password from within the RDS console. (The RDS service itself retains the real superuser, the user RDS creates for you has some privileges dropped.)
Take EC2 as another example. If you have volume attach/detach permissions and EC2 start/stop permissions, you can stop an arbitrary instance, detach its root volume, reattach it to an instance of your choosing where you have login access, log in, add whatever you want to that volume (including rootkits, etc.), and reattach it back to its initial instance.
Giving someone AWS admin should be considered analogous to giving someone the keys to your racks in a datacenter. There really are many surprising ways that an AWS admin can take control of your infrastructure. Can you put countermeasures in place? Sure, but it's a huge attack surface
You should probably assume that anyone that has admin access to your aws account has complete access to everything in it. It’s too easy to get access to whatever secrets you need.
Devs like you were some of our favorite targets in the pentest world. What I wouldn't give for your ~/.bash_history file... I bet I could pivot to three different servers in under an hour.
This is an exaggeration, but only slightly. :)
Security costs convenience. But people love to be too lax. And it's so fun to point it out and see the look on their faces, or pop up an XSS in their favorite stack of choice.
My best one was getting remote access on a server thanks to an unsanitized PDF filename. They were calling into the shell like `pdf-creator <company name.pdf>` (or whatever the utility was called). They were a B2B service, so they never thought to set anyone's company name to something like "; <reverse shell here> #"
I just thought it might be fun to contrast the two worlds. Those big, stodgy companies that we love to make fun of... Those guys were some of the hardest targets. I once spent a week trying to get anything on one, and just barely got an XSS. And I was lucky to find it.
Developers often are a soft target, esp. in small/medium companies, but in my experience it's more of necessity by imposed expectations from management, than people actually wanting to have "root" everywhere. But I and many others dislikes getting yelled at by management, and no one has yet to accept any reasonable security precautions as a reason for delayed delivery without promptly ordering that precaution to be summarily removed, at least not in any of the companies I worked or consulted at the last couple pf decades.
If management understands that security costs time in feature development, fine. But with the role software development has these days in companies, if security and ops doesn't succeed in getting management on board, please don't hold the developers hostage! Work with them and try to find the least bad ways of working quickly enough. Many of them will support calls for better security practices as long as it doesn't imply more sleepless nights because goals haven't been changed, only the speed of which work can be done.
For any substantial deployment, I really don't want to have any access as a developer, but often I have to have root access to tons of machines simply to have any chance at actually doing my work.
Now I work for a much smaller company and they gave me full admin access to the AWS console from day one and I am "just" a senior developer.