Logging doesn't help you when you business has to shut down because someone took over you account and deleted everything.

Separation of access is important and _required_. Developers don't need access to prod, admins maintaining the infrastructure don't need access to the directory, IDM doesn't need access to either QA or prod.

Developers do need full access in an environment to properly test - but that environment should be basically hermetically sealed from the rest of the companies infrastructure. So even if they do screw up the whole business won't be affected.

If someone took over your account and deleted everything, and you couldn't get any of it back, you weren't taking care of the "availability" third of security. I agree that developers don't need access to everything, but I completely disagree that they should have no access to prod.

The games of phone tag and "try typing this" that happens during prod issues is a waste of everybody's time, and I fully believe that the people who write the code should be the ones with both the responsibility of the pagers and the ability to fix the code they've deployed. Everybody is happier, and the job gets done more quickly, when the job gets done by the people most qualified to do it (because they wrote it), and when they bear the consequences of writing bad code.

The environment needs to be set up to be forgiving of mistakes, yes, but that's easily done these days and should never result in loss of data if the infrastructure is properly automated. If giving production access means your developers can screw something up, then your admins can just as easily screw something up. Create environments that forgive these failures because they'll happen one way or another.

There are already examples companies which have folded overnight after losing creds and having everything deleted.

Removing root is not a trust issue - it’s a security surface area issue. You increase the number of audit points and attack options by at least an order of magnitude (1 admin : 10 devs).

In a small shop this might be acceptable, however in a large org it’s plain old insane.

If you believe that devs require root then that’s an indicator that your build/test/deploy/monitor pipeline is not operating correctly.

> If you believe that devs require root then that’s an indicator that your build/test/deploy/monitor pipeline is not operating correctly.

For one, I never said anything about root. I'm not sure anybody should have root in production, depending on the threat model. What I am saying is that the people who wrote the proprietary software being operated should be the ones on the hook for supporting it, and should be given the tools to do so, since they're the most aware of its quirks, design trade-offs, etc.

That means not just CI/CD and monitoring output, but machine access, network access, anything that would be necessary to diagnose and rapidly respond to incidents. That almost never requires root.

I completely agree. Very well said.

If all I need to do is gain admin to my dev machine and then I'm free to nuke your production database, your company is doomed.

> If you believe that devs require root then that’s an indicator that your build/test/deploy/monitor pipeline is not operating correctly.

Or it might be an indicator that you are not relying on archaic and ineffective methods to protect your system.

As a dev, I love it when I'm not able to access production systems and poke them.

Of course, some people need admin capabilities. But a senior developer? Nah.

Engineers whose specific job it is to admin infra, yes.

Logging is great, but it's a 'we are owned, how do we get them out' measure - we want to avoid getting to that point.

edit: Of course, this all depends on your threat model, posture, maturity, etc. I'm just saying - we have lots of very good reasons to lock devs out.

You're assuming that a small company has a separate person solely dedicated to infrastructure.

Yes I have an AWS certification and on paper I am qualified to be an "AWS Architect". But I would be twiddling my thumbs all day with not enough work to do and would die a thousands deaths if I didn't do hands on coding.

As I said in my edit, this is all subject to the details of a company's posture.

This sounds like Amin access on local machines, not admin access at all?

Actually, I realized we don't even give local admin. I forgot. Unfortunately, no editing.

"I'm not saying _all_ developers should have unfettered access", but I'm an exception.

And then add, "otherwise I'll leave for a company that gives me that access".

See how that sounds?

Yes that sounds like someone who doesn't want to have to wait two weeks to get approvals to create resources in a Dev environment.

But as the team lead, I already had the final say into what code went into production and could do all kind of nefarious acts if I desired. Yes we had a CI/CD process in place with sign offs. But there was nothing stopping me from only doing certain actions based on which environment the program was running in.

I've seen what happens to people who are "just developers" that spend all their life working in large companies where they never learn anything about database administration, Dev ops, Net ops, or in the modern era - cloud administration. They aren't as highly valued as someone who really is full stack - from the browser all the way down to architecting the infrastructure.

Why wouldn't I choose a company if given that option that lets me increase my marketability, and gives me hands on experience in an enterprise environment instead of just being a "paper tiger" who has certifications but no experience at scale?

It sounds like someone who knows what they want and who like to get their hands dirty in a job market where they habe that choice.

Not all developers are irresponsible, and not all ops people are impossible to work with. But both certainly exist.

That's what made things more infuriating at the company I left. I came in as the lead developer knowing that if I wanted to get things done, I would have to ingratiate myself to the net ops people. I could fire off a Skype, ask for what I needed on prem (VMs and hard drive space mostly) and by the time I sent the ticket request as a formality, it was already done.

But then they decided to "go the cloud" and instead of training their internal network ops people and having them work with the vendor who was creating the AWS infrastructure, the vendor took everything over and even our internal folks couldn't get anything done without layers of approvals.

So I ended up setting up my own AWS VPC at home, doing proof of concepts just so I could learn how to talk the talk, studied for the system administrator cert (even though I was a developer) and then got so frustrated it was easier to change my environment than to try to change my environment.

So now they are spending more money on AWS than they would have in their colo because no developer wants to go through the hassle of going through the red tape of trying to get AWS services and are just throwing things on EC2 instances.

In today's world, an EC2 instance for custom developed code is almost always sub optimal when you have things like AWS Lambda for serverless functions, Fargate for serverless Docker containers and dozens of other services that allows you to use AWS to do the "undifferentiated heavy lifting".