Hacker News new | past | comments | ask | show | jobs | submit login
Cleaning Up Dead Bodies in AWS IAM (noq.dev)
42 points by securiy on Aug 28, 2023 | hide | past | favorite | 56 comments



At my last company they asked me to find all the users who no longer needed access to our AWS account, as well as create a report for teams to review if each of their members needed access to the roles they have access to. It took a little bit to understand the IAM model, but I created dozens of reports for a few hundred engineers. Dead users were deleted, but literally nobody reviewed group access with the reports I sent out. Just one of many projects I did for a VP where there was no active demand for that project. VP was made redundant shortly after


Why did you have users in your AWS account besides a couple of emergency break glass users?

All other users should have been going through some SSO using Microsoft AD, Okta, etc.


I did not come up with the system


IAM users are still needed in large orgs because of legacy tooling that only accepts access key ID / secret access key sigh.


ACLs/Policies (especially the very fine-grained ones used by AWS now) + groups + roles + users + resources

Probably maps to the SAT / NP-complete space. Congrats! Management of security permissions is virtually guaranteed to be non-polynomial.


Can you share what the high-level goal of the project was?

i.e. was the VP trying to reduce risk? scale out responsibility for managing access?


I'm dealing with the same type of nonsense currently, as an internal audit team sees security groups being flagged by the scanning software that are open to 0.0.0.0/0 which is automatically "bad", even though the hosts have no public IP's and are being automatically managed by EKS to setup links to k8s NodePorts and the ELB.

Same with security groups. Gartner has some "best practice" doc somewhere, someone loads that into a security tool, the tool flags things, and these checkboxes must go from red to green. The technical hurdles to comply or actual value do not matter.


I'm still on the platform side, out of K8s and the like - but this is so painfully true.

They try to tailor a lot of these things to the OS/distribution, but fail in the most wonderful ways.

A recent example: they're aware of RHEL. They're also aware of 'firewalld'.

However, they have not managed to realize that this is simply a management interface to other firewalls -- imposing standards on a long-deprecated backend; iptables

Meanwhile, using incredibly inefficient and 'portable' command lines. ie: using find in such a way that an LDAP query happens for every file

Refusing to use the arguments available to the operating system they 'tailor' for. Ultimately timing out once you hold a certain number of files.


I find interesting the last example with find and ldap queries. I'm not too familiar with ldap, but i do use find frequently. Could you expand on the example and what happens and why it's bad?


Certainly! In this case, they were interested in files that were too permissive.

I don't have a good example of the command, but it was basically looking for 'worldly' permissions that were too open. It's important to note the users/groups could be discarded/ignored.

They were using 'find ... -exec ls -ld {} \;', which does an LDAP lookup on each result to resolve UIDs and GIDs to names.

They could have made the process far more efficient with either the native '-ls' argument built into find, or adding '-n' to the exec'd 'ls'

Either would skip the name resolution/domain. At a certain number of results/files the expense is too high, causing the job to time out


I think correctly configuring nscd should prevent this, ie it should cache some of these name lookups for a period of time. As long as it is properly setup for LDAP to hook into it.

https://linux.die.net/man/8/nscd


I suspect you're right, I'm not as familiar with our configuration (or the service) as I'd like to be

The 'ls' output is honestly superfluous, though - 'find' will report the paths.

I won't even get into how these are batched/time limited. If not this, it'd be something else eventually


Conincidentaly I just found a book in the office on understanding and deploying ldap directory services, seems like today is gonna be spent reading!


Best of luck to you! I've been around them for over a decade now, and still can't commit the various services to memory.

I like to call what I do "taking the coward's way out" -- using FreeIPA

My team setup the infrastructure in question and I've been too slow to learn it. FreeIPA is nice for quick/easy deployments.

I'm not sure how well it "scales", but it's great for getting comfortable with the "Domain Language" (sorry, pun)


Sounds like a spectacular self-DDoS on your LDAP infrastructure if that thing ever got deployed too widely and ran at the same time everywhere.


Dumping audit reports with a sea of red on delivery teams for things largely low-risk in reality and out of their control rarely does any good.

I'm not up on EKS-ELB these days, but if the nodes only allow ingress to NodePorts from the ELB, then it seems like those findings should be suppressed in most orgs.

If the SecEng team was partnering with the delivery team they'd know that before sending the report.


This is the sort of ladder-climbing VP behavior you see from someone who is too concerned about avoiding failures that they don't actually do anything productive.

Don't launch any projects in a firm direction because if they fail, it's a failure of commission. Wastes lots of time churning what-if scenarios, blocking things & generating reports no one wants in case he gets asked for them, so he can't be accused of a failure of omission.

It's the 3d chess played by someone who forgets what their actual day job is - getting shit done.


If I’ve learned anything, the only people who care about doing things are at the very, very bottom. No one up the chain actually cares about doing things. They talk about doing things, present grand slidedecks internally and at conferences about doing things, have project/product/engineering managers constantly planning on doing things and thinking about better/faster ways to do them. But really there’s an entire pyramid of people in the company just kind of keeping themselves busy while the grunts at the bottom turn out as much as their hands can muster while desperately hoping to one day be a non-doer (or move to yet another company where they’re promised they’ll _actually_ do things but really it’s the same exact company with a different logo).


You guys have it so wrong. Their job is to get you to do things. With slide decks. Presentations. Speeches. Roadmaps. Stories. Visions. Carrots. That’s their job. As well as to aggregate the litany of statuses into an über status at the end of the week/month/quarter so that their higher ups see work being done. What they do is different from what you do so you only see them not doing what you’re doing, not what they are doing. However, if we are going to generalize, yes - you are correct on the fact that they spend their time thinking about better/faster without making it better nor faster (mostly the opposite).

They have context into why you are doing something, even if you don’t.


You're describing the theory of managerialism. Or maybe it's better to call it the dogma. But it's definitely not always the reality.

Large companies have huge inertia. And these days we also have low average CEO tenure and frequent executive position changes. The upshot being that what a given executive does can be almost entirely disconnected from productive improvement without notable short-term harm to the company. In that kind of an environment, an ambitious executive can put the bulk of their energies to seeming effective without much worry as to actual effectiveness. Or just to indulging their personal predilections, like feeling important or in control.


That’s why it’s a theory and not a law. You can attempt the same management game and get wildly different results depending on the team, the ask, the tenure, you, the company, you name it. Often what works one place, doesn’t work elsewhere. Not because they lack the understanding but because of Conway’s law.


I think it's a theory because that's what's useful for the people with power to believe and have others believe. It's the same deal with any elite; their first task is to secure a tacit belief in their superiority. CEO salaries have rocketed up in recent decades. Is that because they're wildly better? I don't think so.


What's funny to me having watched some executive hires flame out is the runway they are given.

It's generally assumed/understood that the higher you go the more control you have, but conversely the slower any change you try to effect is. So it's sort of like the old 3 envelopes school of management joke. They get 6-12 months to settle in. Then they do a reorg to bring in their team over the next compensation cycle. Next compensation cycle their team brings in their teams, and so on. About 3 years in and then people may start taking a long hard look at the progress or lack thereof. Finally because C-suite doesn't commit fratricide, they are given a tap on the shoulder and managed out with a nice severance, a process that may take another year.

So I've been at shops where the guy at the CTO was clearly not succeeding, didn't have stakeholders buy-in, and lacked the grunts respect. Nonetheless they got 3-5 years of very fat paychecks during which they hobbled the entire org.

At the IC level I've seen people bounced within their 90 day probation.


>At the IC level I've seen people bounced within their 90 day probation.

Or worse. Yet what you just described is something I recently witnessed at a Fortune 5000. So I can corroborate your point of view entirely.

Also, I once was let go the day after coming back from Christmas vacation (planned, whole office was out) as well as let go once a couple weeks before Christmas. The more time you spend in the industry, the more BS like this you’ll come across. These incidents were 13 years apart, but it goes to show that it’s the same, no matter when.


In a working org you are right.

In many orgs measurement of work takes precedence over actually achieving work.


All hail the oracle KPIs and OKRs to save the organization and steer us towards redemption…


This comment opened my eyes in a strange way to my boss (dir of infra). This is so on the nose for how he operates it was almost painful to read.


Most folks (certainly not all!) at the Director level and higher by definition spend their entire day talking about work other people are doing rather than doing it themselves. It's the nature of the beast, especially if you operate with a manager only having one or two small (3-6) person teams to manage. You can get a flatter org chart where managers have 15 or 20 direct reports, which makes it impossible for the manager to both be a good manager and GSD, or you can have managers who still GSD but you have so many of them you start to add layers so that the C-level can still do what they need to (get investment, or drive revenue, or strategic partnerships or whatever depending on your scale/stage).

It's depressingly easy to end up in a situation where the line employees are overworked and underpaid, the first level managers are stressed out trying to really manage well their half-dozen direct reports while still producing work themselves, and the Directors and VPs end up passing reports back and forth all day, every day.


Studying span of control, managerial time allocation, and corporate promotion criteria becomes more interesting the further I get into my career. They literally define (or change) companies.


“Getting shit done” is 3D chess we play to convince ourselves there is a goal when it’s just more low effort toil.


Work is work they give me a paycheck to write code.

Rather write some code than deal with a chickenshit leader who wants 100 iterations of project plans for work we will never do or generating reports that no one will ever read.

Being paid to sit at a computer and not doing real work all day is more torturous than simply having actual tasks and work to do.


That's my view. Going to pointless Zoom meetings and updating associated spreadsheets is much worse than actually doing real work. Of course, much of the "real work" is also quite uninteresting, but that's life.


[flagged]


[flagged]


You can float measures but they’re always incomplete models.

I did not say all code is useless.

You generalized based upon a specific conversation. Generalization I have heard over and over from speakers who, when measured, do not measure up as highly as they think they should.

I am aware of various studies that measured what technologies are increasing productivity. The variables were too numerous to make meaningful conclusions.

We cherry pick and make small models at jobs to obfuscate and debate who is pulling their weight. But it’s just toil to stay employed. The future value of any given for loop is zero; it’s just hustle to crank them out regardless of their use case, and denigrate others cranking out senseless reports.

There is no model that says the poster I replied to, or you, are outputting more value. It’s just toiling to make it appear so at a small scale (within the business).

14% of working adults in the US has better than a bachelors. Public opinion polls indicate the public thinks it’s almost 40% of working adults have better than a bachelors.

What your missing is how deluded the general public is about its accomplishments.


[flagged]


Those are not the rules of your game.

If you thought logins didn't matter, you'd create one and stick with it. You're going to extra effort because you know they do.

And still, you dodge the question. You claim to be incisively dissecting other people's motivations via various negative assumptions and broad-stroke insults. But you can't even glance at your own motivations, even as you work to hide them under a series of new accounts. Which tells us a lot about your game.


Reducing risk, yes, but I think the VP just sat around thinking of project ideas that sound useful without asking around to see if relevant stakeholders are actually interested


Thanks for the additional context.

Agree that leaders should definitely be ready to motivate stakeholders and collaborating teams to act on this kind of info before spending significant time gathering & producing it.


Often creating work is a goal in itself. You gotta justify all those people somehow.


I've made it a habit to only create IAM roles/users/policies using CloudFormation. This way they can be easily removed, are version controlled and you can see why they were created in the first place.


Just to get an idea of where your solution stands on the market, what tools are you competing with and what "CSPM/CIEM tools that fall short in cleaning up AWS IAM" do you imply?


>Discover why conventional CSPM/CIEM tools fall short in cleaning up AWS IAM, and explore a better solution with Noq and IAMbic

We live in a noun hell where every technical topic has a high barrier to entry that makes it hard to casually learn anything. It's difficult to be even a traditional generalist in this ecosystem, and yet the market treats people as if the only way to be considered valuable is to be a super generalist (depth + breadth).


Acronyms are the worst.

I guess people do it to sound cool or something, but I once maintained a legacy project that had an acronym for a name. Not a single person working at the entire company knew what the acronym originally meant, and of course, it was never documented.


Acronyms are no different from words, in this regard. How many people remember/know why your storage is called a drive? A computer used to be a person. Why is the company Stripe named that?

Then there are backronyms and names that used to be acronyms, but technically aren't anymore.


Analyst firms (ie Gartner) are a big driver of this too. Couple that with the start up / VC model which needs to create new 'categories' to demonstrate differentiation, and you have a total mess.


I work for a vendor that sells a CNAPP. I've worked with this product before and it's been around for several years.

Until last week, I had never heard or read the term CNAPP.

"CNAPP is a term first coined by Gartner in 2021 to describe an all-in-one platform that unifies security and compliance capabilities to prevent, detect, and respond to cloud security threats. A CNAPP integrates multiple cloud security solutions that have been traditionally siloed in a single user interface, making it easier for organizations to protect their entire cloud application footprint."

Thanks, Gartner. What a racket they have as a self-ordained arbiter of market segments.


15 years ago some interns where I now work created a spreadsheet and named it an acronym of their first names. That project has since developed into a fairly important system with a front end and database. They kept the name the same, of course.


A long time ago I worked on a defense program called "XMS 3000", hilariously, it was explained to me that this literally did not mean anything. They needed a name and somebody came up with this to sound cool.


You don't need to know the acronyms to understand the article. Use context clues to infer that the author's point is "traditional tools don't solve the problem I am about to present". They even expand the acronyms later in the article.

Besides, most such acronyms for classes of security tools don't actually mean much. They just represent the current flavor of the week in the security arms race.


> They even expand the acronyms later in the article.

That's the OPPOSITE of how you're supposed to use acronyms. If you're going to spell it out anyway, why not do it the first time you talk about it like you're supposed to?


There is a middle ground, if the author cares enough: the `<abbr>`[1] and `<acronym>`[2] markup tags were designed to solve those problems in an unobtrusive way. It makes folks unfamiliar able to hover-over (or long-press, I believe) the dotted underlined terms to know, and stays out of the way for those who already know them

1: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ab...

2: (evidently deprecated but still parsed) https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ac...


FWIW I’ve worked extensively in AWS IAM for over a decade and have never heard of half the acronyms in the article. Don’t be intimidated.


Crap, I didn't know these acronyms before either and now I've turned into a Gartner shill :sob:


My condolences.


This website is marketing nonsense. I need to understand the technical underpinnings of it. Why does this help me?


If you don't see how it helps you, you probably don't need it. Either because it's not a problem you have or deal with, or because the site doesn't flag up anything with you.


While there's merit to that idea, we don't want to discourage people learning, do we?

Ideally we'd learn about other people's problems BEFORE they become our problems so we're prepared to deal with them. Or we can just want to expand our realm of knowledge.


Managing IAM is a pain in the ass on AWS. I can’t speak to how much this product actually helps with that, but I would assume that that’s the problem they’re trying to solve.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: