Tech stopped being interesting to me when I realized that building a widget isn't hard, it's getting 30 people to build a widget that's hard. Yet the way those 30 people are organized is so shitty it's as if nobody had ever done it before.
I've worked with so many truly incompetent and stupid teams and organizations, that have millions to billions of dollars. Sometimes I wonder if I've just hallucinated all the books and training I've taken that explain the basic business and industry concepts most people should know. And I wonder how it is an IC could seem to know how things should be working while seemingly nobody else does.
Then I go back to my 1/30th of the widget and wait for lunch time.
The larger the team I am on, the less happy I am.
I moved from a 40 person team serving a few dozen clients in a multi-tenant setup, to essentially working for one of the clients directly as their only full time developer.
I have probably written more code in the last 6 months than the preceding 5 years.
It's not because I suddenly got smarter or I am some 10x engineer, it's because all the BS got out of the way.
No agile, no jira, no backlog grooming, no story points, no standups, no sprints, no product owner, no stakeholder management, no churn. No "two masters" or principal-agent problems.
Client/boss knows what they want, and so thats what I work on. It's clear whats important, so that's where I spend my time. He trusts my advice and feedback because he doesn't have 10 other senior engineers whispering conflicting BS in his ear. He is clear on how robust a solution they want, and if it's worth moving onto a new idea or wrapping up the current one.
So, freed from all the other problems, I just build.
I mean, I certainly enjoy that, but I find the challenge of trying to make a 200 people IT org do that a lot more… not necessarily interesting, but it’s something new to do.
Just building by myself has an upper limit. To be fair, that upper limit is roughly 20% of what that whole 200 person org does, but still. I keep imagining what we could accomplish if we reached even 10% of the theoretical limit.
A lot of Fred Brooks' _The Mythical Man-Month_ is about how larger teams are inherently harder to function well than smaller ones, and the larger the harder.
You had me at your first paragraph. But my conclusion was to try to get in the middle of things and figure out if organizing people is really that hard.
Yes, it is. Not sure what you include in "books and training" but the actual hard stuff is the complexity inherent to each individual that you want to organize. Each one of them has moods, goals, desires and bad days. And the number of parameters is huge and they're all different for each individual. And it takes just one member of the group to be sufficiently aligned in another vector to make it all fall apart.
And I am just talking about a team or small department. Figure this at state level and you get why politicians actually don't get anything done.
Not really. It's a handful of common-sense techniques around communication, architecture, and getting out of people's way. There's 3-5 books that compress most of the necessary knowledge into straightforwardly readable form.
Of course, most people haven't read them, let alone applied them. Instead, as an industry, we've converged on having engineering companies led by non-engineers, "scrum masters" galore, and desperate cargo-culting of OKRs instead of having real objectives and real key results. "Top" teams at FAANG companies are routinely having 1980s-style conversations about "staffing projects" while they reassure each other that they don't need to prioritize because "everything is important." (Ask me how I know...)
It's all self-inflicted. I don't know about your extrapolation to broader politics, but back here in the office? The simple task of organizing 5, 10, 50, maybe even up to 100 people? That part is perfectly doable without making any major mistakes. We just don't want to do it.
At minimum, though, I wish we'd stop sitting around and saying it's too hard. Here's Ben Horowitz, writing in 2011 (twelve years ago!) [1] about it:
> If you manage a team of 10 people, it’s quite possible to do so with very few mistakes or bad behaviors. If you manage an organization of 1,000 people it is quite impossible.
If we as an industry could hold the bar higher on running companies/teams when they were this small size, personally I think a lot of things would naturally sort themselves out at the 1,000+ person size. Besides, change begins at home — even just improving the running of your local team would go a long way. Then, even if the executives are flailing around with their latest harebrained take on why they don't need to write rigorous OKRs because they need to run off on yet another vacation, at least your team is good.
Leading a group of a dozen engineers is a perfectly straightforward and learnable skill. And it doesn't "take just one member... to make it all fall apart," because you can just fire that one member. I'm not claiming all this stuff is trivial, but I wish we'd stop making it sound like some sort of mind-blowing skill of living on the knife's edge and wielding secret, arcane leadership knowledge.
(Sorry for the morning rant. I know you meant well, and you're also right to empathize with each person's individual needs. It's certainly a difficult job... it's just not impossibly difficult.)
"Quality, Productivity, and Competitive Position" and "Out of the Crisis" - W. E. Deming
"Workplace Management" - Taiichi Ohno
"Lean Six Sigma" - Jeffrey Ries
But aside from these "management philosophies", there's also standard industry/role practices that companies don't require their employees to implement, and when they do it's only enough to say they technically did it. Often the employees don't even know the standards exist. It's embarrassing.
Jocko's Extreme Ownership and Leadership Strategies and Tactics are also very good.
I would also recommend The Goal by Eliyahu M Goldratt as an excellent primer on the Theory of Constraints as an effective management philosophy.
Most people recommend The 7 Habits of Highly Effective People and other books in that ilk, but I've found far more value in these three books alone than every other management theory book I've read, combined.
And if you go and read some old management cybernetics you see all that stuff back in the fucking 60s, 80 years ago. I think you absolutely hit the nail on the head tho
- Essential Scrum (not because Scrum is the best process, but because this goes through the deep reasons for processes of Scrum, so that you can modify and make a process to fit)
- Management 3.0: Leading Agile Developers, Developing Agile Leaders (pushes a team-first approach that I think supports and reinforces the core concepts of agile )
I feel you! Unfortunately, all those millions to billions allow those organizations to ignore reality far longer than they should be able to.
Just recently I came to a realization, people and organisations exist in a competency / incompetency matrix.
The competent orgs with incompetent people are your blue chips and "traditional" companies. They ship stuff, have customers, grow along with the overall market and economy and are abble to survive most non-existential threats and down-turns and even existential ones from time to time. All, or most, the competency is institutional, recorded in implicit knowledge held by people and various tools and processes. Unless those companies loose that knowledge for some reason, things are good.
Competent orgs with competent people are incredible places, e.g. Amazon. They grow like crazy, rarely screw up and develop. They ship and have happy customers, they innovate. These places are hard to come by, harder to create and even harder to maintain.
Then you have incompetent orgs with competent people. By definition start-ups, those only reach competency when they serve, sucessfully and not depending on outside funding, markets and customers for a while. Doing things right, those orgs can create orgabisational competency based on the people competency.
And then there are incompetent orgs with incompetent people. Places were you think, as you said, you haluzionate all your training and experience. The only thing standing between a start-up (incompetent org and competent people) and such a nightmare is leadership. The thing that allows everyone to pretend to not be obe of those doomed places is endless cash. Sooner or later, those places come crushing down.
And billions makes possible to buy big offices, and suits to appear way more competent than you are :)
I shared room with people probably a bit less skilled at working than min wage clerks I worked with a few years ago. But they got a Java cert so they think spending hours in meeting is work, and efficient.
We should have mandatory non IT work to make people see what it is to work lean and fast.
Not to defend the poster here, but some of the stuff I have seen is truly to the point where you wonder if anybody of those people actually cares about getting things done at all.
As someone who works as an educator at an art school where chaos is the rule, I must say if your company is less organized than a spontaneously formed group of twenty year old crazy-chaotic (and probably drunk) art students, you should probably just close down operations and change your career.
And yes, I have seen such organizations, companies and "professionals".
If you exclude VC money in the near future, bringing in those boatloads requires a minimum of actual competency so. Even VC funding requires that, just in different departments.
I mean, it's easy to get hired into team that's less competent than you as by definition you'd breeze any question they have on recruiting so it really shouldn't be surprising
Then maybe step out of the comfort zone a bit and leave frontend for some embedded real world stuff?
I see your point though. Software has grown so big and detached from the problem that people don't get proper feedback about what they delivered and that makes for a bad quality. In niches it can be better that is why I started that way.
Alternatively, it's demotivating to see how success has seemingly no correlation with functional ability. You can't help but feel like it's all a giant lottery where winners are determined at random.
And in big corporations it isn't very different. There used to be theory of management and of organisation, that somehow got lost along the way, with only vestiges popping up here and there in stuff like the Toyota model, the original ideas behind agile and stuff like team topologies...but almost everything is these ... cargo cult organisations, that just blindly implement whatever rituals this or that business consultant recommends and hope for the best ...
I find Tech extremely interesting but "Silicon Valley" really boring.
The problem I see is that technology innovation got completely confused with finical instrument innovation. This happened in Silicon Valley. Now the whole world thinks Silicon Valley is the heart of technical innovation when in fact it is actually Wall Street Innovation. There is and was good technical innovation in Silicon Valley but I think the tech talent pool has been polluted.
- Don't build product before talking to prospective customers. Don't build your products in a vacuum.
- Pay your suppliers on time.
- Take good notes. You'll need them later when a CEO is breathing down your neck about why his darling (yet useless) feature was removed. It's hard to remember context around decisions that were made X months/years down the road.
I'm not an economist, but I don't believe that's always true. Let's say I can buy replacement server at my own pace for £100, or in a panic when it's urgent for £200 (obviously made up numbers to make the maths easier for me). I can do the former at any time up until the server breaks, when I can only do the latter.
If I invest that £100 instead in another part of the business, that returns £150 a year on average. After two years, that's already £225 - enough to pay for the crisis and still have made £25 profit. If I can get a third year without the server breaking, I'm closer to £140 profit, and after four years I can already afford to have two crises off the back of my initial £100 crisis-avoidance money.
Obviously this is a very simple example, and there are lots of examples where this wouldn't work out. If my rate of return was only 10%, then it would take eight years before I'd have made back enough money to pay for the crisis situation. And sometimes gambles don't pay off and the crisis happens sooner than you expect, and other times people are just consistently bad at estimating these sorts of odds. But the point is that, even if it's cheaper to do something now then later, if you can find something more useful for that money to be doing right now, it might still be more profitable to take the expensive option.
(That said, there's also obviously the human cost - if your team is constantly operating in crisis mode because everything's been used until the last possible moment, then it's unlikely to be an effective team for very long. But then that's kind of the point here: this is all a matter of tradeoffs, and trying to predict the future as best as possible with limited data.)
It is generally good to analyze a situation in those terms, but the frustration comes when spending a few hundred dollars would have saved hundreds of thousands. Alas, those few hundred would have come out of someone’s budget, and thus their bonus…
You may also save money if it turns out you don't need that thing after all, and if you keep the cash on hand instead what you have is insurance - the ability to deal with a new problem you weren't expecting.
Even without knowing the principles I would expect people to stop talking if they don’t have knowledge - but most annoying thing is when people don’t know and are trying to help genuinely and they are convinced they are contributing.
Don't get me wrong. But if you're smart why you don't have your own company or business? Making a successful business in a capitalism atmosphere is not that hard for smart people.
The thing that makes me wonder is cases like yours.
Smart and and yet they are working for dumb or evil people.
Well, I'm smart, but very lazy, so in other words a programmer. We don't like to do work, and business seems like work. It just strikes me as odd that there's few people who are both smart and not lazy.
I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it should have gone and what had happened, and if it's indeed wrong to have gone this way, what to do about it.
I gained two key insights from experiencing this several times.
First, the really useful diagnosis often transcend realms. Eg, what may have occurred as a technical issue was really the result of a process issue, which may have been a result of a hiring issue, which may have stemmed from the personality of the manager who made the hire.
Second, the goal is to necessarily to find the "one true cause" but to drive some improvement. For example, if something was caused by a confluence of factors, if you dove in and improved just one of the factors, you've already made the situation better. You may still chose to fix just the tech or process, but the insight into the deeper causes helped shape our understanding through time.
An especially painful truth in software. Years back I heard a talk by Kent Beck, one of the originators of Test-Driven Development and Extreme Programming. In response to some audience question, he said, "I got into programming because I didn't like dealing with people, and now I'm a @#$&% family therapist for software teams." That comes back to me often.
One aspect that I like to emphasise is the systemic or second order aspect. Yes, doing a retro/rca/post mortem/coe to prevent reoccurance of the same problem is good. Better is to prevent similar problems, or even classes of the problem, across the system in the future. A quick litmus test is "Do the lessons learned and action items apply to the observed problem, or do they apply to similar problems."
Similar to this I look at it through Meadow's leverage points. Does the solution improve information flows in the system? Does it change the time scale of feedback loops? Does it improve material flows?
A solution that improves information flow solves problems you don't even know existed, yet! This is one way to deal with unknown unknowns. (Or black swans, depending on whose terminology you prefer.)
I think of "five whys" more as an example than an actual methodology. The goal in my opinion is:
1. Dig into the root causes. Five Whys is at least helpful in giving the idea that the root causes may be several layers away; you should definitely go past 1 or 2. Untrained humans find it very cognitively tempting to stop at exactly 1; this must be overcome and the Five Whys meme is a good entry point for that.
2. Don't be stuck on Five Whys literally, as essentially a linked list of exactly length five. Causes can be trees. (I mean, technically nothing stops them from being full graphs, at least from a high-level view that ignores temporal dependencies, but generally you need to tree-ify them for this discussion or you'll literally loop around the graph in your postmortem and analysis steps just like a computer would!) A 5-element linked list is merely a particular shape it can have. Management encouraging "Five Whys"-like thinking is a good thing; management mandating that an incident come with a Five Whys analysis with exactly five whys in a linked list would be missing the point in that oh-so-classic management methodology way.
3. Once you have your root cause tree, you examine it with the same engineering/business cost/benefit eyes as everything else. "The entire company's organization is inimical to this sort of quality" may be true, but extremely expensive to fix on its own terms. Even more so when you look up and consider that the company may in fact have other good reasons to be organized that way; there is no perfect organization that solves all problems optimally, and it may be a net loss to even consider re-orging the company to solve one particular thing that in the global context is an annoyance, if it would in the process break the organization that is producing actual value. But you may find a process tweak in some other layer that will mitigate 80% of the problem for hardly more than the cost of a meeting. It is perfectly valid to come up with a portfolio of solutions; one pattern I end up seeing a lot is the cheap short-term bandaid and the longer term fix. Cynicism aside, yes we often do end up with the long term fixes getting put into place... the key there is that they are usually themselves a portfolio of other fixes we needed that long-term fix for, not just this one issue that we've put a bandaid over.
I'm yet to ever participate in an analysis like this that didn't produce some good actionable item. Often at least one of them is indeed quite cheap, even if fixing the root causes is expensive.
The company I work for has a postmortem process that asks for the 5 whys. I'm part of a group that "levels up" other team's postmortems and the 5 whys are always the most difficult.
5 Whys are used a lot of time to shift blame away from the team where the issue originated, or someone does the 5 whys dogmatically and ends in a non-actionable final why.
One of the things I teach is "5" is a suggestion, not a requirement. Trees are great. Each root node in the tree MUST be actionable. Asking people to do better next time isn't actionable. Blame isn't the goal of the document.
> 5 Whys are used a lot of time to shift blame away
> Blame isn't the goal of the document.
5 whys is a tool in root cause analysis. The point of 5 whys is to identify links where "failure would not have happened if" and not act on those.
Blame is a tool that helps dig deeper by aligning incentives. As a moderator you want to find those ifs and blame helps point the finger at perceived causes. The hard part is triage done on blame shifting to assess whether that is actually a cause or just an excuse.
I don't for a second doubt the insights derived from many of the Lesson Learned Retrospective Hot Wash Exit Interview type structured exercises of which this is one of the better, but there is an air of the Maginot Line preparing for tactics of a previous conflict while any ensuing false sense of accomplishment can increase exposure to new risk factors. The exception is if you can identify patterns over time across samples.
If the parent is talking about who I'm pretty sure they are talking about (Bridgewater Associates) it's absolutely not just about investigating production bugs for them. It's about finding the truth of a given situation and working to improve the processes and understanding that led to the problem. You can find out more about the philosophy behind this by reading Ray Dalio's Principles, the published version of which is close enough to the version that everyone at Bridgewater gets a copy of when they walk in the door and is treated as almost a bible[1]. In particular read up about the 5-step process which is what they are talking about here unless I have completely misread.
I didn't work for Bridgewater but I spent some time working at Bridgewater (for a vendor).
Edit to add: Bridgewater's diagnosis process typically goes waaaaaaaay deeper than your standard 5-whys. Like the difference between the doctor just taking your blood pressure and the full physical where they put on the rubber gloves.
[1] Like literally. People will quote principles to you by number if they think you have violated some of them.
I actually don't know how you got to this comment from my narrative. Personally, I deeply enjoyed the ability to surface deep insights into the org and the people based on what superficially may have been a technical or process issue - it gave us much better leverage to continuously optimize the org in great ways.
Yes, if your client is genuinely looking for an improvement.
There are bad examples also, especially in cash-rich companies, who tend to be infested by parasite managers. Those are not interested in org optimizations.
I'll note that he was talking about a hedge fund. It's been a long time since I worked in finance, as I don't find zero-sum games rewarding. But despite the number of assholes in the industry, they were also very focused on maximizing gains, so they could also be very thoughtful and clear-minded. I found that as long as I could explain the value to them in dollar terms, they were very effective at improvement.
> I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it should have gone and what had happened, and if it's indeed wrong to have gone this way, what to do about it.
Do you know of some publicly available material on this? I’d love to hear about their approach.
In big corporations many problems occur because of process/data siloing and impedance mismatch in the inter-departmental communication. Everyone is building their own, no organic intensives to share and cooperate.
Both of those things are good, useful versions of the "holistic" school of thought that engineers/westerners/males are supposed to be bad at, if you believe dumb pop psychology books. It's also a good first step to intersectionality, if you really want to get everyone to try to jump on your head.
I've had times where I stopped the "5 Why's" process short of getting into the human-squishy organizational root causes because I knew the problems were too big to be solved in the context of that particular COE/postmortem. I've also had times where I knew my director/senior manager would've taken issue with the publication of those squishy organizational issues, /even if/ they were true, simply because he/she surmised it wasn't worth burning the political capital for it. (And I've been asked to revise my 5 Whys by leadership exactly because of that reason before.)
I've successfully pushed through organizational changes at least a few times from the output of the 5 Whys process too. If the COE/postmortem revolves around a critical issue or has gravitas, it's easier to put your full weight behind it.
To be honest, my personal hit rate on these is probably about right. If you insist on it every time, you're probably going to burn out on it, personally or organizationally. Sometimes it's OK to try to address a symptom, if the medicine is cheap, easy to administer, and has little to no side effects. And in tech, the symptoms are often inherently layered -- you can often address entire classes of symptoms in reasonable ways, /even if/ that solution is itself working around a deeper organizational shortcoming / root cause.
The level of "5 why's" depends on the level of the person doing it, going as far as its needed to reach that level where you should be driving change.
In the "car won't start" example of the original article, the motor pool mechanic needs to stop the whys at the broken alternator belt, because that is something they can and should fix; the motor pool manager needs to go on until reaching the practice/policy for how often preventive maintenance is done, and the top management (if the consequences are severe enough to warrant attention) need to go on until the "why do we have poor car maintenance" questions reach things like mid-management forcing people to ignore the maintenance schedules, or staffing policy, budget for whole departments, and/or the choice to outsource/insource that function.
But going on much further than your own level is not going to be productive - you can make an actionable suggestion to your boss if some root cause isn't fixable at your level but is fixable there, and that's about it, at least in my experience.
This post links to an article, “People can read their manager’s mind”, which is truly a gem. It argues that employees have the time and the incentive to figure out what their manager truly values, and the employees will then pay lip service to, or even outright ignore, what the manager requires of them - in favor of doing what they know the manager truly values, because the manager will subconsciously reward them for the “truly valued” stuff more than the “stated goals” stuff. The examples are illustrative:
“ A manager truly appreciates original mathematical ideas. The [same] manager requests to rid the code of crash-causing bugs, because customers resent crashes. The most confident people ignore him and spend time coming up with original math. The less confident people spend time chasing bugs, are upset by the lack of recognition, and eventually leave for greener pastures. At any given moment, the code base is ridden by crash-causing bugs.
“ A manager enjoys "software architecture", design patterns, and language lawyer type of knowledge. The [same] manager requests to cooperate better with neighboring teams who are upset by missing functionality in the beautifully architected software. People will tend to keep designing more patterns into the program.”
It suggests a unique and amusing approach to hiring management: executives should find people who truly value the goals of the executive, hire them as managers, and (perhaps via intentional lack of oversight!) give them leeway to let their subconscious values impact their work, thus making their minds legible to mind-reading employees.
> It suggests a unique and amusing approach to hiring management: executives should find people who truly value the goals of the executive, hire them as managers, and (perhaps via intentional lack of oversight!) give them leeway to let their subconscious values impact their work, thus making their minds legible to mind-reading employees.
"Hire people who share your values/goals and let them get on with things" is hardly a novel insight. The trouble is that aligning them is hard.
>”Hire people who share your values/goals and let them get on with things" is hardly a novel insight.
Right, because it’s not the same thing. If you hear “hire people who share your values/goals and let them get on with things” and implement it, you’ll likely end up focusing on hiring developers/engineers and giving them resources, rather than hiring managers and giving them fiefdoms to run according to their whims. You’ll also likely end up hiring people whose true values match your true values and whose stated goals match your stated goals, but this argues that you want people whose true values match your stated goals, and you should ignore their stated goals. Despite sounding similar I think it is really quite different.
(Arguably, this is WHY we find “alignment” so hard: we are constantly aligning their values with our values and their goals with our goals, and we are even achieving that alignment some of the time, and yet things still don’t work - instead we should be trying to align their values with our goals, because this is the kind of alignment that produces results)
I’ve always been bothered by the five whys, simply because it’s such a dramatic oversimplification of cause and effect. The fatal flaw is the assumption that causality is linear, when it is more often combinatorial in nature.
To illustrate consider a drunk pedestrian jaywalking and being hit by a speeding driver who was texting while driving. Who caused the accident? They both did, together. Or maybe one of them did and the other didn’t. Or maybe the stars aligned just right and both would have been fine if they hadn’t been doing it at the exact same time?
I can’t help but feel like anybody who has found the “root cause” for any problem, their answer will reflect their biases far more than will reflect true causality. If you can come up with a single answer for a complex causality problem, something hints to me that you would probably point to the same “cause”for other problems too.
If someone pitched it to you that way, I agree that's a dramatic oversimplification -- not of causality, but of what's meant by the "Five Why's."
"Five Why's" is a tool, a reminder to separate the symptoms from <waving hands> whatever complex system dynamics that caused them. Nothing about that analysis has to be linear nor yield a single answer.
The key lesson is to never solve a problem on the same level it was created -- i.e., the symptoms. Take the time to think through the causality. If you can go five levels up/away from the symptoms and still find an actionable mitigation for a whole class of problems, great! More likely, you go up one level and at least you won't experience the exact same symptom twice.
Agreed. I think some people must take the "5" portion very literally, but that's never how I've done it.
I just recently did a 5 whys session on some production data loss. We ended up with a branching tree that had 22 items and was up to 6 items deep. In the past I've ended up with bigger and deeper ones. To me, the "5" is just a reminder to really dig.
For me it's also a practice that's most valuable when used frequently. As you see similar things crop up in different sessions, that's a great sign to dig deeper. E.g., one of the leaf nodes in our recent retro was "no official process". That's deep enough for us; we'll create an agreed process for this particular thing and put it in the wiki. But if I see that a couple more times, I'll definitely drive the cause analysis deeper.
>I think some people must take the "5" portion very literally, but that's never how I've done it.
My personal experience has been anyone taking the "5" part literally is a cargo cult 5 why'er and they will not produce anything of value from the exercise.
> The key lesson is to never solve a problem on the same level it was created -- i.e., the symptoms. Take the time to think through the causality.
An example of how to illustrate the wrong behavior is -- for software devs -- something like a null pointer dereference. I've seen code reviews where the suggested fix is simply to guard the dereference to check if the pointer is non-null. Sure, sometimes that actually is the right fix. But a lot of time - maybe even most of the time - you need to to take a step back and ask yourself why it was null, whether it should have been and what else must change if it shouldn't have.
There's a common fallacy in your example in that you're asking "who" caused the accident and not "what." This is why blameless retros are important. Whenever you're dealing with who, the responsibility is never binary. There's always some responsibility someone can take for a situation they're a part of. That's good advice for life in general.
The intention of the five whys is to find breakdowns at each level...not just the last one, and mitigate each.
This is why 5 why's is (or should) be paired with something like fishbone diagrams, or why tree's what ever to reinforce that it's not sufficient to regress along a single line - you must try to build out your causal tree out to 5 layers where possible.
Ultimately 5 why's is an arbitrary guideline that balances trying to keep you from being complacent in your analysis, with the need to actually like... stop and make a decision.
Who caused the accident isn't a problem statement. A driver hit a pedestrian is. Sure, from there maybe you have some weird bias against pedestrians, but in most cases five whys would say the root cause is driver inattention. Poor framing of the initial problem is as much an issue with the method as finding bad answers to "why".
Right, and "root cause" only has meaning in the context of some goal being achieved. If you're a detective investigating the scene of the accident to determine culpability, your goal is very different than if you're a civil engineer investigating the accident for ways to improve traffic management. It'd be really strange for either of them to conclude their report with a recommendation to increase World Health Organization's budget for alcohol abuse treatment.
--
Therefore, Your Honor, my client is innocent. Why has this tragic accident occurred? I shall tell you to look no further than the root cause...
> If you can come up with a single answer for a complex causality problem, something hints to me that you would probably point to the same “cause”for other problems too.
That's basically the point of the 5 whys approach. You work up the tree/into the graph of contributing factors based on the specific problem and you end up solving lots of problems. If you don't go back far enough, you're likely just putting a band-aid on the problem and some other variation of it will waste more time again. You get "5 whys" back and now you can really start figuring out what went wrong and how to fix/prevent/mitigate/avoid/etc. it for next time.
If you asked my dad to root cause literally any problem in my life, from my failed marriage all the way to a random nasty smell emanating from my kitchen, he would trace it back to the fundamental root cause that I’m no longer a believing mormon.
I don’t have any problem with the idea that you should dig deeper for any form of causal reasoning. My problem is with the inherent linear flow and singular root cause reasoning of the 5-whys methodology. It is just too easy to lead you back to the thing that you are personally biased to think anyway. If you want to solve actual problems, you more than likely have to solve multiple problems that contribute to a complex system than to solve any singular problem.
I don't mean to be snarky, but if you're looking for a linear flow or singular root cause, you're doing it wrong. It's literally just getting you in the door and past the most immediate symptoms.
If you can work your way back from a smelly kitchen to a change in religion, you ask, "Does anyone who didn't change religions have a smelly kitchen?" Obviously the answer is yes, so flag it as possibly correlated and move on to investigate factors that have a better chance at causality and aren't so easily dismissed. Or maybe you need to go even further back, and there's some link between why your kitchen smells and why you changed religions. It's just an approach, like therapy, not an algorithm.
> I don't mean to be snarky, but if you're looking for a linear flow or singular root cause, you're doing it wrong. It's literally just getting you in the door and past the most immediate symptoms.
If you read this thread there are literally multiple interpretations of what the 5-whys actually means, each of which results in a roundabout No True Scotsman argument. Including yours.
I’d prefer to see the 5-why’s as what it actually is: an aphorism about not treating symptoms but rather problems. The aphorism is useful…attempts to define it as some kind of formal tool are a waste of everybody’s time. There are plenty of tools (such as Ishikawa diagrams) out there that represent a better implementation of the aphorism, and 5-whys can be replaced entirely by them.
Is it actually that? I'm pretty sure that the term comes out of the Toyota Production System, in which kaizen, or continuous improvement, is the heart of the process. Given that the Ishikawa diagram also comes out of post-war Japan's industrial process quality improvement community, I would be very surprised if Totota didn't make use of Ishikawa diagrams in practice.
In all the five-whys investigations I’ve been a part of, it’s always been accepted that each node can have more than a single cause, and you try to explore as many of them as the time allows, and provide betterments for as many of them as possible.
I've always looked at these exercises (like the NTSB reports) as trying to find as many possible ways that this could have been prevented - not trying to find the single most proximate way to prevent it.
And in doing so, you likely realize that you came close to disaster a number of times and never realized it.
5 why's refers (afaik) to the depth of the why tree. If the first why has 3 answers, then you ask why 3 more times to each answer (n = 4, depth = 2)
If each of those 3 whys have 2 answers, you would ask why 6 more times (n=10, depth = 3)
If each of those whys have 3 answers, we would need to ask why a further 18 times! (N=28, depth = 4)
That is leaving out the last depth, but suffice to say there are a lot more than just 5 questions. It seems rare for any system where the cause and effect tree is linear. Hence, 5 whys is meant to be very holistic, not just deep.
Sure it’s combinatorial in any specific instance, but you can still identify how much each individual root cause contributed to the final outcome and then alter outcomes in aggregate that way.
For your example, if texting and driving is associated with 80% of pedestrian collisions but a drunk pedestrian is only associated with 5% of such collisions, you can probabilistically influence most potential outcomes by focusing your effort on the texting-while-driving root cause.
All models are false, some are useful. Five whys strikes a useful balance between the overly simplistic single cause model and the incomprehensible detail of full reality.
It's funny how humans try to simplify things to black and white all too often. Why do we do this? I'm sitting here trying to think why I do it, and I can't figure it out. Something maybe about needing someone to blame.
For example, when I think of the 2008 financial crisis, I always feel this need to find one group to blame, but really there wasn't one group.
Five why's states that if I have to ask "Why?" more than 5 times to get at why something is the way it is, it needs to be simplified, in the sense that lack of simplicity/excessive cognitive load is in and of itself, a form of defect.
Because it isn't. 5 whys is for exploring the neighborhood of the problem without going deep down a rabbit hole. It's like timeboxing an activity or putting a bounds on a search algorithm. The objective is to find something near enough to act on and partially [0] address the systemic causes, but far enough to not just be the proximate cause. "Battery was dead" is a proximate cause, many people stop at proximate causes. 5 whys forces you beyond that.
It should also be combined with other techniques that elaborate, more properly, on the model of the system in which the problem occurred. There you will find the deeper systemic causes and can start addressing and mitigating the issues. But that will take a lot longer to explore and resolve than the 5 whys answer of "Get a proper tuneup every 5-10k miles".
[0] Ideally fully, but 5 whys is an expedient, non-thorough technique so that's a reach.
No, you have it exactly backwards. Five whys is explicitly depth first search. It was originally employed by Toyota Motors, and the key to actually getting at what the actual causes of issues were. This allowed Japan to avoid entire classes of defect due to their acceptance and implementations of the teachings of W. Edwards Demmings. It never made the leap to the United States, because doing things right is such a foreign concept over here, where management would gasp at the concept of a single worker shutting down the entire factory upon detecting a problem, whereas in Japan, the andon was key to getting issues that would compromise the Quality of the end product addressed.
It's:
I have a problem,
Why? Because of this other thing.
Why? What causes other thing?
The other thing is dependent on yet another thing and that thing.
Are those things a problem? If so, why?
Iterate until done.
The "Five" aspect came into play in that after a certain number of "Why's", it was often the case that the fact it took so many Why's to get to the root cause, that the complexity itself constituted it's own type of defect.
I might need to revisit the book I first read of it in, believe it was titled "Elegance" or some such.
EDIT: Found it.
In Pursuit of Elegance: Why the Best Ideas Have Something Missing by Matthew E. May
EDIT2: Nope, false alarm. Drat it, I will find that reference.
It's in this library somewhere... In the meantime, forget it; maybe I'm hallucinating it, because clearly I haven't refreshed it in a while, and the old wetware has taken a hell of a beating over the last couple years.
I didn't say it wasn't depth first so I have no idea what you're responding to with this. Can you point out where in my comment I said it wasn't depth first?
I just said it's bounded, because it is. Even the link you've posted states that "why?" is only asked 5 times. But even if you take the 5 as suggestive rather than a strict limit, giving a limit (even a soft one) implies a boundary to the search. If it was meant to be unbounded it would just be called "Whys". 5 pushes you beyond the proximate cause and towards the actual root cause (of course, "root cause" is itself a poor phrase as it implies a singular cause that can be addressed to prevent the problem from reoccurring), but then the root cause could be 10 or 20 or 30 whys deep. So unless you go beyond the point suggested by the name and publications about it then you won't find it.
------------
EDIT: I see you edited your comment after I hit reply and didn't mark it as such, good form.
Anyways, about it being depth first I'd actually disagree. Even if you believe that it should be taken as an unbounded search (which I also disagree with, obviously, because there are better tools than 5 whys for such a major task) it should not be an unbounded depth first search. That would be a form of malpractice.
The more accurate description would be something like iterative deepening. You go down a few layers with something approximating a depth first search, and then you back track and try the other branches. If you go down one rabbit hole arbitrarily deep you may or may not find a causal factor but you will have missed many others along the branches you chose to ignore for the sake of your unbounded depth first search (again, a form of malpractice).
This is a common failure in authoritarian organizations. We'll be in a much better place when Corporate America either adopts or is forced to adopt a more egalitarian way of management.
This is one of the major reasons why I chose to make less money working at a smaller company, as it isn't big enough to accommodate these "reality distortion fields".
I keep wanting to read Moral Mazes, but unfortunately I've yet to get to it.
In a more moderate version of this, I believe to be powerful also requires one to be responsible (among other attributes). I won't go into the details but I've found this to be also true in my personal life, when I've been in both the powerful and powerless roles.
In the context of big corporations, where I had experience working at, I often recall Aesop's animal tales.
The animals have bought all the expensive shiny musical instruments, in order to build an orchestra. But nobody knows how to play (and how to play together). So the only thing they can adjust is the seat order arrangement, stings to the left, brass to the right.
One of the companies I worked for had licensed all possible expensive applications, databases, analytical tools. But nobody knew how to make use of those. There were many initiatives, but all efforts were spread thin.
Perhaps in German this would be called Gießkannenprinzip (the water sprinkler principle)
Yup, and for this reason, I will never work on a compliance app again in my life.
Compliance is all about reducing risk to an acceptable level, but if you care about it, it is also about reducing risk to as close to zero as possible. Compliance generally affects multiple organizations, and that means convincing a bunch of people to do something that doesn't help them accomplish their primary goals.
To be clear, I'm not alleging any wrongdoing whatsoever against any group that I worked with. I'm simply saying that it's hard enough to get people to do the job they accept as their main job; adding 'extra' work is never a rewarding experience - especially as an individual contributor.
...yeah, there are several roles/functions that are by definition thankless tasks. In compliance, the best you can produce is nothing, as in no incidents that could have been prevented through better compliance, while causing a ton of cost and frustration across the organization. The worst you can produce is ruining the company and sending people to jail in a way that you were tasked with preventing and could actually have prevented.
It sort of reminds me of this thought experiment that Nassim Nicholas Taleb mentions in one of his books: What if a bureaucrat had figured out, pre 9/11, that terrorists flying a commercial airliner in to a building was a risk worth trying to prevent and had passed legislation for all cockpits to be bulletproof cells that could only be opened from the inside, or something like that. 9/11 would have never happened, and that bureaucrat would have caught shit for the rest of his life for championing legislation that is causing a lot of people a lot of cost and frustration and has never had any observably positive impact anywhere in the observable universe.
If you go below the "tech whys" and into the organizational whys, then you are thinking at the team leader or VP Engineering level. I think you want to be eyes opened to the full list of whys, but be happy that some cannot be changed. After all you are hired by a business, run by people, not a perfectly functional institution.
This is about the logic of sufficiency and necessity, and it requires the skill to identify narrowly-scoped syllogisms for each step of the causal graph. For anyone really interested in this stuff, read about Theory of Constraints "Thinking Tools", and supplement with a smattering of simple philosophy, such as Hume's Guillotine and the Munchausen Trilemma. Basically, for any examination of "why did our actions result in this outcome", you're going to come down to a collection of premises, some of which are facts and others of which are values. Many of these values are obvious, widely-shared non-negotiable values, but others are slippery biases or assumptions that have snuck into the system, and focusing on one of these is often identified as the "root cause", even though there are by definition several; any one of which was sufficient to cause the outcome. Really "root cause" just means the identification of one value that could and should be changed, which thereby will propagate its changes throughout the causal network.
I will say though, that this is a difficult skill, and it takes practice, because there are so many personal biases that can creep in and lead to a corrupted causal analysis. It's also a bit thankless because once you come up with a decent root-cause analysis, it reads so clearly that it basically comes across as obvious and common-sense to anyone that reads it, leading them to underestimate just how much work it took to get there.
I like to think about this sort of stuff in terms of strategy. The definition I use is 'strategical goals' are the things you are trying to achieve, and 'strategy' is the 'why, how, what's the rationale for these, when do we revisit the goals and the strategy, etc.'. Strategical goals can be high level or low level. In some cases you can take part in a big picture company strategy process. In other cases, the strategical goals you have as an individual are to support the goals/ways of working of your boss, and the other people you work with, which you can only comprehend in a very blinkered way.
In all these situations, it seems like some of the time, and for a critical part of what you do a lot of the time, you have to have strategies that take into account other people having hidden motivations, and particularly people who don't think strategically and cannot be reasoned with on the basis of strategy, or they refuse to be open with these sorts of things with you in particular, or will be evasive, or say they will do things and then not do them, etc.. A lot of the time you cannot fix these things, you can only take them into account and/or find workarounds. And it seems reasonable to say that this is the part of the picture where you have the most leverage to make a difference, or to reduce things like the amount of bullshit you deal with, or the amount of wasted effort you put into things.
You can also have the insight that other people will see you the same way - as someone who is obstructive, or fails to be clear, or isn't persuing the right goals, etc..
I guess you just have to ask why people have these motivations and why they hide them.
It feels like people would act dishonestly first to seek survival (job security), but then also to seek power (promotion/influence).
I wonder what the relationship is between survival and power. I guess at some point survival manifests into the desire to hold onto power. And the stakes get larger as the power accrued increases.
I also wonder how the human reward schedule plays into it. People are motivated by consistent rewards but at random intervals. If you take out the corporate power politics, you remove the randomness of the intervals, and it perhaps becomes less motivating. The chance for an early promotion provides motivation to play the game harder.
> I guess you just have to ask why people have these motivations and why they hide them.
> It feels like people would act dishonestly first to seek survival (job security), but then also to seek power (promotion/influence).
The first place I go to is to see that people aren't always doing good strategical thinking, then they get hung up on mistaken goals, or ineffective ways of achieving them, and aren't very good at communicating what they are doing or hoping to achieve, and this often leads to what looks like subconscious (or sometimes very conscious) obfuscation.
In the technology area, I've worked with many coders, through to product people, sales, marketing, directors, etc., who get hung up on particular technical ideas - and claim these are strategic product or architecural decisions, and the same happens with people getting hung up on ineffective project management - especially when it's an approach that worked in another situation much better than it's working in the current situation.
So it's not about being good at strategic thinking. It's about motivation, boredom and self-discipline.
Here is what you are not seeing:
Dev sits down to work on an assigned task. They might be on the ADD spectrum, and the work feels incredibly boring and their mind cannot focus on it. They are interested in novelty though. This provides a rush of motivation and hyper-focus. It will usually involve the choice of some cutting-edge new technology or architecture. For someone without ADD, it's hard to understand the intense pull of this pursuit of novelty. Tugging at you constantly.
The problem is this motivation is derived from something that probably doesn't align with strategic goals of company. They will have super-human 10x levels of motivation. So they try to push the company in the direction of this novelty. They are just following a high for weeks on end. Eventually this high will run out though and the need to show progress towards business goals arrives, and they can't hide behind "rewrites", "tech debt", "unanticipated problems with new architecture", etc.
So then the survival kicks in and the blame game starts with management.
I think some of this is true for almost every developer.
So when I read your complaints they are all easily explained:
> hung up on mistaken goals...ineffective ways of achieving them
They want to keep working on the novel thing that interests them. They try to shape the goals that allow them to do this. This can be subconscious.
> aren't very good at communicating
Because they have become too side-tracked by this novel thing or the next novel thing.
> looks like subconscious obfuscation
Yep. Exactly. It is often subconscious. To confront the reality of how far you have strayed from the strategic goals of the company would be like being out for a night drinking and at the height of the party, you have to leave and go immediately to bed.
> Hung up on particular technical ideas - and claim these are strategic product or architecural decisions
Just chasing dopamine.
> people getting hung up on ineffective project management
Survival. You went too far off track and need someone to blame.
---
I think if you view developers through the prism of chasing dopamine highs from the pursuit of novelty, you can much better manage them.
The only thing that can tackle this I think is having hard, concrete, and unavoidable deadlines. The survival system overpowers this creative exploration. But this usually only kicks in a couple of days before a deadline.
Your strategic goals are essentially always competing with emotions of a developer towards some other thing. I think a lot of people underestimate the mania feeling that comes from the other thing. So you need to make the the dev believe the current strategic goals and the achievement of them is really, really, really important. This is why visionary/bombastic founders and so important.
What could also help is allowing everyone to be open and honest about their true motivation and goals. But it would be hard for anyone to say: "I am only interested in playing with this new tech, and have no interest in the company". So there will always be a level of concealment. And usually, revealing such things will mean they are monitored more closely which they don't want so it doesn't make sense to reveal these things...or they will feel like they are being monitored which increases stress.
> I think if you view developers through the prism of chasing dopamine highs from the pursuit of novelty, you can much better manage them.
I think the crux of it for me is that you cannot really ever know someone else, nor can you try control them like a puppet and it work. So you have to accept that you will often fail at trying to manage people in situations like this (and to understand your bosses, etc.), and you need strategies that work in the face of these factors.
But from another perspective, I'm not all that convinced about viewing developers the way you say, trying to indulge them then find a way to use the output. My alternative is to spend more time giving people good context - a programmer may be less inclined to chase their own technical novelty compulsions if you can give them an understandable and non-fake view of e.g. specific sales processes that need to be supported by development and what the significance is, company strategy, what the operations team is dealing with, etc. One of the simple older ideas in this area is having developers sit in on sales/customer meetings.
So I guess this is a kind of argument that alienation is part of the story of why technical people can often be totally unaligned with their company's goals and this is a barrier to improving this.
This may be a perspective coloured by my experience mostly working in small companies, not sure how well it translates to big ones.
To me, the real layer-8 in a car model is that you probably shouldn't have built out society to depend on cars in the first place. Thats why we have the idea of the 15 minute city: places you can manage without private transport.
Applied to s/w this would be "do we really need to make this system, in the first place: How about if we decide not to"
Yes. This is a system problem. Why are cars allowed to be near pedestrians? Why were cities/streets designed to make it possible for cars to run over pedestrians?
Saying it was an inattentive driver misses the point that we can't expect fallible people to not make mistakes. We should be designing systems to minimize the damage from inattentive drivers and drunk pedestrians
I've struggled to take my teams (in the past: I don't do PM now) beyond "ship it" to "kill it": the C-suites don't like the idea of stopping doing things. They have a point, we maybe don't have community permission, but we should at least entertain the idea (in the not for profit sector)
People rei-ify things. "oh we need to do X we've always done X" when in fact, you could say "doing X is stupid"
Some people like writing code. Some people like acquiring power. The latter will tend to consolidate in management positions, the former will tend to wonder why incredibly obvious technical and social problems are being ignored
An excellent write up of the changes in perspective which occur with experience. No organization is optimal and neither are we. The key is to incorporate these details into planning. For example if the organization can't set durable goals, don't start a 3 year plan.
Don't passively accept it, or become fatalistic about dysfunctions. If there are achievable changes to make, try to make them within your sphere of influence. Do what you can to produce positive changes, but don't allow frustration to lead to cynicism. Think of layer eight problems as engineering constraints like any other.
Humans have always had the ability to mess up a one car funeral. The kingdom building manager has always been the problem where I work. He will hire those below him who are woefully inept, so he can insulate himself from his own errors. Those of us not in management, who actually know to job, are ignored. This causes an incredible amount of waste and time. And yes, if you try to go above his head, you will be scapegoated out of your job. One can imagine what this does for morale.
> That might lead into "the entire company is obsessed with hiring even though the tech equivalent of the Drake equation says there is no way they can find anywhere near that many qualified people in the entire world.
Fortunately, we don’t have to characterize this equation to quantify bloat in the corporate structure. We just have to wait for layoffs in the next recession. Unfortunately, this is a lagging indicator.
This is actually a legitimate cause, though not always expressed that way ;)
Many problems don't get solved well (or at all) because nobody in the organization finds them interesting.
The only way to fix this that I've seen is to change the composition of the team and/or change the composition of the problems. This is why so many startups struggle, they have a small team and eventually encounter problems nobody is energized by solving and they can't hire someone who is interested. Eventually people burn out doing things they hate and it all falls apart.
The follow-up question should be along the lines of "what could make the right choice easier/the lazy choice?" Often you CAN actually make it so that fixing the issue IS the lazy option, but you may need the right skillset applied to it. E.g, manual process that could have been automated it someone with automation skills had been available.
Pedant: If your "alternator" belt is broken, you are likely not going to have a dead battery - you probably won't be able to steer, and you're likely not going to have a dead battery since you are not driving/using the car since it will also overheat very quickly.
While I find the idea of perfect debuggability appealing, i.e. that I can trace problems back to upper management or the CEO - I think there is way too much entropy involved to do this in a meaningful way. Remove manager M1 or M2, and things might actually get worse. Why? You’ll have no clue.
> The car didn't start... because the battery is dead... because the alternator wasn't charging it... because the alternator belt broke... because the belt was beyond its useful life but wasn't replaced... because it wasn't maintained according to recommended schedule.
This isn't "five whys" except in the most superficial sense. It can be useful to do this to build reliable technology but it isn't what five whys is trying to solve.
> because the battery is dead... because the alternator wasn't charging it... because the alternator belt broke... because the belt was beyond its useful life but wasn't replaced...
All of these are a single "why" (call it "technical faults" or something).
> It turns out that when you start doing this root-cause analysis and really keep after it, the "squishy human realm" is actually the no-longer-hypothetical "layer eight" from those T-shirts.
Is this actually a revelation to anyone? These hard questions are exactly what five whys is designed to address.
Sometimes though if you track the problem all the way to layer 8, the origin of the problem has "resolved itself" i.e. left the organization, fired, retired, died, etc. Then you can solve the issue with no pushback.
I have a socket wrench. It's a really good socket wrench, it would probably cost me close to $200 to replace.
I use it for nearly all of the bolts and lags that I need to loosen or tighten.
The bolts on the spindles for my tractor deck are massive, they rust up quickly, and my $200 socket wrench doesn't touch them. For those spindles, I have a special tool that can solve the kind of problems that a $200 socket wrench can't solve.
Despite my socket wrench not solving all problems, I'm still really glad I have it, because it is helpful most of the time, and I trust myself to be smart enough to know when I need a different tool.
I've worked with so many truly incompetent and stupid teams and organizations, that have millions to billions of dollars. Sometimes I wonder if I've just hallucinated all the books and training I've taken that explain the basic business and industry concepts most people should know. And I wonder how it is an IC could seem to know how things should be working while seemingly nobody else does.
Then I go back to my 1/30th of the widget and wait for lunch time.