Creo (based in Vancouver, BC) used to be a company that tried to address this. The concept that was used was called "unit presidency". Each employee was empowered, expected, and trained, to make decisions as if they are the president of the company. The principles behind making decisions were called "economic thinking" which the CEO used to say was everything he learnt in a Harvard (EDIT: or maybe it was Stanford) MBA distilled into the core principles. Basically looking at the ROI (Return on Investment) of the decision. Decisions were generally made by consensus though depending on the nature of the decision sometimes other methods were used. This extended to decisions that involved spending money, not just should you pick language X or language Y for your next software project.
I think it worked pretty well for quite a few years. It gradually stopped working when the company acquired a large company with a different culture and also hired people (well, managers mostly) who weren't aligned with the culture. Eventually this basically disappeared when the company was acquired by Kodak.
I've seen flavors of this in other places. Famously Andy Grove of Intel also preached that decisions need to be made by those closest to the decision and empowered people to make the right decisions. More generally this can be reflected in a servant-leadership model where leadership sees itself as facilitating the growth of the people underneath them.
Another requirement for this to work well is that management (e.g. the CEO or other leaders) are able to lay down a broad strategy for the people of the company to execute on. If the leadership has no strategy then tactical decisions can not be made properly. They also need to make sure there's coordination and structure.
[Edit: apologies for the wall of text. A lot of pent up emotion around Creo.]
Best place I ever worked.
This was “the Creo Philosophy”:
Our priority is to provide unique and sustainable value to our customers.
1. All decisions must be based on sound economics.
2. Key decisions are made in consensus, with full team agreement to accept and implement the decision.
3. We believe that people are most effective when self-managed.
4. Compensation is based on contribution, gauged largely by an annual peer review.
5. All employees share the wealth created by their hard work and innovation.
Creo was extremely proud of how “flat” the organization was but my primary take-away from there as an ex-employee is actually how important management is.
First and foremost, management created a shared vision of the future. Not silly posters to laugh at but a real shared mission. It was exciting and motivating. Every Creoite knew how the world would be changed when we were successful. Decisions could be distributed because it was obvious which outcomes were aligned with the companies goals. Trying to push outcomes not aligned with the goals was hard ( as it should be ).
Second, there was a strong framework for how decisions were made and how to identify good decisions from bad ones. Economic thinking and consensus were two important principles that you were expected to follow unless you could demonstrate very good reasons not to.
Third, management provided a great deal of mentorship, both through direct education and through calling re-enforcing the primacy of the principles. One of the reason Creo could be so “flat” is because everybody knew how the executives would behave if they were in the room and they could insist that others act accordingly. It was easy to assume executive sponsorship without having to resort to politics. “We do not tolerate politics” was an important mantra through the best parts of the company history.
Most importantly, I got to experience the everyday effectiveness of the organization under three different CEO’s: Ken Spencer, Amos Michelson, and Mark Dance. The “culture” was only as good as the man at the helm.
At Creo, we made the claim all the time that we basically did not have hierarchal management. It was true we did not always have “supervisors” but we had the best management I have ever worked for.
As said above, Creo stopped working when it acquired a rival that had more employees than it had. The politics exploded. The effectiveness disappeared. Financial performance followed. The company was sold to avoid a shareholder revolt. A sad end for a spectacular organization.
The original Creo employees stuck to the principles. Well, except management. Hierarchical authority was suddenly more important. Decisions did not have to make economic sense. What mattered was who was doing the deciding. Consensus stopped being a requirement. Speaking truth to power stopped being a path to better decisions and started becoming a career mistake. Executive sponsorship became real and aligning with people in position became the most important criteria for individual success. Empire building replaced shared vision. The lack of alignment between making the company successful and being successful as an employee completely broke. The company failed.
Managers and employees looked at Creo folks and their “philosophy” like naive children. Sure “the philosophy” worked well enough for Creo to beat them to begin with but, once merged, the acquired were right. Creo was idealistic and naive. But this was not inevitable.
Why did Creo fail? Did “the philosophy” not scale as the final CEO I think believed? I do not think so. I see it purely as a failure of management.
The final CEO provided little vision. Other than a focus on the bottom line, there was no insistence on economic decision making ( eg. instead of breaking up meetings because they had too many people in them - too expensive - he held meetings with dozens of people in them ). Instead of coming down on politics and insisting on consensus, executive authority became sacred. In fact, the term “consensus” became most often used when an executive used it as an excuse for their own lack of leadership by claiming the team should have to solve its own problems.
If strong management had provided the leadership, the mentorship, the vision, and support for the culture ( most importantly the principles of good vs bad decision making ), Creo would still be with us. If I was lucky, I would still be working there.
Creo ruined me in a way. I have never been able to accept why things cannot be as good anywhere else. Such simple ideas. So dramatically effective. Unfortunately, good management ( executive level ) is an absolute requirement. Even hundreds or thousands of well trained employees was not enough to make these principles work without strong management behind them. Management matters.
I got to experience some amazing leadership for a while. All I can do is try to emulate those role models as best I can. And everywhere I go, I try to make economic thinking an important part of how decisions get made.
Author here - I'm simultaneously quite pessimistic about, and very interested in heterodox organizational structures and especially real-world stories about them. I feel that failure or regression to the mean are quite likely and scaling or replicating success stories is very hard, but I am almost certain things will evolve beyond the current status quo eventually, just really not sure when and how.
So thank you very much for sharing and recommendations for further reading will be much appreciated!
I put together a "High Performance Organizations Reading List" which includes a reference to "Reinventing Organizations: A Guide to Creating Organizations Inspired by the Next Stage of Human Consciousness" by Frédéric Laloux. You and "LeFantome" who wrote about Creo might find that book and other resources there of interest. Laloux's book provides examples of various companies with better management.
https://github.com/pdfernhout/High-Performance-Organizations...
Better and healthier organizations are possible, as Laloux writes about. They are rare though -- and difficult to sustain in our current Western society. As with Creo, you definitely need enlightened management or enlightened major shareholders to "hold the space" as Laloux writes.
I have a theory that organizations that grow fast and scale well all have this “cellular model” at their core.
Investment bank trading desks in the pre-2008 era, partnership at the big strategy consulting firms and even “multi-strategy hedge funds” now are actually all collections of very incentive aligned businesses. They share the Creo quality of making lots of millionaires and people looking back on their time there as one of great freedom and achievement.
In all these places, employees are paid according to the revenue they generate, with seemingly no ceiling to what you can take home.
It is true that the size of any one cell doesn’t scale beyond a small number of people. But all the organisations I mentioned above scale by having units tackling small pieces of vast markets.
The main lesson I took away from reading “Barbarians at the Gate” is that big companies hugely suffer from the principal agent problem, where management is mostly out to enrich themselves at the expense of shareholders and employees (sometimes). This looting is however only possible at a company that was established by a founder with a deep vision and passion for the product and has set up systems and culture that generates sufficient cash for the professional management to leech off.
What I have not read yet is a systematic study of these “cellular organizations” and what the common features are that make them successful. My guess is that the key is that each “unit” or “cell” has measurable economics that makes it possible to share the economic value over a sustained period of time. A bit like why sales people get paid a lot.
Agreed on cellular, but life is not a picture, rather a movie… what works at a stage starts attracting people, and eventually you let the wrong people in. A bit like the hype curve. These wrong people start poisoning internal processes and culture, seeking to cash out. And then the model blows up.
The migration of shitheads from Wall Street 80s 90s to Silicon Valley - technobros is for me a solid example of this.
Indeed, an organism can only survive long enough if it can resist infection. For this, an organization should be openly hostile to certain forms of conduct, and should have a way to expel people who bring in wrong values, especially in management.
This is a really hard problem, due to its influence on the morale, and the danger of weaponization of these mechanisms by bad actors.
I also worked at a company that was similar to the Creo story above.
This was a High Frequency Market making company, total employees ~ 250. "Flat" management, nobody had titles, just responsibilities. Everyone understood the mission and goals. Everyone was highly compensated and empowered to make important decisions without explicit approval.
The whole thing fell apart when the company grew and finally failed when merged with another larger competitor.
These great orgs only last when they are kept small.
> I feel that failure or regression to the mean are quite likely
i suspect that these failures are a failure for humans to adapt to a larger herd than the traditional tribal size - anything beyond a couple hundred people max.
Have you ever read the book "Loonshots"
There's an entire chapter about how primitive human societies probably maxed out at about 150 individuals. Companies that exceed this headcount basically need to change themselves into highly hierarchical organizations, and many companies die in this transition.
I went through this stage with a smaller company and there's definitely this point where you no longer know everyone and things change. Creo however did manage to scale this well beyond 150 people. You don't necessarily need a lot of hierarchy but you do need to figure out a structure. Creo always had a hierarchy, it wasn't like Valve or something. It's just how things worked within that hierarchy.
I experienced this as two phases of growth in the path from ~100 people to ~15000 people: the first phase where you don’t know everyone and the second one where you don’t even know every department/major project.
150 is Dunbar's number, made by extrapolating a monkey species tribe size / brain size correlation to human brain size. Something interesting to think about, but not exactly hard science.
The book Loonshots says as much. Extrapolating from monkey brain sizes to human social interactions is a bit reminiscent of phrenology IMO, but it is an interesting observation. Anecdotally, I've found it to hold true, particularly in startup land when companies transition into blitzscaling hypergrowth.
At Meta, we collected tons of real world data to pretty definitively prove this is true. Humans can only really relate to about that many others a time. And most people have about 5 others that they are actually close with.
The observation that there is a (very) strong statistical correlation between cranium and social size is an interesting and correct observation.
That much is true and uncontroversial.
What the article takes issue with is the exact number of 150, which can vary from person to person. But really, the criticism entirely misses the point. The number is just a rule of thumb derived from extrapolation. It's not a hard and fast rule like "You can only have 150 friends". Obviously.
Perhaps it should be something like cranial volume / body volume, or else we'd expect whales and dolphins to have massive tribe sizes, yet they typically hang with something like 20 peers.
Unnecessary hierarchy is a cancer. CEO's aren't geniuses or rockstars, and if executives can't convince their employees- the actual technical people who will implement whatever plan is- that a given plan is the right one, 99% of the time that means it's not.
Good management is about leading people (i.e. who follow voluntarily), not dragging people along.
A very important distinction, in my "Creo" example above, our company specifically used the word "leadership" for the founders and other "leads" of the company.
Nobody used the word manager or boss annd anyone could be in a leadership role, it was based on meritocracy.
I think what's interesting here is that you must have had access to the raw financials (eg. sales numbers, P&L by department). These are very tightly controlled at all companies I've worked at, making it hard to know if a particular product is important to the bottom line or not.
This is also a great point. Teams should have detailed knowledge of the performance of the company or groups. It helps with the shared ownership, pride and responsibilities.
Yeah, I'm in a company that has transitioned to an ESOP. Management haven't figured out that financials have to be open, yet.
They are begging people to engage but I have no idea what projects are paying the bills, which ones are drags, and various other missing information.
If you keep all that a secret people have no choice but to tune out and just expect management to handle everything.
I don't think the current CEO wants anyone else loudly expressing concerns over the future of the busieness as we are trying to shift away from contracting to SaaS. This is an 80yr old company, mind you.
Now we are trying to grow and bring in employees (for a contract), promising them stock and supposedly stable employment but in the end most of the dozens of employees who are brought in will be let go before they vest. The long term base employees (there is a cache of 20+ yr employees) will have their stock price increase after these people (they will let go) put in years of work to deliver on record-breaking contracts. None of this is the stated intention but it's going to go down like this. About that time the CEO will retire and cash out.
Spooky parallels to the Boeing and McDonnell Douglas merger. Thanks so much for sharing the history, great ideas and suggestions, and a lasting question to answer around how to defend the model from the barbarians.
It almost looks like the culture is the competitive advantage, and everything else follows. No market and technology synergies are important enough to risk the destruction of that culture in a merger. (Alternatively, all the upper and middle management of the acquired company has to be honorably discharged, and the remaining employees are to spend several months training in the new culture. A pretty hard sell for most mergers.)
> Creo stopped working when it acquired a rival that had more employees than it had
Huh. That'd make sense. From the outside, Creo Scitex looked like an unstoppable dreadnaught.
Creo acquired ScenicSoft in 2002. Alas, I saw no evidence of economic thinking.
Whoever was in charge of software was proud of having never canceling one of their ~dozen products. Me asking if there was a P&L for any of them was among my top 20 most brave/stupid work related selfowns.
They did cancel Color Central in short order. (Payback?) Basically CUPS with features for print manufacturing and a purdy UI. Super boring, but a necessary product. Although no longer affectionately known as Cash Cow (the internal code name Illumious had given it), it still had a huge installed base, had steady revenue (upgrades), and maintenance was on auto-pilot.
About a year later, Creo licensed Color Central's biggest competitor. (Sorry, forget that company and product name. I remember that they were a great team though.)
Rowan's UpFront was part of the ScenicSoft acquisition. Advanced print manufacturing planner. Pretty much unparalleled. A true diamond in the rough.
Further, I had invented a novel imposition engine for bookwork. Specify all the sizes and (crucially) the desired binding -- voilá, out comes a Preps template production plan. (Preps was ScenicSoft's flagship product, the gold standard for print imposition.)
Automating bookwork planning was the holy grail. The long sought replacement for Preps' very complicated, very necessary template editor.
I don't think the Creo people understood what they had. They were so focused on their (awesome) digital plate stuff and adjacent. Like most, they probably treated the bindery as an afterthought.
I forget how Creo managed to let Rowan slip thru their fingers. But don't worry, he did it all again and earned a pretty good exit the second time around.
--
Whatever Creo's shortcomings, the president and owner of ScenicSoft totally failed. After many rounds, he managed to negotiate Creo's initial $70m offer down to $9.5m. So saavy.
Interesting. I worked on output devices (and some very specialized ones at that) so I didn't really see a lot of the "software" side of the business. But I have seen some "not invented here" and rivalry between the Creo and Scitex teams. Part of the problem there was actually maintaining parallel products and teams that targeted the same markets.
Acquisitions are just tough. There were good people with good intentions and teams did try to work together. I spent some of my time trying to bridge the teams, with some success. What was missing is a good idea of how to restructure things to eliminate the duplicate efforts which is super tough when you're also dealing with large install bases. There were a few technical attempts to bridge the architectures and bit and pieces from "Creo" machines found their way into "Scitex" machines.
Thanks for sharing, you're post really resonated with me and I want to learn more about Creo now.
While I don't have the shared experience, I think I've also landed on economic thinking as part of my primary model of how I think an eng team should function and been trying to push the concept with limited successes (although I've termed it differently). And I've never observed a team that I think has executed on it really well.
Yeah. Creo was a great place to be and had some very good people. The focus on culture really permeated the company.
I would say though that it wasn't simply the dilution of the culture, there were also some real business challenges that led to external pressures and eventually the sale of the company. A takeaway from that is that business is an important part of, ya know, being a business. A financially successful business has more room to be a great place. Creo did well and grew well but operated in an area where making money isn't easy (the printing industry).
It's important to note is agreement being everyone can live with the decision. I.e. it's not really a bike shedding thing. Just that nobody has strong objections (the reason being is that if there are strong objections there are probably good reasons for that and ideally those are understood before taking the decision).
At the end of the day, if there was a time critical decision and there was no consensus someone like a project manager would make the call. I haven't actually seen that happen, a well functioning team/project generally is able to make decisions.
I am sorry you have never worked anywhere like it. For over 10 years, the company grew like gangbusters. It transformed a whole industry. It grew from under 50 people to 5000. It made hundreds of millionaires.
It is a massively self-evident fact that a lot more money was made during the “principles” period than after. And the company outperformed its rivals to the point that it was able to acquire competitors with twice as many employees. If “money to be made” is your core objective, you should strive to be more like Creo in the decade that ended a year before its demise.
Decades after the company disappeared, it has a very active LinkedIn Group and people are singing its praises online. If you are right that it was so “toxic”, this is a strange legacy. Interestingly, the core technologies developed by Creo are also still very much alive and, as a very rare example in technology, have not been surpassed technically nor replaced in the market. Technology moved very fast at Creo but their tech has stayed virtually unchanged under Kodak. Amazing.
Where I work now, we do at least three M&A deals a year ( as the acquirer ). I have many ideas about M&A. My ideas about leadership are still very firmly planted and aligned with what I experienced at Creo.
> It is a massively self-evident fact that a lot more money was made during the “principles” period than after.
Is it though? I've seen a lot of claims about this or that management style or company structure to be responsible for huge profits and growth. It's usually a more simple explanation, such as market timing + smart people. Ultimately it's survivor bias. It could be argued that if this were really the case, all top companies would be using it by now.
But that last sentence is the Creo supporters whole argument - self interested management get much more out of rent seeking in a nice profitable company while seeing their hierarchy position reinforcing their social
Status- in short as long as everyone keeps being “performative hierarchical management” and looking like they do a good job, there is no incentive to actually do a better job
If we are talking “founders”, the two Creo founders made hundreds of millions of dollars when the company sold. Long before that though, they gave away a third of the company to a guy they named as second CEO. Then the original CEO biked around China for a year while that guy made him that fortune. The two founders pooled their shares and then split it three ways with the new guy ( second CEO ). Many rank-and-file employees made millions as well. It was a different set of folks.
I am not sure how many hundreds of millions you have made but their way seemed to work.
I've run many teams like this and have generally received good feedback from my teams. It's pretty simple in my experience. Don't punish people for making generally sound decisions that end in failure. Reward them for decisions that end in success. The issue is that managers need to eat some of the risk of those failures although it is offset by the benefit of the team's successes. Few managers are actually able to do this versus trying to dump all the risk somewhere else (usually onto their teams). But if you can maintain a management layer like that then it's amazing.
No problem. I am not sure what we are disagreeing about.
The company was amazing. No offense but it is not important to convince you of that. And you are certainly not going to convince me that my experience or the stories told by hundreds of others are false.
You could be trying to convince me that management does not matter, as my core claim is that it does. If that is what we are disagreeing about, I would like to know more. Great ideas can come from anywhere but I want to stay evidence driven. It is the only way I know how after working at Creo.
You keep explaining imagined features of an alternate reality where the people who worked there actually hated it.
This isn't an abstract thought exercise. You're talking to someone who worked there, so it comes across as irrelevant to imagine people who hated the politics.
I agree with you generally that abstractly one should discount claims of lack of politics. founder soul, founder, sold it, Google for 7 years, then founding again.
This isn't reddit. People will downvote and flag far reaching toxically negative comments about a company the poster has no actual experience with. As they should.
Throttle because of your arrogance and poor assumptions based on thinking your reality is everyone else's? Boo fucking hoo. Stay in your sad world and we'll stay in ours.
I'm not sure what "empowered" means relative to the parent article.
Aligning each employee's incentives with ROI seems extremely difficult.
The former Yugoslavia had a system of self-managed enterprises where large firms shared profits equally among employees and employees ran the enterprise democratically (usually electing a manager). The problem is that profitable enterprises would democratically decide not to hire any more people since doing so would dilute their profits.
> The problem is that profitable enterprises would democratically decide not to hire any more people since doing so would dilute their profits.
which is fine, because if hiring someone would dilute their profits, it means the new hire is decreasing the efficiency of the organization. Normally, you would hire if you find that the existing employees cannot complete all of the work required, as there's too much - aka, leaving business/profits on the table due to lack of room.
No, it doesn't: if there's that much demand for the company's products, someone else can start a competing company making the same products, presumably for less because with a monopoly supplier, prices will usually be very high. Of course, this doesn't work for cases where supply is artificially constrained (like products with IP protection), or if the communist government doesn't allow competing companies to start.
I did not attempt to cover heterodox organizations such as what Creo appears to have been in the article. I think these are extremely interesting, though quite likely to fail, regress to the mean, and be hard or impossible to scale or replicate.
But, I'm sure the first multicellular organisms were no picnic, either. Eventually humans will learn to work together way better than today and anyone trying already now has my attention and sympathy, especially if there are positive testimonials by employees (there's no shortage of managers who claim to run a different kind of org / culture when it's either false or true in a bad sense, meaning, their org is much worse than the standard system; in fact I like managers who apply zero lipstick to the pig that is their org and do the standard ugly thing in an openly ugly way - teaches you how to operate in this environment more quickly vs trying to get people to remain naive to their detriment)
Your analysis could be applied to Creo without much change because you're looking at implicit incentives in any place managers (or resource-controllers) are expected to accomplish goals.
That said, I think your analysis is a bit weakened by a moral elements (good managers vs bad managers). And even more, it's not necessarily bad for an organization to have locally controlled resources. One can posit an omniscient central controller that would manage resources better than local managers but it's likely that there are limits to both central controllers and local managers. What's "best" depends on the task, the organization and who's judging the result.
I had a similar experience thought not quite as outstanding.
The best manager I had was:
- focused on organizational efficiency. She’d tell you “you three, make a decision” instead of groups of 10+, which seemed to be the default
- wouldn’t sugar coat things, but was never mean (many leaders miss this). If she disagreed you knew it but it was done in a respectful way.
- the best part of her management was goal setting. I was 3 goals, and even 10 years later I still rattle them off. Short, actionable. Her attitude was “if your work doesn’t move towards these goals, stop doing it”
Like you, it’s hard to work in jobs later when management isn’t up to snuff.
It is possible to have efficient, well managed large organizations.
It’s just that the vast majority of managers don’t have the skills and as you said, without someone at the top setting the rules it never works.
> This extended to decisions that involved spending money, not just should you pick language X or language Y for your next software project.
Just want to point out, the choice to pick language X or language Y is very much about money. Training, triage, getting into production, scaling, testing, etc. Most bigger picture technical decisions are in fact, economic ones.
For sure. I was just pointing this out because I've worked in organizations where there was no problem with delegating language choice to engineers but the minute any actual money had to be spent it was treated very differently.
You are absolutely right that technical decisions are economic ones.
> Another requirement for this to work well is that management (e.g. the CEO or other leaders) are able to lay down a broad strategy for the people of the company to execute on. If the leadership has no strategy then tactical decisions can not be made properly. They also need to make sure there's coordination and structure.
In my experience this isn't a core requirement so much as the ability to discern if a proposed strategy is good or not based on first hand experience. In other words, the ability to spot very well written bullshit yourself without relying on anyone else's perception of it being bullshit or not. If you need to rely on other people's opinions or the way a proposal is written or something else then it will not work. It's very rare for this to be the case at a larger or public company. Even if the CEO has this ability if the board doesn't then it will be the same.
1. Could you expand more on the practical details of how decision making by consensus worked? Eg if I want to buy a $1m machine, how many people and which people do I convince that this makes economic sense? Who can actually sign off on the purchase?
2. How did incentive based compensation work for employees whose work could not directly be tied to the bottom line? For example, legal and compliance staff sometimes prevent profit making activity that is deemed as risky - making these functions prime candidates for internal monopolies that drag down the performance of other groups. Or recruiters whose natural incentives revolve around number of people hired rather than efficiency.
3. How is economic decision making enforced? What stops a middle manager from acting out of self-interest and expanding his own headcount or project scope to increase his own importance at the expense of company economics? Even if there is some consensus required, it’s probably not hard to convince a few friends with fanciful projections. Is this mostly a camaraderie / honor system thing?
If you feel like these are the wrong questions and are somehow missing the point - let me know! Sounds like Creo pulled off something special and I’m sure there is lots to learn.
1. You were supposed to identify who the stakeholders were. Let's say you're a mechanical engineer and you need some $1m lathe. The economic decision making basically says you need to consider options like contracting the work to someone that has that machine and show some reasonable ROI period on actually buying that lathe vs. contracting it out or renting it or whatever other reasonable alternatives. People involved in this decision could be your team lead, if you're a mechanical engineer, and the project manager. If the amount is so large that it impacts the entire company maybe the CEO is also involved. You get everyone in the same room and talk about it. Anyone can sign off on the purchase. The project I was on built some large very expensive machines (multi-million dollar per machine) and we've done things like build clean rooms and buy pretty expensive equipment without a lot of friction. I worked on software so I wasn't personally in the loop for those purchases.
2. I guess it's no different than RSUs, stock options or a bonus plan or any other form of profit sharing. At the end of the day people are trusted to do their job. At least during my time with Creo it didn't seem to create a barrier to experimenting, if anything there were some projects that were maybe were too experimental.
3. Not enforced in any formal way. Peer review and just culture/peer pressure would do the job. I am aware of one case where someone abused the system and someone found out and they were fired. In terms of your example, the power was fundamentally less with the middle manager and more with the engineers in the first place, so there wasn't a lot to gain personally from empire building as in traditional companies. For the most part managers did not manage people, they managed projects. For example, as a software team lead I was under a project manager but I didn't report to him in the traditional sense of the word and they did not control my compensation (which would be mostly the outcome of the peer review process). The project manager's success was also measured by peer review.
I think the interesting thing about Creo, and what made it tick, was that the founders were in it not just to make money. They wanted to create a place they would want to work in, were passionate about this, and tried to hire people that fit into this vision. They have seen the things they didn't like and thought they can do better. It sort of happened in the right time and place where it managed to gain momentum.
These are great questions and match my conclusion after many years of big5 management consulting: (A) Organizations are not optimized towards efficiency or value, they are optimized to align power between the executives and their self interest. Any heterodox organization with virtuous claims of egalitarianism will quickly reset to the default (A).
The problem is finding people that have those traits, it is far from common even when you try to empower people to take decisions. It is also good to take into account the different cultures regarding this.
A way to not be extreme is to do that with x% of people (beyond their hierarchical level in the organization).
How did the consensus bit work? Like, how much consensus would you need from people before making a decision? How much time did everyone spend voting on consensus?
Presumably you'd have moments of contact and separation where strategy and policy are communicated. I don't see any impression this is a replacement for executives.
They made decisions relevant to their work. i.e. it's not that a random employee would go our and acquire a competitor or buy a business jet for themselves (though in theory they could but that theory never got put to a test). So the CEO still made CEO-level decisions and employees made decisions in their area while needing to ask themselves what's best for the company, economically, just like the president should. Trust people to do their job kind of thing really, trust teams to work together etc.
Is ‘responsibility’ really something that deserves outsized compensation compared with, say, actually creating the product being sold? In this context, it doesn’t seem to come worth outsized personal consequences for failure, indeed quite the opposite. I agree there should be some consideration for the additional stress, but not multiple extra figures on the salary.
> for the new insurmountable levels of responsibilities right?
As it turns out, companies that don't concentrate all of the power and decision-making in one person tend to avoid ending up with "insurmountable levels of responsibilities".
Trouble is most middle/upper-management is so technically incompetent, that not only can they not do technical work that their moronic decisions entail, but they can't even distinguish b/w arguments for how something is to be accomplished.
As a metaphor: for problems in NP, verifiability is in P. So you have some super-duper savant alg. that outputs some answer, but even a dumb-system should be able to quickly verify. Our managers are typically dumber than this.
I'd argue the opposite: it's the technically-savvy people who got promoted to management/leadership positions, while lacking the people skills, the strategy chops, and the basic understanding of business realities - these are the truly incompetent managers.
But you need to be able to promote good technical people or they will finally leave and you ll be left with gif managers an no one good technically to manage.
A boss doesn’t need to be technically competent to a degree that they make important technical decisions. They just need to understand things to a degree so they can trust and delegate.
And tust is a two way street. It requires effort from both sides.
Many times, they were technical people/engineers who weren't that great at the technical stuff, but were really good at networking and self-promotion, and talked their way up in the organization.
Things work the same with any resource, not just actual money - it could be team size, or processor cycles and memory bytes. If you free up 200 ms of CPU cycles and 500 megs of RAM, someone else can deploy their functionality using these newly available resources, and then you won’t be able to. In fact, a mature, well-run CI system will measure everyone’s resource footprint after each commit, and will not let you exceed your budget, which was frozen at some point based on how much you were using at the time (hope it was a lot! - always spend like crazy before the baseline is established!) Is it any wonder that people learn to never optimize their code - unless they want to deploy something new themselves, and only after asking for more resources to deploy it and not getting any?
This explains so much. It also sounds like broken water laws in the western US that incentivise farmers to intentionally waste water to preserve their future ability to waste water.
No reason to go that deep. Spending in large bureaucracies (public or private) often works similarly: there’s a lot of hair on fire and throwing money frantically around at the end of a reporting period, because you know if you don’t spend it you’ll get a smaller budget next time. (It was that way at all levels in late Soviet Union, it’s still that way in at least one F100 IT giant.)
Why does it happen? My CI launches separate kubernetes pod for every incoming task. If there are not enough servers, autoscaler will spin up new and remove it after some time. So there are not much expenses, we're paying for what we using. I feel that fast CI iterations are very important for developers.
This has been the bane of my entire career. I have never, not once, encountered a CI system that could run the test suite faster then my development machine.
Which is absurd. Cloud resources on servers with more CPUs then I have, and yet the difference in timing is measured in tens of minutes.
From what I've seen this is usually caused by CI pipelines being overly cautious, starting from a pristine state and trying to cover every possible edge case.
Spin up new node, congrats, now you have to clone the whole repo from scratch. Oh, "npm install" was not cached, let's download half of the internet again, pull ":latest" of some multi-GB docker images while you are at it. For good measure, "make clean", because...you never know and nobody bothered to architect our build system to use a more sane tool than Makefiles, btw remember to only use make -j3, because rumor has it that the build fails intermittently when using more cores! Finally, run every test that couldn't possibly be affected by the commit in question, preferably the big e2e-suite. As someone else mentioned above, all of this running on some dog-slow cloud network attached disk.
There you have it. Spinning up more compute is not the solution, it's the cause. Keep one machine on and improve on your build tooling. I guarantee it will be magnitudes faster than any ephemeral pipeline.
This is usually a result of following 'cloud best practices' instead of being pragmatic.
For example, using Kubernetes with Docker where each pod is some virtual ECS 4-core whatever instance, assuming scaling the workload over a 1000 instances will be fast. Which in your case will lead to said pods individually spending 90% of their time running 'npm install' , and each pod is way slower than your desktop PC.
Different tests have different needs and bottlenecks. For example, if you're running unit tests, with no external dependencies, you'd want to separate out the build process, and distribute the artifacts to multiple test machines that run tests in parallel.
The choice of using a few big machines or more small ones is usually up to you, and is a wash cost-wise. However I do need to point out AWS sells you 192 core machines, which I'd wager are WAY faster than what you're sitting in front of.
And Amdahl's law is a thing as well. If there's a 10 minute non-parallizable build process, doing a 10 minute test run on a 192 core machine vs doing an ~0 minute test run on infinite machines ends up being only a 2x speedup.
And there's such a thing as setup costs. Spinning up an AWS machine from an image, setting up network interfaces, configuring the instance from scratch has a cost associated with it as well. And managing a cluster of 1000 computers also has its set of unique challenges. And assuming that if you ask Amazon for a 1000 machines at a drop of the hat, and plan to use each for a minute or two, AWS will throttle the fuck out of you.
All I'm trying to say with this incoherent rambling is KISS - know your requirements, and build the simplest infra that can satisfy it. Having a few large stopped instances in a pool might be all the complexity you need, and while it flies in the face of cloud best practices, it's probably going to be the fastest to start up.
I would argue the problem is for all the CI systems out there at the moment, there all "stupid": i.e. none of them try to predictively spin up an environment for a dev who is committing on a branch, none of them have any notion of promoting different "grades" of CI worker (i.e. an incremental versus a pristine) and none have any support for doing something nice like "lightweight test on push".
All of this should be possible, but the innovation is just not there.
> notion of promoting different "grades" of CI worker (i.e. an incremental versus a pristine) and none have any support for doing something nice like "lightweight test on push".
Which ones don't support that? Anything with a Docker cache, for example, can build layers efficiently, reusing previously built layers. Build triggers let you choose when to start a job, so GH actions, for example, can trigger a full test on any PR and a light test on any commit to a branch.
> And assuming that if you ask Amazon for a 1000 machines at a drop of the hat, and plan to use each for a minute or two, AWS will throttle the fuck out of you.
I haven’t scaled that far out, but I thought that was the whole point of cloud platforms like AWS, GCP and Azure
But then what will you contribute to the hour long weekly meeting where 13 managers present the status of their poorly defined CI metrics and other "action items"?
Not my experience. The CI systems I dealt with most recently, for example, were very much compute-bound. Way too weak VMs for the runnres, and way too few of them. With CI builds taking each of upward 2h, and capacity to run maybe 10 of them in parallel, this pretty much destroys any ability to work with small commits - you have to squash them before submitting for review anyway, otherwise you destroy anyone else's ability to get their changeset through CI for the rest of the day.
This is entirely solvable by getting more resources. At least 2x more runners, each at least 2x as beefy. You could go 10x and I doubt it would be anywhere as expensive as the amount of money company wastes on devs waiting for CI. Alas, good luck convincing people managing the infra of this.
There was caching of third-party dependencies, but at least 2/3 of the time was spent on various tests, which for domain-specific reasons weren't trivial. Sure, everything there could be optimized further, but this is a textbook case of a problem which can be solved by adding more compute for a fraction of the cost of dev-hours spent trying to optimize compilation and test times.
> I guarantee it will be magnitudes faster than any ephemeral pipeline.
Only if you have a single job to run. In practice it doesn’t really matter to the dev whether CI takes one or 5 minutes. When it all goes over 5 minutes is when it gets annoying.
It depends on how much policy your organization has. If your organization requires AV scanners and other security crapware to be on all instances, how long does that take to install? And perhaps they require that you only use their AMIs, which then must be loaded up with extra software to be usable. And perhaps they require all auth use their special auth service, which takes 5m to sync with. And on and on. I've worked in places where it takes 20m to spin up a usable instance, so we ended up just keeping them on most of the time.
In an ideal world, the policy groups imposing these inefficiencies would get billed for the cost of their policies, but I've only ever seen situations where everyone else pays the cost of their incompetence.
Ah, yeah, we kinda have to work around the fact that the antivirus and scanner stuff keeps installing far after the time the instance is done processing the job.
On the plus side, I actually get actionable notifications from the stuff, so it’s not purely bad for me.
And yet the data is ephemeral, so in many cases could be done faster without the "real" block storage guarantees.
Github Actions is pay-per-minute-used, unfortunately, so they may have a negative incentive to speed things up. Unless people become frustrated enough to switch to a non-bundled provider.
Not all the data is ephemeral - stuff like node_modules needs to he cached and if you’re suggesting to use tmpfs then it’s like 50x more cost than fully tricked out cloud ssd which is already ridiculously expensive
You should just add an admission webhook that kubeadm’s your dev box onto the CI cluster with a taint that only your pods tolerate whenever you need to run the test suite. Call it edge cloud and people will love it.
There was a tech talk many years ago when one of the github engineers talked about how fast the github CI tests were (this was way before the microsoft aquisition). Literally seconds. I wonder if they're still that fast.
They absolutely are not. Hundreds of thousands of Ruby tests end up being pretty slow to execute. The CI machines have all dependencies pre-cached and different tests are parallelized across multiple VMs, yet still a full suite takes ~12 minutes.
> Cloud resources on servers with more CPUs then I have
The server the resources run on may have more CPUs than your dev machine, but are you allocating more CPU to the specific test job than you desktop has?
My experience is the opposite due to the insane overhead of the spyware^H^H^H antivirus and security services running on our personal devices. Some colleagues successfully applied for MacBooks since these are reasonably fast even while running the security services.
> We should forget about our coworkers, say about 97% of the time: premature coordination is the root of all evil. Yet we should not pass up our opportunities in that critical 3% to take credit for their work.
I work for a government research lab. Our business model is that we don't receive government-budgeted money, we actually sell our services to other parts of the government. So in effect we have a bunch of researchers and engineers who theoretically report to "management", but actually report to a local program manager, who reports to an external sponsor.
We had an official manager, a "branch head", who was worse than incompetent. He couldn't find his butt with both hands, but he also thought he was God's gift to management, and would forcefully and emphatically make bad decision after bad decision.
Eventually, he had screwed up the group's major program so thoroughly it looked like a sure fire failure, and he found another job, and didn't bother to tell anyone; he just stopped showing up for work one day.
The level of management above him had bigger problems to deal with than replacing him, so they made sure we had a competent secretary and left us alone for two and a half years. It turned out to be arguably the most productive period of my professional life. My buddy and I took over business development. THe team turned the big project around and made it a rousing success, and grew the funding from it by two orders of magnitude.
The point being: there's bad management, that acts randomly or not at all; and then there's really bad management, which takes up your days with constantly changing orders, fixing relationships with customers or sponsors that they've screwed up, and levying time-taxes in the form of training, reports, and morale boosting exercises.
If given a choice between bad management and really bad management, pick bad management.
Fifteen years later, my buddy runs the place and I'm the senior scientist.
I've really found this to be true; I've had just a small handful of great managers and most of them worked more like "team secretaries", with just two who would also deploy political capital in an expert way to keep things running well. Those were the best. Most of the good managers topped out at acting as personal assistants to the team, helping (not dictating) to keep things organized.
The just ok managers barely did anything, which was still not bad. They didn't make things worse but didn't make them much better either.
The worst would dictate, randomize and foist cleanup of their own making onto you. Those folks I often wished would just stop showing up to work even if they kept collecting a paycheck; things would have run smoother if they did.
> I've had just a small handful of great managers and most of them worked more like "team secretaries"
What if you have really passive team members? So somebody need to digest the big project into smaller pieces and assign people to them, you can't just throw the project and expect it to go smoothly. If you don't it will take extra 2-3 months to start things up. I suspect in your case the team is motivated and experienced and/or there are some engineering leads in the team who can do that instead.
If you look at the world's must successful military initiatives, many of them practiced "maneuver warfare" in which individuals were given broad goals and the ability to use their initiative to attain them.
Decisions should be made by those with the best information. Sometimes that person is not the leader, and opportunities should just be seized by someone at a low-level. When this happens, leadership should back these initiatives. My favourite example of this is the Battle of Arsuf during the Third Crusade.
Richard the Lionheart of England spent weeks slowly marching down the coast with his heavily armoured cavalry/infantry, getting harassed by Saladin's faster cavalry. His army of 10k men could not attack and break formation, since Saladin's cavalry was faster and could pick off Richard's men.
Richard's goal was to wait until Saladin's horses got tired, and when that opportunity arose, charge and catch him while he was slow.
After a few weeks of nothing, suddenly one of his units (the Knights Hospitallier), led by Baldwin le Carron, started to charge on Saladin. Richard the Lionheart had no way of knowing if this was a good idea (to this day we don't know why Baldwin made his decision) but he immediately backed the charge with all 10k of his men and defeated Saladin's army.
This wasn't being lazy or poor management. Richard understood that he did not have the information needed to execute his strategy, so he delegated it to his officers.
When Baldwin le Carron made a decision Richard didn't understand, Richard didn't ask for a Jira ticket or a design doc, so he would have more information to make his decision (as a "good" manager should). He trusted his subordinates and won.
I believe software engineers are getting the wrong idea about a company being in "war mode".
If one reads military strategy, a proven tactic for leaders is outlining strategy and trusting your subordinates to execute it, because feeding information upwards takes too long.
> When this happens, leadership should back these initiatives.
this requires trust in the "low-level" people. It also implies that the "low-level" people have sufficient power at their call to execute those initiatives, without directly involving the leadership.
This works if leadership is not under threat from the low-level people (e.g., in a democracy, where your survival is not contingent on kinship with the military's cooperation).
> He trusted his subordinates and won.
So this is why i suspect this does not work in most modern organizations : the leadership does not (or cannot) trust their subordinates to make the correct choices. In other words, the leadership doesn't want to cop the cost of a wrong decision in the hands of the "low-level" people, and insist on seeing evidence/plan/etc (which basically means they're not really deligating the decision down, but pushing the decisions up!).
The Roman Army also left a lot of decision power to their lower levels, at least on a tactical level. They most definitely were not a democracy in the modern sense.
"Leaders" who are overwhelmed, completely out of their depth, or both, become control freaks. They can't trust themselves, so how could they possibly trust others?
This might be a bit of an isolated example. Numerous battles against the Mongols and Turks in the middle ages start with a haphazard charge, followed by the cavalry getting out of position and completely crushed.
Seems like the success in this specific instance relied on trust between Richard and Baldwin that the latter wasn't leading the Crusaders into a classic Turkic trap.
A few kilometers into the charge, Richard called for a halt to the advance, relying on his own knowledge of Turkic/Mongol strategy. Some of his leaders kept going anyways and died, though most of his army quickly obeyed. That maintained the victory since he didn't get pulled into the trap you mentioned.
Trust cuts both ways. Senior leadership needs to be trustworthy, so when the time comes to make a decision based on information you can't easily reveal to employees, they follow you.
You need to tell people to take the initiative to charge, and cut them off before they die.
That's a difficult balance to strike. But it's how you pull off a victory against an army twice your size thousands of miles from home.
The US Marines tend to greatly value delegation. In one instance, a Colonel boarded a helicopter to direct a rescue operation. It ended his career as it was clear he either did not have the ability to delegate or had not sufficiently prepared his troops.
However, the US military actually trains leaders - extensively. And it also trains people to work in coordinated units. The more well trained a unit is, the more broadly a commander can give orders and expected a good result. There's a lot of context embedded in the training and the doctrine that informs the training.
To the author - this is the best/most realistic (and funny) depiction of how things actually work in medium to large organizations. Also nailed the part around how small companies with “lax” technical hiring standards but strong cultural components triumph over their bigger counterparts. Because people working there actually care and they can see the impact of their work. It is real to them vs. some made up KPI metric to sound progressive.
As an aside, I’ve been lucky enough to have sufficient influence at all my employers for removing unnecessary layers of interviews. Anecdotally, most average engineers would be at least half as productive at most jobs out there. A couple of interviews at most would increase the odds significantly in the employers favour. Google scale could be an exception, but for 90% startup and mid sized corporations, the 3-4 technical interview rounds and algorithm problems are an absolute overkill.
From my experience, those extra rounds and puzzles don't improve the quality of hires. But they:
1. Create objective consensus. In a small company, you can hire based on intuition - talking to a candidate once (and ideally seeing any kind of work sample) are mostly sufficient for me right now. In larger organisations, I had to _defend_ hiring decisions, so you need to create evidence of objectivity.
2. Filter a prohibitively long list of candidates. Gotta have _some_ way to do it, even though I'd argue selecting for those most likely to hang in there for a long hiring process isn't the best way.
3. Create a (usually faux) narrative that you rigourously assess candidates and therefore only the best work with you.
Now, I won't say there aren't some benefits to more exposure to a candidate and their work, whatever it is. But in my experience, extensive hiring processes are really more about the stuff I listed above, even if nobody involved would think of it that way.
Thanks for your kind words! In terms of what I think & meant to say - I'm pretty sure that incompetent management (or incompetent at least in some key areas) can scale to fairly large org sizes, and that eventually competent management will evolve for the org to survive. I used "competent" and "incompetent" unironically, not as proxies for big & kafkaesque vs small & nicer places, but as in can/can't set and achieve goals, which is something you will need to be able to do to survive eventually.
Sadly a lot of the KPI driven madness has trickled down into smaller and smaller orgs. I've seen it even in ~200 person engineering orgs these days.
Always because management is some guy that's never been in the hot seat himself and doesn't know how to talk to users/customers. So he needs a pretty dashboard with green lights and KPIs he can shout about.
I worked at a 6 person startup with 2 hour weekly all hands where we went over dashboards, metrics, and KPIs. We had over a million in funding and less than 30 users. It was hilarious. I put in my 2 week notice 6 weeks in.
> Whatever objective you are expected to achieve, a bigger budget makes it easier.
This should be true but I've seen bloated projects that would have had better outcomes had the team been more constrained.
And to quote Fred Brooks:
> There are many examples from other arts and crafts that lead
one to believe that discipline is good for art. Indeed, an artist's
aphorism asserts,'Form is liberating' The worst buildings are those whose budget was too great for the purposes to be served.
Bach's creative output hardly seems to have been squelched by the
necessity of producing a limited-form cantata each week. I am sure
that the Stretch computer would have had a better architecture
had it been more tightly constrained; the constraints imposed by
the System/360 Model 30's budget were in my opinion entirely
beneficial for the Model 75's architecture.
Had Mr Brooks proven this belief in a revealed preference sense, by voluntarily shrinking his budget, it would add a lot of weight to what is otherwise a nicely sounding but not very believable passage. But I doubt that anyone could be so insane as to not trust themselves with making a good use of a larger budget. One's underlings - quite possibly!
Also it's just so much easier to get the benefits of constraint without the downsides if one is in charge of the decisions. Like if you can do something with $10k and have $30k to work with, many adults manage to constrain themselves by merely moving $20k to a savings account, and those who lack the requisite executive function for this simple "out of sight out of mind" organizational schema can often function with slightly stronger artificial constraints, sometimes aided by a collaborator. People do NaNoWriMo or whatever, and while many fail, the benefit of the artificial time constraint motivates a lot of people, despite being self-imposed. In either case, this informal method of constraint is more responsive to situational changes or emergencies than an optimization valve that always wants to bring costs down and doesn't understand any of the functional constraints, as is the case for most finance and management arms of "efficiency"-driven orgs. The game theory of budget inflation is really obvious when you consider it in terms of locus of agency
> But I doubt that anyone could be so insane as to not trust themselves with making a good use of a larger budget. One's underlings - quite possibly!
If I need 4 people for a project, what am I going to do with $20M/year in budget? I can’t pay them 5M a year each. I would not trust myself to do something sensible with that money, no. Nor do I suddenly want to hire 100 instead of 4 people.
California high speed train to nowhere comes to mind. Billions and billions and billions of dollars lost to graft and waste. Indeed, as government budgets increase they become less effective.
The comments on resource utilization are one of those things that is only true now that there is a 1:1 correlation between $ and available CPU/Memory, and the pool of available CPU/Memory resources is effectively unlimited.
Embedded is maybe the last refuge of where there are hard resource constraints that cannot be violated. If you only have 64K of RAM, you had better make it work!
IMHO this is why you can end up with embedded code that is easily 10x to 100x more optimized than code running in other environments. It is also, why IMHO, the user experience on Smart Phones doesn't improve linearly with the hardware improvements - see https://en.wikipedia.org/wiki/Wirth%27s_law
This train of thought, along with basically flatlined GPU performance / $, explains also why we are seeing real algorithmic improvements in the ML/AI space. If we were still operating under Moore's law and GPUs and VRAM were 2xing every couple of years, I doubt we'd see all the research efforts going into optimizing training, fine tuning, context windows, inference, and so forth.
TFA author - I have worked on low-cost embedded systems. (Tens/hundreds of megs of RAM, not KBs.) The only case where you don't hoard resources is if the system is so small that it's programmed by a small team since the functionality is severely limited by resource scarcity, so there's no reason to grow the team to deploy more functionality. Above a certain not-so-large size, people will hoard resources in an embedded system - they can't ask for more capacity like they would if it was server-side software, but they sure can avoid giving up whatever resources their code is already using.
Researchers actually have a limited and smallish hardware budget, so academia is likely to come up with cost-saving ideas even when hardware performance grows very quickly. In the industry you can throw more hardware at the problem even if it's not improving (outside embedded/client devices)
> TFA author - I have worked on low-cost embedded systems. (Tens/hundreds of megs of RAM, not KBs.)
I worked with kilobytes, small team, everyone sat together. No one was hoarding resources, because just to ship out the door we had to profile every single function for memory and power usage!
IMHO Hundreds of MB of RAM isn't embedded, it is just "somewhat constrained". :-D Regular tools run just fine, you can use higher level languages and garbage collectors and such, you just have to be a little bit careful about things.
> Researchers actually have a limited and smallish hardware budget, so academia is likely to come up with cost-saving ideas even when hardware performance grows very quickly.
Agreed, but I also think it is difficult to determine when forward progress is stymied by too many resources VS too few!
As someone who works in embedded systems- I agree. Teams will deploy whole desktops to an embedded system if permitted to expand enough. Architectures will evolve to require hardware that can support the bloated stack.
Even on power constrained robots operating in the deep sea it's a pretty safe bet that some of them are running whole Windows VMs.
> the pool of available CPU/Memory resources is effectively unlimited.
This isn't true. Every CPU you use is sitting in a physical server somewhere. Every server is sitting in a rack in a building that's sitting on land that used to be nature. It's using electricity that could be used instead for cooling or heating a person or just not generated at all. It's cooled using fresh water that is needed by every living thing and is in short supply in many places.
The idea that compute is unlimited is a horrible myth that cloud companies have sold which has effects and externalities that will be seen as on par with the myth of unlimited fossil fuels.
> The idea that compute is unlimited is a horrible myth that cloud companies have sold which has effects and externalities that will be seen as on par with the myth of unlimited fossil fuels.
Granted, all true, but my point was that for any one given software team at a mid-sized company with a team of a 50 engineers or so, CPU resources are effectively unlimited. Unless they write some N^3 algorithm, the fact is that it is almost always possible to write AWS a bigger check for more CPU to a pretty much inexhaustible extent from any one given customer's persepctive.
> If you only have 64K of RAM, you had better make it work!
Hah, this brings back fun memories. I worked for a spell at a large company producing smartphone components. One of their primary products was a do-it-all embedded chip that handled many disparate functions, which eventually grew to rather massive proportions; rather than optimizing the resulting bloated software, the company just petitioned their chip supplier for a special part with way more flash memory than a standard part would have. IIRC it had something like 16x the Flash memory compared to the largest public part.
The code quality was awful. People were churning out C++ monstrosities with zero concern for code size. The final product was the result of linking together modules from dozens of teams, none of whom really cared at all about code size; I’m frankly surprised the CPU didn’t fall over from all the excess code it had to run (the RTOS was, at the very least, very good at prioritizing…). But: I’m sure it saved a lot of engineering effort that went into shipping flashy features instead.
I like this, and it seems a lot of people are thinking this way lately
A common theme is that optimization usually does mostly bad things, while maybe arguably from someone's perspective doing one thing really well. I particularly like the example of the threshold cost to get something done versus the optimizer trying to lower costs. Both stakeholders in that example in actuality care about not going below the threshold, but the one drunk on optimization is incentivized to be at odds with keeping that cost center above it, let alone providing any cushion
The model many people here likely have for optimization is compile-time optimizations, but that's actually a weird class of special cases where you actually can prove you're not breaking anything by doing so. Most optimization looks more like strip-mining. It leaves the structures it touches barren and brittle
Most extant institutions have a desperate need to build better resistance to optimization-like objectives
Yes, and solving actual problems is a great way to frame a goal. People often equivocate between "optimization" and "improving things", so I don't blame you for it in particular. Optimization, by definition, requires that something be "overdone", that an objective function is maximized or minimized. Organizations with human decisionmakers aren't incapable of assessing other factors, what priorities not captured by the optimization objective might be relevant, but by framing decisions as optimization, especially throughout an organization or an economy, the pressure against that kind of sanity check is implied and grows the more "competent" an organization is
The problem stated by the article is that consistent incentives inevitably results in employees “gaming” of the system.
I suspect but cannot prove that inconsistent metrics and incentives do actually work.
In other words, measure and reward different things each month! Latency this month, resource efficiency the next month, customer satisfaction the next, and so on…
Anyone attempting to cynically game the incentives will lose out on average, but people that naively do a good job overall will tend to be rewarded. Maybe not every month, but most months.
I've actually worked at a place that did something like this, although I suspect it wasn't for positive reasons. It was a call center, and they had status tiers for wages - e.g. if you were an "A player" you would get $.50/hr more, or if you were a "star player" you would get $1/hr more (or whatever the amounts were). These statuses were tied to various metrics that the business cared about - average call time, quality scores from customer surveys, repeat calls, and so on. So ostensibly, if you got the current metric down to whatever the goal was, you would get a raise for a month or whatever.
The problem was that the business changed these metrics on the phone reps all the time, so it was common for some poor bastard to really strive to get his call times down, only to be told "oh sorry we're focusing on quality scores this month" when reviewing his work with the boss. My impression was that the company was doing this to avoid having to actually pay people more, not to keep people from gaming metrics - but I think the results would be pretty similar regardless of motive.
What happened wasn't that people focused on trying to be great generalists. Instead they got demoralized by the games the company was playing, and they would either quit trying to improve at all or they would exclusively focus on one metric (so that they could get the big raise when that metric was being rewarded next).
Based on that experience, I'm somewhat doubtful that what you propose would work. But it might - unfortunately the workers at that call center were very often real fuckups all around, so the idea might work at a business which has better quality employees.
If you have management failing to communicate business needs they'll make everyone beneath them look just as bad if not worse.
At the end of the day there's nothing naive about doing a good job. If you have any employees focusing too much on metrics, but especially your management, they're incompetent dead weight.
Doing a good job is simply less work for everyone, but it starts from the top down.
To clarify: each month measure a real business need. The trick here is that there are many needs, and the selection must be random each month.
Take a service like Google search as an example: if the number of searches per month is the consistent KPI for the tech team, then they’re incentivised to sacrifice customer satisfaction by returning junk results so that users are forced to do multiple searches to find what they want.
But if they’re only rewarded for the increased search volume the one time, the next month the metric might be customer satisfaction instead and they’ll be penalised for cheating in this manner.
The key aspect is not repeating metrics — but the metrics should be valid.
The problem with this is that it incentives short-term thinking, allowing you to juice the numbers temporarily, and at cost to the metrics that are not currently prioritized. But there's something to the idea.
You’re not told ahead of time what the metrics will be, so you can’t cheat and/or plan ahead other than by being generally good at your job across all likely metrics.
You mean to say that the metrics are applied post-hoc. I.e. you reward last month's efficiency this month.
Otherwise, you incentivize poor performance all the time, so that I can show massive improvement on this month's "efficiency push", next months "productivity (lines of code, completed stories) push", the "bug bash", etc.
My team once walked ourselves right into this trap with management.
We had hit the refresh cycle with our existing servers. We also noted that some new vastly improved CPU generation was going to come out mid-cycle and that we could survive til them with just a partial refresh.
So we "negotiated" with management - let's do a half order now, and another half order in 6 months. The head of the department gleefully agreed and said that makes sense. This was spelled out in a deck that we went thru together, we had his buy in.
6 months later .. same guy - "why do you need servers again / you guys need to be more cost conscious / why can't you write more efficient code / you must be using the wrong architecture / etc".
And so we limped along on a partially refreshed estate.
They should have put the second order on the books immediately. All companies have a ledger. Failure to understand the use and management of this ledger causes these types of problems.
In the articled example, you could put something like a "buy back" on the books, so while your budget is temporarily reduced to benefit the company, the end of year shows the same amount you _would have_ spent, so next year you don't have to fight uphill for "more money."
I have worked for a company where the ethos among the workers was very high, but management was exceptionally incompetent at managing. Zero people skills, and deeply skeptical that people skills were even necessary. (Many companies founded by hackers are like this.) So the CEO and CTO asked people to do things, but did no process management at all and just waited for them to be done.
This worked a lot better than you would think. We had hired people who believed in our mission, and mostly people do want to do what's asked of them. Our codebase was pretty lean, even boring, because there was no incentive to do any promo packet type spectacular development.
Projects sometimes took a long time, but they did get done.
But this stops working when you have projects that require coordination between people with differing incentives. Teams grow tired of waiting for the other team to do their half, so mistrust begins to become endemic. In the vacuum, all your hacker ICs grow fatigued and start doing more "interesting" things just to amuse themselves, because, who's really watching them?
TLDR benign neglect, even under ideal conditions, will eventually be a problem. The company's progress becomes slower and slower, and may even start sliding backwards according to some metrics.
This is a great comment! I only hope to add color, not to dispute any of it.
In my experience software shops that find product/market fit exceptionally early can not only tolerate benign neglect, but are best served by it while that market is still contended.
Capitalism works amazingly well when the referees just Godzilla stomp on cronyism and nepotism and monopoly and other market failures: the sharpest people want to work places that are in the fight and give everyone a stake in the outcome: such people in such settings are largely self-organizing on the basis of mutual respect and esprit des corps. A clear and serious competitor or competitors rapidly weed out ceremony and careerism while strengthening ruthless meritocracy (which is incidentally good for inclusiveness: no shortage of ethnic minorities in pro ball).
The tricky thing is keeping the incentives aligned as markets mature and the rot of capture sets in. All of today’s high-profile behemoths lobby extensively and employ former McKinsey people in policy level roles (Google is run by one).
Unfortunately it’s not as simple as throwing out the careerist middle managers or domain-soft executives when the scale exceeds “hacker culture”, you need executives that use their organization-fu to hold the rent seekers in check, and that is a rare breed.
Sure. Thanks for the color, but let me add some shape.
Perhaps you saw some confirmation of "ruthless meritocracy" in my post. You could not be more wrong. Most people in the organization I am describing would have been rejected as undistinguished weirdo losers from flyover towns. Our strength was that we supported each other.
I personally do not think that market discipline or financial incentives have much to do with producing excellence. At best, those systems can notice, after the fact, that something wonderful has occurred, and that it might be worth bringing to a larger group of people.
I still think we might be in violent agreement. The current sad state of our industry is IMHO mostly a result of the rejection of undistinguished weirdo losers from nowhere. I’m an undistinguished weirdo loser from nowhere, and I vividly remember the days when no one ran a background check on your politics or friends or habits at the weekend or aspy ticks or anything but “can this person code”.
I used the terms mutual respect and esprit des corps and you use the term supported one another: I think we’re talking about the same thing.
The prevailing wisdom, then as (ostensibly) now, was to judge people by their demonstrated achievements, your assessment of their ability, and what other hackers think in that order.
Today that’s a weird loyalty test (Carmack and Luckey will tell you all about it).
I agree with what I think you’re saying: good hackers are generally highly motivated by money until the point they get to work on cool things they believe in, and pretty bored/unmoved by it beyond that point: excellence is about pride in the craftsmanship and quality of our work.
The author is describing two cases of poor management. The first are self-interested managers. The second are laissez-faire (in a bad way) leaders who have all but abdicated their position. Both are bad.
There is no "healthy laziness." This is a cynical term that does a disservice to leaders who skillfully practice with a light touch.
I always thought laziness was a good thing, at least in our field. I'm pretty sure I'm lazy. When push comes to shove, I will do boring work very diligently, but if there's a better way, I'll look for it.
Contrast this with someone just happily diligently working away no matter what, for the sake of keeping on moving. It's very easy to get stuck on a local maximum if you don't stop and think (i.e. do nothing) on a regular basis.
I think the author's point is that the latter can be quite harmful, which is evidently unintuitive for many people.
Granted, calling an effective servant leader "lazy" is possibly not great, and the word does have a negative connotation whether I like it or not.
Somebody has to do the boring work for the whole to be functional, and pointless work may be only pointless to some (ok, not always).
Let's just say you're not interested in the thing you work on overall. Because you aren't or because you're disincentivized to think of the whole picture by the system you're in.
There's a lot of fantastic Russianisms in this post, for example
> “What is to be done?” is a pamphlet by Lenin, who proposed some things to be done, and went on to do them and then some, with results most charitably described as mixed.
The author seems confused about what constitutes competence vs incompetence. In my opinion the budgeting problem can most easily be understood through the lens of bounded rationality: senior management often has no idea what kind of budget is required to meet certain objectives, so can’t really separate the case where a frugal manager truly needs more budget and the one where a self-interested manager just wants a bigger budget to show off. In my book this is due to senior management’s incompetence, not competence.
An ability to operate reasonably effectively under the constraints of bounded rationality is an ingredient of competence.
You seem to define managerial competence as making the right decisions, and thus file bad decisions under incompetence. I believe this definition to be intuitive but useless for practical purposes. Instead, I define managerial competence as an ability to set and achieve goals, and claim that a few undesirable conditions commonly observed in the workplace are a natural consequence of managerial competence according to this fairly uncontroversial definition.
While I sort of agree with the premise, isn’t that a fairly low bar?
From the organisations perspective it doesn’t matter if the goal is achieved with 5 or 300 people, as long as it is achieved, but it feels wasteful to me.
I’d also kind of require the goals that are achieved to be the ‘right’ goals to qualify it as competence.
It’s hard to call a general that leads his army orderly into a trap competent.
> An ability to operate reasonably effectively under the constraints of bounded rationality is an ingredient of competence.
Sure, and that’s a reasonable argument for the claim that the less senior manager with the budget is competent if she refuses to reduce it, because she understands that it will not be increased again after the crisis is over.
But it’s not a good reason to call the senior manager who doesn’t understand the situation “competent”. The root cause of problem is that her rationality is too bounded, which I would prefer to call “incompetence”.
> But it’s not a good reason to call the senior manager who doesn’t understand the situation “competent”.
I'm not so sure that this is actually true... It certainly seems like a certain flavor/kind of "competency", for a manager to wander into unfamiliar territory, or come across an unfamiliar topic, and to have the wherewithal to then say, "well I don't understand why we have a $10M budget, and I don't understand what will happen if we reduce it to a $6M, gee my rationality is extremely bounded... but I know enough about management in general, to know that making big changes is risky, and all my underlings keep telling me that cutting budget to $6M is a terrible idea --> therefore, despite my too-bounded rationality, I will be competent enough to realize that not changing anything is the safer path (even if I don't fully understand why, and even if I have no crystal ball to predict next year's budget process)"
Maybe you would just call that having "self-awareness of their own bounds"? And if you have self-awareness than maybe that automatically disqualifies you from being "too bounded"? But absent arbitrary semantic rules like that, the ability to make wise choices with imperfect understanding or large gaps in knowledge, is itself a demonstrable and valuable skill that some people have, others can learn, and also some that will utterly fail to ever grok.
> The problem is that it’s hard for a well-run place to get people to fix non-showstopper bugs.
This is why mature products work so much better as open source- at some point you just want someone to care about the sharp edges instead of upselling new SKUs.
This article shows, in many examples, Goodheart’s law and Pournelle’s iron law of bureaucracy. You can’t make a hierarchical organization achieve outcomes people outside of the organization see as valuable through incentives because they cause side effects that will be counter-productive as soon as they move your organization out of its current state.
The article says it’s impossible to solve this problem. But I think the real problem is hierarchical organizations. They act as extractive institutions, removing value from people until those people realize what has become of them and then become parasites on the organizations.
The way out was provided by the framers of agile. I’m not talking about modern, consultant driven agile that uses the language of scrum to give organizations renewed extractive power. I’m talking the agile that provides value through close customer collaboration. That provides people inside the organization and outside the ability to influence each other. Closing the feedback loop eliminates disconnects and maps problems to solutions.
Most (all?) organizations are hierarchical above a certain size. I don't think agile ever addressed the resulting problems; some of it was designed for smaller teams. For an "agile" take on organizations of a medium double digit size and up, see SAFe - there's nothing quite like it
I'm on the first part and I struggle to not scream.
- the well run place seems too keen on micro managing. Estimating every step to the point of cancelling improvement is not 'well run'. I'm sure every book about advanced company management will tell you that.
- the badly run is hmm.. at least partially (if not totally) naive. What are the odds that people not being asked anything don't just talk all day long ? The most probable equilibrium point will be most employees doing a little bit of the mandatory duties in the morning, a little bit in the afternoon, separated by lengthy bits of smalltalk (not they kay kind). I have yet to see one person trying to fix anything in their environment because they had free time. 99% of them will just have a new topic to vent about in the coming coffee / smalltalk pause.
ps: anybody knows talks or books about people operating like emergency teams ? where the natural spirit is optimize everything until you reach hard limits ?
Not to sound too naive, I think this whole paradigm creates a bad soil overall. Before I joined $big-corp I was working better and growing faster. Now that I'm in it, I gradually morphed into this kind of employee.. it's a relativistic game and if the system is stupid and rewards the bullshitter, you end up like them (at the risk of your mental health quite often). A good group is one where people are motivated to be sharp, creative, fast, efficient, curious, sharing, open minded IMO.
> “how much CPU time do I have for running this code?”
When someone asks me something like that, more likely phased like, “how quickly does this need to run”, “how fast does this page need to render”, etc., that’s the engineer I want to work with. Pair that with “how much of a benefit would there be if I get it to run faster” and you’ve got the makings of a superstar.
My experience:
Good engineering is about achieving organizational objectives, while understanding and managing tradeoffs.
People who aren’t asking questions of the above style, either fail to see the real-world purpose of their work, or they fail to accept that when doing even semi-competent work, the decisions they make tend to have both positive and negative consequences.
Without understanding those two things, it’s tough to be effective over the long term.
You say that good engineering is about achieving organizational objectives. I say that competent management is about setting organizational objectives. Therefore, you're saying that good engineering is about satisfying competent management. From this POV you see increasing efficiency beyond the level required by management as the opposite of being effective. This is correct from the angle of the game played in the org. This is not necessarily correct from shareholders/other stakeholders' POV - sometimes it is, sometimes it isn't.
But I agree that going against the grain of managerial objectives will make you ineffective in the long term - management will see to it. You need to do less of this or find less competent management if you want to do more of this.
I don’t regard efficiency as one obvious thing. Is it single threaded efficiency? Best hardware utilization? If it’s an application that’s guiding external resource utilization, should you even be limiting your efficiency measure to just your code? How often does that code even run? I.e. does its efficiency even make a difference?
And once we define what we mean by efficiency and come to a conclusion that for the code we’re writing improving it will be impactful, recognizing that it’s only one of many worthwhile engineering goals including readability, test-ability, correctness, security, flexibility/adaptability, and time/cost effective implementation, and come to the obvious (though slightly sad) conclusion that at least some of these goals are always in opposition. We can’t really write code that optimizes for all of these things. So now we have to prioritize, and if engineers are taking it upon themselves to decide which goals are most important to them, without considering the needs of the organization (that’s paying them to do the work) to me that’s not a good sign.
Sometimes managers are ineffective and ask people to do dumb stuff. Sometimes engineers like to ignore the fact that by some metrics their work is always going to suck and avoid the mental overhead of having to pick which metrics to optimize for.
Some real examples of these tradeoffs:
* Pure functional code is more straightforward to test and IMO easier to prove correct, but it will likely have more GC overhead than imperative code with mutable variables.
* Techniques like STM can lead to much higher throughput on multi-core systems (while effectively eliminating entire classes of concurrency-related bugs) so you can do more with fewer servers, but it introduces some overhead on each invocation.
* More effective route planning software could be much slower and CPU-intensive to run but that inefficiency may be dwarfed by vehicle fuel savings.
* In the language I work in, monomorphic code (all things being equal) is always faster and more efficient than its polymorphic equivalent, but I often use and implement polymorphic functions so I’m able to handle many types of data down the road.
> that crazy fuck who went on and on about how your source control sucked and should be completely different, and then used a single dot character, “.”, as the commit message when he finally committed something.
I've actually worked with this guy and the description is so perfect I wonder if it's literally the same guy.
Only in our case, the company was too incompetent to fire him and he ended up leaving claiming we were "bullying" him by reviewing his code and pointing out his mistakes!
Well, I'm 2 days late tk the party, but here I am.
I know of several groups that have the best of all worlds for a painful learning cost: apprenticeships. I know that the typical job growth of a apprentice involves doing your tasks, anticipating, then moving to management if you have the propensity toward that. The most intense desk jobs of the accounting world is the "big 4" which have 2 cycles: boom and consolidate/bust.
The boom cycle promotes the technical "senior" to the management position and requires the manager to review the work of the guy who just got promoted to senior. The manager becomes more concerned about roi over time and eventually goes into sales/estimating.
The consolidating cycle is where the fat is cut because it is "up or out" to maintain a pyramid structure. Those who get the boot often manage well (due to the high exposure) and are confident when starting a apprenticship business of their own (have technical and management and client service skills)
This pattern is similar for all apprenticeships I think like plumbers, cement masons, accountants, etc. The work product is well known, the skills are transferable, and correctness are easily observed by others with the skill.
The pain is when a manager takes tips from a "competent manager" as defined in the article who then has subordinates prioritize low roi activities like formatting or making it clean from sawdust before closing a wall up. Soon a roi-focused manager will look over the competent-manager's shoulder and say why is your team slow, and the answer is "they are slower than me (because they are being trained to look pretty) and i am telling them to work fast, maybe I need to tell them that they are low performing" and turnover increases until that manager stops training people as the manager isn't the client.
Ricardo Semler experimented with this at Semco Partners and wrote a book about it which I read in the 90ies. It was called "The other way" in it's Norwegian edition. In English I think it is this one "Maverick! The Success Story Behind the World's Most Unusual Workplace". It was a about self run teams in and corporation democracy.
All though the article has an 100 percent correct description on how things can be in an organization, I do believe that what he describes is a badly run corporation, not a well run. I also believe that the way Semler reorganized his organization is an anomaly. It seldom works this way.
People, us all included, does not work well in cooperation when we are in large groups, without some sort of guidance. In corporations the most effecient guidance are incentives. That said, setting the right incentives and be able to adjust as you go is a very hard task. There is no "one solution" to how this is done, but if you succeed you usually reach your targets.
I don't trust someone claiming to have run a heterodox org as much as someone claiming to have worked in one, like Creo sister comment threads.
Any incentive turns perverse, and incentives are not how our zillion cells are prompted to make our bodies work. Incentives work to an extent, but have inevitable adverse effects.
> If you’re used to such sprawl, you’d be surprised how effective sleepy HR practices are at preventing it. Suppose you always get a standard, shitty raise at the end of the year by default, unless you bargain loudly, which works rarely and only if you’ve really made an impression throughout the year. There is no defined budget for raises; every significant raise is hard to get, and you never get it proactively without bargaining, but there’s no formal system to avoid spending too much on raises except for the reluctant, reactive approach to giving them. There’s also no system for firing low performers, and it’s only very rarely that you see anyone fired - like that crazy fuck who went on and on about how your source control sucked and should be completely different, and then used a single dot character, “.”, as the commit message when he finally committed something.
I am interested in the rest of this thought… “suppose you only get a standard raise unless you speak loudly”. Then do what?
I took his point to be: some people will always get incredible work done no matter what. Enable them to still get paid well. For everyone else just keep them around for when there is stuff that has to be done, and when there isn’t (most of the time?), let them spin around on their chairs/compensate accordingly.
This is all true if there is a single hierarchy that everything is directed through.
What works better is introducing forms of competition, so that you might have different shops with overlapping, even duplicative responsibilities within the organization. This always appears wasteful to managers, but is the only way I've seen for change to occur in larger orgs.
As Adams noted, people really don't like competition as it forces them to work and innovate. They will go to great lengths to create a situation where they don't have to and can just coast.
Incompetent management is where the leadership sees management as a line of business vs a role. Places that operate that way tend to latch on to micromanage finance and accounting. When organizations operations or development teams are run by the CFO, only bad things can happen because the CFO’s incentives are not aligned.
I ran a large line of business for a big government agency during the pandemic and faced a similar issue. Government bureaucracy is not typically associated with visionary leadership. However, faced with this conundrum, I called the CEO equivalent and said “We need this.” (In so many words), and later that afternoon we were moving along with critical procurements.
Competent management has tight process controls to prevent waste. But the leadership is the head and the process the tail. If a VP (or whatever) is slave to some bean counter in accounting, that VP cannot achieve her mission. Leadership means we follow the process, but have the ability to act.
In smaller orgs, much less formality and lighter process makes more sense. You don’t want to be running romper room, but the processes required to run a global company like Exxon or JPMC is
inappropriate for a startup.
> Bug fixes work a lot like efficiency improvements, the main difference being that competent management makes things much worse. You can’t make fixing bugs into a “goal,” same as you can’t make optimization into a goal, because people will just add more bugs up front and then fix some of them.
If competent management makes things worse, then it isn't really competent management. I'm not sure why introducing additional process is considered a sign of competence
Instead of setting arbitrary goals, couldn't they just pay attention to the things that people are actually working on and give out rewards/punishments based off of their own judgement? If you need to create some elaborate resource quota CI/CD system because you can't trust employees to do things decently, then it's a people problem
The post is saying that it is a people problem. The gist is that efficient or beautiful architecture does not mirror efficient or beautiful management.
> But if there’s a reliable & scalable way for vigorous, systematic management to reward the spontaneous human drive towards efficiency instead of punishing it, I am yet to see it.
This is the gist of the article, it has nothing to do with software architecture
Reminds me I registered the domain incompetent.mangement a few years back during a tedious meeting when I discovered the joys of word games with gTLDs.
> Mostly incompetent management which is very bad at setting and achieving goals is perfectly capable and all too likely to cargo-cult effective management by setting up an elaborate bureaucracy for assigning work and tracking its status, thus preventing work from happening spontaneously.
I have recently started to recognize this pattern of idle development in some very well known software projects.
What happens is this.
At a certain time the team delivered a perfect version. So perfect that nothing is needed to be refactored or added to it. It's done. Then the team immediately gets bored. In order to protect job security, the team invents new work for themselves, a huge architecture refactoring, that sort of things. Sure, why not, we have time and resource, reduce technical debt, etc.
A couple years go by and now the product is a rotten swollen mess with tons of useless verbose documentation, tons of integrations, a mind-boggling number of features and, of course, AI. And the product cannot even perform its original task because that was abstracted away a long ago.
Not gonna point fingers, because I'm gonna immediately get downvoted to hell. But we all know examples of that taking place.
On the other hand, money is not free: many poor souls worked their ass off to pay for our toys. Putting their work in good use is the least of respect.
> What constitutes managerial competence? As a vague starting point for an answer, we could say that competent management sets achievable objectives and then achieves them, by organizing and incentivizing the necessary work.
This is a terrible definition, and the author proceeds to make their entire argument based off of specifically the shortcomings of this definition.
Any idiot can set an achievable objective and then achieve it. That's not competence, it is the bare minimum level of functionality to claim something is being managed. Having people dig a ditch and immediately fill it back in, over and over again, would be labelled as competent management under that definition.
Competent management identifies critical objectives and prioritizes resources such that they are accomplished adequately and efficiently.
If your goal is to get something done, then yes, obviously having more budget to do so would be better. But that's never the actual objective. The real objective is to advance the organization's goals - increase profit, raise capital, gain market share, avoid liability, what have you - which means making optimal use of finite resources.
> But wait, seriously though, what’s the math here? What are we maximizing? Revenue minus cost? Revenue divided by cost? I mean, shrinking the cost has got to be helping with these?.. Well, sure it’s helping, but it’s not helping you, because you don’t bring any revenue by yourself, unlike cost, which you very much do incur all by yourself. The math with you is, we tell you to do something if the cost is below a threshold. If you won’t commit to doing it cheaply enough, we’ll find someone who will, and if we can’t, we won’t do the thing, or reconsider the options in some other way. But exactly what the cost below the threshold is changes nothing in any math related to you, except for a lower cost making your job harder, since you have the same objectives to achieve. The firm’s bottom line - sure, lower costs help there. But the impact on the firm’s “revenue - cost” doesn’t trickle down to your “cost < threshold,” because you have no revenue.
This entire line of thinking is incompetent management all the way down. At the top level, they have either failed to recognize their critical objective (improving the revenue to cost relationship), or failed to allocate resources to accomplish it adequately and efficiently (setting up an incentive structure where middle managers can successfully deliver). The middle manager in this scenario is not even doing any management - the decision of what would be done and what resources would be allocated was made above them. This isn't a manager, this is a messenger assigned to communicate orders from on high to their peers. Perhaps they will get to make some lower level decisions over how exactly their team accomplishes the goals, which could yet be competent, but given that their goal is advancing their corporate fief it seems unlikely.
> What happens in a badly run place? In a badly run place, management is bad at setting objectives, so you have people aimlessly wandering about, lacking clear goals, and just doing stuff because they want to. They see an optimization opportunity and they gladly pursue it - it’s interesting, it’s fun, it’s good for the company, it’s what they’re here for... they don’t mind shrinking their resource footprint, because nobody monitors the resource budget properly, nor presses them to meet any targets very hard.
That's not a badly run place. That's just a place where people aren't being micromanaged. Top level leadership is supposed to give broad direction - are we more concerned with growth or cost reduction, for example - and make sure that there are enough people who are sufficiently motivated to make that happen. The specifics of how they can best accomplish those goals ought to be determined at that lower level either by the individual employees or, where collaboration is necessary, the team level. Incentives, targets, budget monitoring, these are all tools that management can use to solve problems, but it's not a failure when they are unnecessary, it is a mark of competence.
This post is mostly half-truths, I'll just stick to the core errors:
> Of course you could say that this is a badly run company, and to avoid arguing what that means, let’s stick to the definition of managerial competence as the ability to set and achieve objectives. Whatever objective you are expected to achieve, a bigger budget makes it easier.
Yes I would say this is a badly run company, and for some reason they just glossed over that. Probably he glosses over the "set" part of objectives and focuses only on achievement, when setting the right objectives is part of good management. Also bigger budgets don't make every objective easier to hit; it's certainly easier to make $1M of revenue with a $2M budget than a $500k budget but it's not necessarily easier to make a 20% profit.
Let's distinguish between two kinds of management and managers. There's high-level strategic stuff (ie should we deploy this proposed product line) and more operational stuff (which vendor should I pick). Likewise there are different kinds of objectives. A bigger budget will make some objectives easier, like a revenue target or a shipping date. But it won't make it easier to hit a certain profit margin or return on capital target [0]. For-profit companies should always be focusing on returns on investment. For government/non-profits the overall principle is to use money effectively; measuring that is harder but it's definitely not making Tom Everyworker feel good about his clean code. It's delivering value to somebody who doesn't work at the org.
If the senior management has set terrible objectives, and middle management achieves them effectively, then yes, bad things will happen. I would call this bad management; even if most of the managers are good the management overall is still bad. Bad middle managers might be better for the org overall in this case, if we assume workers are all angels who dream of nothing more than doing the "right thing", all agree on what that is somehow, and agree with the authors definition.
On the other hand, if senior management knows what they're doing, they'll set good objectives. It will involve profit not just revenue. It will involve delivering products/features customers want, not delivering things they don't care about, and doing so at a low price. If a division doesn't bring in profit directly (eg IT) they'll figure something else out which rewards good behavior and doesn't punish it. And then competent middle management will achive those goals effectively, and achieve the goals of the org. Note that making Tom happy and letting him ship as much code as he wants is at best only a small piece of those goals, so Tom might actually be unhappier in this scenario.
[0] Assuming there are no accounting shenanigans involved, which again, is part of good management.
I liked this article. It's original in thought, and gave some attention to why things work stupidly, and are even upheld.
Let's now turn to what one does about. I want to pick one issue to discuss here. Consider, this excerpt:
"Great idea - now I can commit some CPU or memory hog, then you can fix it, and we’ll split the reward."
Here and in other places the author implicitly refers to an underlying tolerance, existence, or undertone of lying, BS, and effete manipulation. In my time in corporate America I've not seen code like that, but I have heard, seen, and been at the brunt of similar actions:
- I manage up and down (a boss I once had said this) ... meaning ... I'm selling up and down the chain of command to put the spin on it I want. And if you're not doing this you're a naive moron.
- The same guy orchestrated work in our team that was sub-optimal (details unimportant here) precisely so his team would be involved, and have work in a larger project.
- The same guy, prior to me joining his team and unknown to me, went around the corp getting internal customers to use a particular service. What internal customers did not know, what that his intent was to make it a one way join: once the customer's code was integrated there was no practical way out. Besides job security, the ultimate aim was in his words "take over the floor by taking over the data."
- Internal service groups whose visions for what to do are what they want to do, and not what their ``customers" need. I've beat that into the ground in other posts. I will just say here: internal service groups MUST be customer driven just like they are on the outside. A roofer can tell me his vision, his price, what he wants to do etc.. I can agree or not. But ultimately I tell the roofer if he's value add or not, if his prices are too high or not, if his work is any good or not. And I will pay or not based on what I see. He doesn't tell me any of this. However, in corp america internal customer's don't have that agency because asshole internal supplies know management implicitly or explicitly will make customers use them.
- My better half works in cloud migrations. JFK the BS, games, lying that goes on to protect internal data centers is beyond even what I've been through. But I can say this: a leading reason companies move to the cloud? They are sick and tired of crappy internal servicing from those who run the private data centers. With-holding costs of labor, data floor space, or lying about it was common in her telling.
Where am I going? A good way to improve the climate at a company, which MUST BE DONE STARTING C-suite, then top management, the middle management, then TLs: BS, lying is fireable. As soon as people learn a company does not pay people to be cynical, that's already way better. I say starting at C-suit, because I learned the hard way substantive change will not happen unless it comes from on-high. See Ishikawa's tunnel analogy in "What Is Total Quality Control?: The Japanese Way." which is a focus on management's roles and responsibilities in quality.
Some examples you right read about where that's worked well: Berkshire Hathaway; Renaissance Tech. An old boss (a Chem-E) in Silicon Valley was a guy like this. These people don't like problems or mistakes ... making one is not a crime but not consequence free either ... but lying/BS always makes them worse, harder to find, and harder to correct.
I got the guy I mentioned earlier kicked our of our division. So I mean what I say.
Openness/truthfulness is a an critical intangible to good companies.
> My better half works in cloud migrations. JFK the BS, games, lying that goes on to protect internal data centers is beyond even what I've been through. But I can say this: a leading reason companies move to the cloud? They are sick and tired of crappy internal servicing from those who run the private data centers. With-holding costs of labor, data floor space, or lying about it was common in her telling.
This sounds backwards. If someone doing "cloud migration" is around to observe, that means the internal data center people know their heads are on the chopping board, and desperately trying to preserve their jobs. You cannot expect honesty from someone fighting for survival, and it's disingenuous to assume they deserve it because they act that way.
I think it worked pretty well for quite a few years. It gradually stopped working when the company acquired a large company with a different culture and also hired people (well, managers mostly) who weren't aligned with the culture. Eventually this basically disappeared when the company was acquired by Kodak.
I've seen flavors of this in other places. Famously Andy Grove of Intel also preached that decisions need to be made by those closest to the decision and empowered people to make the right decisions. More generally this can be reflected in a servant-leadership model where leadership sees itself as facilitating the growth of the people underneath them.
Another requirement for this to work well is that management (e.g. the CEO or other leaders) are able to lay down a broad strategy for the people of the company to execute on. If the leadership has no strategy then tactical decisions can not be made properly. They also need to make sure there's coordination and structure.