It is amazing how culture problems often start with the CEO. Sometimes, non-technical CEOs are very good at their own speciality (selling, motivating, etc.) but extremely bad at systems thinking.
Other times, CEOs hire yes-men and yes-women below them and everyone is afraid/unwilling to tell the CEO the truth.
Other times the company has raised a ton of money and attracts people who would otherwise not work at a startup and whose major focus is resume building and moving to the next job.
If the CEO is the only person meeting with investors or only trusts one or two other people to meet with them that is another big concern. It means there is likely no transparency into the actual team and what is actually going on, which means the CEO is likely BSing the investors.
All of these are reasons why you should not work at a startup if you do not 100% believe in the CEO. Be skeptical b/c most startup CEOs really do not have a track record of insight and integrity. If the CEO is aloof or seems out of touch and not someone who you think would really enjoy talking to you about your job and who respects the work you do, be wary.
Replace CEO with "owner of small business" and this is what I see endlessly. The chops needed to build a business and sell a product are a long way along a spectrum from how to grow that business and how to keep the codebase stable.
Agreed. Amazingly, some of them are even "technical", but have no concept of technical debt, and zero respect for software engineering practices that don't lead directly to new features. Either they never understood these things in the first place (since this is usually not taught as part of a degree in CS or EE) or they've long since forgotten them.
You can be "technical" without a need for a concept of technical debt.
- Indie video game designer/developers, for example, live in a world where everything they write is a one-off product that will ship, at most, a v1.0.4 six years after v1.0.0. To them, code is a medium for crafting an experience, and whether the code is of quality is immaterial as long as it creates the correct experience.
- Data scientists (and bioinformatics pepole, and a few other similar groups) live in a world where everything they write is effectively a script-as-experimental-apparatus. If the script is producing output conforming to the proposed experimental method, then it's done. They run it, then discard it. It's not usually even under version control; only recently have such scripts been considered worth including for purposes of peer review and replication.
And there are other examples. I'm not saying that these kind of people are "software engineers"—engineering of any kind is never even a consideration for them; whether the product is well-engineered has no impact on their livelihoods.
But these people frequently are lumped into the category of "programmers", and that tends to confuse this sort of discussion a lot. The "software industry" doesn't just contain engineers doing engineering (well or poorly); it also contains prototypers and illusionists and tinkers and scientists, all working to their own criteria of art, none of whom will have any reason to pick up a single shred of engineering knowledge, even with decades spent in their own industries. And then these people apply for "programming jobs", and we wonder why they don't know what git is or how to properly modularize a codebase for maintainability.
It's a communication problem, I think. We need to stop calling everything we're doing "programming." Maybe we need an entirely new verb for doing software engineering, of the kind you do when you're writing something like car firmware, or Amazon S3, or the Googlebot. "Programming" is about as precise in describing that as "doing math" is in describing the invention of the lambda calculus.
I've toyed with the idea of having separate titles & job descriptions for "Software Features Engineer" (those responsible for pumping out user-visible features) and "Software Infrastructure Engineer" (those responsible for the long-term health of the code). In my experience, they really are completely different roles, down to the languages used (features engineers tend to use Python, Ruby, JS, and other dynamic languages, while infrastructure engineers tend to use C++ or Java), attitudes, personality types (infrastructure engineers tend to be much more attentive to detail), priorities, and points in the lifespan of the product when they are most useful. There's already precedent for creating a new job title to solve an organizational problem, eg. Site Reliability Engineer.
Many big companies have such a split in their org chart, with separate "infrastructure" and "features" teams reporting up to different managers, but IMHO that's the wrong split. Very frequently you want a cross-functional team with both infrastructure & features engineers on it, but they remain different roles, just like you might embed a UX designer and UX researcher on a product team even though their job descriptions are very different.
I have seen this taken further into the personality types:
Starter - someone who can solve design problems and pump out new features
Journeyman - someone who can be given a spec and turn out solid, maintainable code. This kind of developer is also good a debugging and resolving edge cases.
Finisher - someone who is really good at completing and polishing features/applications
A single developer is likely to be a mixture of Starter/Journeyman or Journeyman/Finisher. It is rare to find someone who is strong in all three traits.
I think I am more of a Starter (too many design ideas) with some Journeyman. I don't think I am so strong as a Finisher (too many other distracting design ideas).
"Software Features Engineer" vs "Software Infrastructure Engineer" is an excellent differenciation. Actually I'm an SFE and I have always felt that I am not a real programmer. SFE will always write more-shiny less-durable features, more often working under an unreasonable management. I feel like I'm tainting the reputation of real SIEs.
Question is: I've always assumed I had to evolve into an SIE or die (or become a Product Manager if I'm lucky to be better than 90% of my colleagues at understanding users). Is it a correct picture of the reality, or is it possible to strive as a "good" SFE?
No, but it will take you down a career path that exposes you to more luck & uncertainty than an SIE.
Features engineers can have lucrative, long-lived careers. It usually means they need to be the primary or lead engineer responsible for a significant user-visible product. Think of Andy Hertzfeld for the Macintosh, John Carmack for Doom/Quake, Anders Hejlsberg for Turbo Pascal and Delphi, or Aston Motes at Dropbox. Some of them cashed out directly with F-U money from a startup; others used past track records to get a startup of their own funded or a lucrative job at a big company.
The downside is that you're very exposed to the risk of your project failing and nobody caring about it. At the stage of a project where a SIE is most useful, the project is already underway, people know it's a good idea, and they just need to execute solidly. At the stage where a SFE is useful, the whole point of the project is unknown - that's why you need an engineer that specializes in rapidly testing things out. This applies in big companies as well - my role at Google could best be described as a SFE (though they don't have the title), and I had to prototype a number of features, a couple of which worked out and a good many of which did not.
There's very significant crossover between the SFE role and that of a technical cofounder - the main differences are risk tolerance, willingness to go out and interact with users personally, and product vision. It's very common for people who enjoy the prototyping aspect to end up starting their own companies, as that gives the best alignment of incentives and ability to capitalize on their own work. It's still possible to be a "professional early-employee" or even make a prototyping/R&D role work in a big company, but you need to demonstrate wins in your career.
Indeed, I have created my own company ;) It feels like a failure compared to the steady money that my SIE ex-coworkers have, but I enjoy it much more. Your insights give me good context to visualize my career, thank you.
I've always liked the way the games industry approaches this: a "game designer" (at least in healthy companies) is a developer whose job is effectively to produce a continuous stream of prototypes that encode all the business logic. You can use the prototypes for smoke testing (in the games industry, seeing whether the resulting game is fun), but can also use them to write component-level contracts based on the observed properties of the prototype.
Once you have such an "empirically-derived" contract, you can then throw that over to a larger team of actual capital-E-Engineers, whose job is to make something obeying those same observed constraints (which are really quite detailed; they basically get to do TDD where many of the tests have been written for them), then apply actual engineering to the problem, to output another, non-prototype product.
This scales up: your stream of prototypes can come from a team, instead of an individual; and you can have as many people doing engineering as you want, because they have a stable set of requirements—translating a fixed, interactive spec (the prototype) into good code—that allows for simple, static top-down software architecture.
The interesting thing is that successive prototypes will tend to be frequently ground-up rewrites (because prototype code is unmaintainable, and is also small enough that there isn't much lost in throwing it out.) The engineered product, meanwhile, will be a continuously-evolved codebase, where each new prototype marks the start of a new branch of development, new user stories and issue tickets, etc. Features might not always make it from an arbitrary successor prototype into the product!
Note also that the game designer not only encodes the requirements in their prototype, but is also the perfect place to lay the responsibility of collecting the requirements in the first place. Effectively, they're acting as a tiny embedded start-up, pivoting around and trying to find product-market fit, where the "market" is internal to the company.
Or, another way to think about it: your "features" team is like a four-piece garage band producing music (or even, perhaps, one guy with a keyboard producing chiptunes.) Those songs are played to internal stakeholders, who decide whether they like them. Then, the approved songs get thrown over to an orchestra conductor (the given team's engineering lead), whose job is to turn the motifs embedded in the piece into a symphony, and get their 100-odd performers to create a coherent, grandiose sound out of it.
A lack of understanding how to set up this kind of pipeline might be why (in the best cases, at least—let's ignore the acquihire-and-disassemble cases) so many large companies choose to continually buy start-ups. Effectively, they're doing the same thing we're talking about here: taking a small team with a finished prototype, and then reimplementing it "for real" using their engineering talent. The problem being that there's no support for the continued existence of the absorbed "start-up" mitochondrion within the host cell, so it only ever gets out the one product it was working on at the time it was acquired, rather than continuing to pump out new little ideas. (A great game-industry example: EA. They constantly buy fledgling game publishers, use their own engineering talent to produce a sequel or two, and then run out of steam because the people making up the creative engine that generated the original game designs have left, or have been reformed by their engineering culture into something non-startuppy that has no power to generate original products.)
I'm also of the school that different words or terms should have different tangible meanings.
So for example a programmer is a distinct concept from a software engineer. One can be one without the other, or both. I like to think I started as merely a programmer but... now can also say I'm a software engineer.
Because I do try to think about the full spectrum, the trade-offs, the real world concerns, as much as I think about how to see a certain computation or result occur, in isolation.
What I would call a developer (or duh-veloper, to be less diplomatic) thinks no further than how to see a certain computation or result occur, only, and only as per written reqs, then, moves on into the proverbial sunset. Even if it doesn't handle edge cases, or unwritten desires, or doesn't scale, or is hard to upgrade or support, etc. A true software engineer should consider and design for all those considerations. Not simply how to achieve 1 + 1.
I'm upfront and start the conversation with, "You employed me to give you an honest opinion. I think we are going in the wrong direction, we need more tests and some refactoring. This will cost us money now, but save money later." and I always get the reply that sure, they totally understand, but we're going to do a quick and dirty hack today and sort it all out next week...
I used to just refactor and test as I went along, but then you end up ruffling feathers and being slower that others. This is then used as ammo against you. So you lose by doing the right thing.
I think pouring energy into other things (open source) is more productive.
> If the CEO is the only person meeting with investors or only trusts one or two other people to meet with them that is another big concern.
I know you said 'concern' and not 'problem', but I generally do this and have a totally different motivation for it.
Our guys and gals are really busy. There is a lot of work (often very specialized work) that needs to get done, and it's important to shield them from distractions. Investors generally tend to helicopter up -- they see our world/company from thirty thousand feet. It's a totally different perspective that takes some time and experience to get used to. Even most CEOs have trouble shifting into this mindset. It requires an enormous context shift. For people who don't have a lot of experience with this kind of thing, it can be dramatically harder to do it on a whim.
A minute ago you were sitting at your computer, trying to figure out a C++ boost error message, and now someone asks you "how do you think the feature you're working on will impact the company's revenue in the next quarter?" That can really throw people off. They might feel like they should have had a better answer. Investor conversations can seem extremely important to people who aren't used to doing it on regular basis, and it can distract them from critical work. I think it's good general practice to try and avoid it.
That doesn't mean undue paranoia -- if someone wants to talk to the investor (or the other way around), they're welcome to. But I think being thoughtful about protecting your people that are neck deep in work from distractions is usually a good policy.
I think at a certain point a company reaches a size where there are functional units that generally do independent work and the founder has little to do with it -- if an investor has an interest in that area (or ideally experience/advice to offer) it makes sense for some members of the team to be involved in the dialog, though probably not the guy writing the algorithms in c++.
Our CEO hired a VP of e-commerce last year, before there was just an over-worked director handing me down directives from on high. As the sole back-end engineer inheriting an ancient codebase, I've been able to keep things reasonable and clean and not build new crap on top of old crap. But it's taken the careful exercise and maintenance of leverage.
Now we're gaining traction and pushing back on the on-high directives and coming up with a solid long-term plan.
I shudder to think what it would have looked like if there were a whole team of engineers with nobody to add their voice to the leadership.
Engineers don't help this problem when they treat management like idiot work.
Engineers don't help the problem when they treat management like glorious god-like perfect beings either.
When your management is bad, there is nothing an engineer can do. They can attempt to bring their concerns to leadership, look for another job, or play the fiddle while the product burns to the ground.
Speaking of bad management, I was very surprised when he claimed that the situation where you are asked to "...make architectural decisions on the fly without knowing anything about what their problems are" he had never experienced before. In all of my experience, it was standard practice.
> When your management is bad, there is nothing an engineer can do. They can attempt to bring their concerns to leadership, look for another job, or play the fiddle while the product burns to the ground.
Well, they could step up and start fixing problems in the organization themselves. Improving communication flow between departments, building better processes, discovering metrics, these are things that you can do to help make your company better without having to get the leadership involved. Not every improvement needs grand strategy, there's often lots of low-hanging fruit that one person can own and push by themselves.
Just Friday I was sitting in on a planning meeting and realizing that while we have a front-end guy, we don't really have a decent web designer that can draw out ideas and come up with plans for implementing them. So now I'm thinking about how to go about the task of managing that information flow.
> Improving communication flow between departments, building better processes, discovering metrics, these are things that you can do to help make your company better without having to get the leadership involved.
This requires someone with a surprising amount of knowledge and ability, outside of the normal field of software development. It's a rare skill to find in anyone, let alone a highly skilled tech worker.
Ideally everyone is thinking about the processes and systems that impact them critically, but even that requires a minimum of knowledge and ability to think in systems. It's one of the most under-appreciated and ambiguous skills to learn.
It also requires that the engineer be within an organization where they have a reasonable expectations that their attempt will be well-received. I know I have spent time in organizations where attempting to go above and beyond your "place" as an engineer is punished harshly.
You could talk about all the things it 'takes' to be great at your job, or you could just come up with your own idea of greatness and chase that. Eventually you'll outgrow your current position. Whether you get promoted or find another job at that point is an implementation detail.
That's the theory, yes. In practice, many engineers are averse to unemployment, and will be discouraged from pursuing particular notions of greatness that risk it.
Unfortunately, it's not so easy. If the culture (i.e. tech is not really important, business comes first) is wrong, your effort is not gonna fix a thing.
This applies to more than just start-ups. I work for a company that regularly claims 'we are not a software company'. But... we're writing software. So yes, we are a software company.
It's amazing how often the "we're not a software company" phrase is bandied about. My employer has been around for a while, the IT department has over a 30 year history. Yet, we have the phrase repeated to us whenever we discuss using source control, developing libraries, having a web services strategy, etc. It is like saying, "We're only a youth hockey team, not a professional one, therefore we don't need to use the same equipment and strategies as the pros."
Just because you're not planning to be [put your favorite professional software company here] doesn't mean you should avoid learning their strategies for dealing with everyday problems associated with writing software - unless you don't write software at all.
In the end, the phrase is nothing more than an excuse for avoiding the risk of cultural change.
If you're working at a software shop that doesn't even use source control I suggest you get out of there ASAP. It's one thing to be missing core tools and methodologies when you're applying for your second job 1-2 years in, but if you hit the 5 year mark and have zero familiarity with source control or web services (for industries where this is valuable), etc. you're going to get passed over for interviews. Staying there may be causing irreparable damage to your career.
Saying you shouldn't use source control because you aren't a software company is like saying the janitor shouldn't have a mop because you aren't a cleaning company.
Couldn't an engineer go slumming into the world of office politics and social manipulation in order to get into the good graces of management? Might that not be a possible approach? I realize how odious it sounds and I don't know how well I could do it.
This was a very interesting read, but I was a little disappointed that it didn't leave me with a sense of resolution. Did company B get "fixed?" How? How have other companies resolved culture issues that stem from the leadership at the top? Are companies with these kinds of problems just doomed to fail?
Usually companies like this can fail for a very long time. As long as someone is willing to pay, you can live with the slow development speed, the high number of bugs and the constant team changes.
this exactly. I expect they are paying below average wages, have huge churn of people looking for anything to do while the perfect job arises. Throw in some people new to the area who need a job now, some old contractors who have money and are looking for pocket money and more stability (who's CVs are amazing, they have the gift of the gab but get bored really really quickly and just don't care about the job) and that is a the instant recipe for churn and crap code.
Yeah it did feel like a very abrupt stop. Hopefully there is a part 2 at some point. How does the consultant deal with communicating the real problem? She must, at some point, tell the executive team "the code is just the symptom." How do they deal with that? And is there a way to empower developers to do this themselves?
+1 Some great writing but I was also disappointed with the lack of a conclusion.
It's cool to note that even in the valley things are FUBAR'd. I'm a UK contractor still waiting to wander on to that magical project that functions coherently. Maybe I never will!?
I've outlined a blog post series about 30 headings long starting with this at the premise. There is a significant territory to cover, and it's difficult to present coherently.
This topic is traditionally covered under the umbrella of Quality, with the premise that output Quality is primarily a function of the systems, knowledge, and understanding of the people at an organizational level—and is not a primarily technical pursuit, despite what most people think.
We need this discussion in the software world now more than ever. Really glad to see it.
This is what happens when engineers subordinate themselves to non-technical management in a business-driven (as opposed to engineering-driven) organization.
The ironic/tragic thing is that the consultants who are brought in at much expense (and who therefore have some credibility with "the business") are often no better than--usually worse than--the in-house engineers they are supposed to help. If you ever find yourself an engineer in a dynamic like this, you should interpret it as a sign of mistrust--even outright disrespect--of you and your peers by your bosses. Imagine if you were a physician and hospital management thought it necessary to bring in outside physicians to help you with your patients. Not many doctors I know would put up with that, but engineers are culturally so submissive that they go along with it and deal with the Dilbert scenario of the $500/hour consultant giving the same recommendations the in-house engineers have been saying all along (which the consultants get credit for, of course).
If you're ever in a position of authority and feel that it's necessary to bring in consultants to "fix" your codebase, it would be worthwhile to reflect for a few moments on how this situation came to be. Either you failed your organization on the hiring front and have terrible engineers (unlikely), or your otherwise talented engineers cannot do their jobs effectively due to the environment you've created. Perhaps you've forced them to incur so much technical debt by nonstop "sprints", unrealistic deadlines and multiple pivots that any forward progress is all but impossible. Consultants probably won't fix this.
I get the idea in general, but you know sometimes you bring that outsider in for knowledge you don't have in-house.
I'm a full stack developer, but I'll be the first to tell you I don't have a deep knowledge of any RDBMS. If that sort of knowledge is necessary, they're either going to pay me to figure it out, pay someone else to teach me, pay someone else to do it, or some variation of the above.
If you accept that engineering skill fits the normal distribution, it's less likely that they are "terrible" than merely average.
Amazing 10X engineers could still pull off shipping a working product even with all the organizational handicaps I mentioned in my earlier post. This doesn't excuse the deficiencies of the organization, nor does it mean that "average" engineers are undesirable or can't do good work. I agree with the theme of the cited article--rather than scapegoat the engineers and pay external consultants to "fix their mistakes", fix the organization and enable the engineers to do quality work.
"Amazing 10X engineers could still pull off shipping a working product even with all the organizational handicaps I mentioned in my earlier post."
They could. They're also good enough that they could easily say, "Fuck that shit" and go work somewhere that doesn't have that toxicity. And unless you have a significant ownership stake in the company, I don't see why you wouldn't do just that. Life is too short to work shitty jobs if you have the option not to.
There must be a corollary to Conway's Law here... the dysfunction of the code produced by an organization reflects the dysfunction of the communication structures of that organization.
Yeah, I think that's an interedting amendment to Conway's law: it's not just about the structure of the organisation/communication, but also about the feelings associated with the communication.
In the case of an architecture diagram being handed down from on high, sure. But in the case of product feature whack-a-mole, this doesn't seem like something that bottom-up decision-making could fix.
When I was younger, immersed in usability, databases, and big sites, watching the initial rollout of Facebook etc, I thought that a good technical solution could solve social ones.
Now I'm a lot more pessimistic and want to approach it the other way around.
One of the common themes at my last gig is that I would start conversations and presentations with "There are no technical problems here. These are people problems."
Eventually they tried to solve both problems by adopting a technical solution so bad that the people left.
Love this article, but I'm not sure the origin of the sloppy practices described there always originates with the VP/Eng, CTO or CEO. I've worked with plenty of engineers who just don't want to write tests, or tighten up code, or expend the effort to socialize their ideas/accept others. The higher ups might not have the visibility, inclination (or time!) to focus on those issues and encourage best practices, but I don't always detect acts of commission from them to subvert best practices.
I don't know if he was saying it always originates from the top but more that it's always allowed to perpetuate from there.
If management doesn't have the visibility/inclination/time to peer into problems and gain understanding, they are encouraging whatever has happened to continue happening for better or worse.
Agree with all the responses to my top level comment, in that management is ultimately responsible for setting and maintaining the engineering culture. It can be tough, however, to balance competing (tensioned) requirements - if you have to choose between engineers that follow best practices (user stories, standups, TDD, pairing, etc) and no engineers at all (let's face it, they are scarce today) - what do you do? Wait to hire engineers and let the product deadline slip even further? It's a tough balancing act.
If you have developers who refuse to write tests and can't or won't communicate with the other people on the team, then it's definitely a management problem to not get rid of those people. And the culture of not getting rid of people is definitely one that tends to start pretty high up the chain of command.
Other times, CEOs hire yes-men and yes-women below them and everyone is afraid/unwilling to tell the CEO the truth.
Other times the company has raised a ton of money and attracts people who would otherwise not work at a startup and whose major focus is resume building and moving to the next job.
If the CEO is the only person meeting with investors or only trusts one or two other people to meet with them that is another big concern. It means there is likely no transparency into the actual team and what is actually going on, which means the CEO is likely BSing the investors.
All of these are reasons why you should not work at a startup if you do not 100% believe in the CEO. Be skeptical b/c most startup CEOs really do not have a track record of insight and integrity. If the CEO is aloof or seems out of touch and not someone who you think would really enjoy talking to you about your job and who respects the work you do, be wary.