This is ringing particularly true to me right now.

The organization I work for has allowed bit rot to occur for a number of years, to the point that it's becoming increasingly common for things to break whenever even seemingly small changes are implemented.

The "solution" that we've implemented is to require a detailed RFC document whenever you want to create or modify anything. You're required to outline every anticipated step along the way, all the tiny minutiae that you foresee yourself encountering and how you'll deal with it. And in a Google doc, no less.

It's utterly exhausting.

At some point I got so pissed off about this that I started assigning code reviews to him - he contacted my manager and asked that I stop, because he had no understanding of the project in question.

What did this mean from the documentation side?

I would have loved to accurately document what I was doing and how the project was structured - but I actively avoided doing so.

Every time my manager's boss knew what I was doing, I was about to be shat on from great heights.

Months of work and planning erased at the drop of a hat - time and time again.

Dependency injection? Fuck off. Spring's too heavy.

Modularity? Fuck off. The project's too small.

Hierarchical profiles? Fuck you, keep using a properties file (with an increasing number of dynamically generated keys).

I was constantly asking for a new developer to be brought in for the transfer of knowledge - at best I was told "That's not a bad idea."; yet nothing was ever done about it.

I took an unmaintainable pile of bugs and made it stable and maintainable. But nothing I ever did meant that I was someone worth listening to.

Eventually I got so burnt-out (and consumed by feelings of uselessness) that I walked away without finding another job.

I was damned near suicidal from fighting for acknowledgement.

I cannot imagine that my departure went too well for them.

> I took an unmaintainable pile of bugs and made it stable and maintainable.

Same here, except it was supposedly using all best practices e.g. Spring Boot, DI/IoC, Mock testing framework, YAML configs, extremely elaborate package hierarchy and so on. The only thing it did not have was working solution for the business problem it was written for.

I think two things play into this: Layers.. it is just easier for people to add layers around something, not touching and understanding the original code and doing it right there. And then profs or someone else always says layering is good! (It may sure be, or also sometimes such "hacks" are the right thing to do, but if that is always the default mode... uhh oh).

And second, university training: OOP, design patterns, software layers, all claimed to lead to structured good programming automatically still taught in 2023, which is just not true. By untrained or unexperienced personnel leads to worse than 80s spaghetti code..

Just yesterday I exercised with relative who is in university for exam.. Java, design patterns, arggh. Imo it just makes no sense at all to try to teach students inheritance and design patterns with Cat/Dog/Animal et al, it is so backwards. You need to program for a while bigger real systems to see some patterns emerge and their usefulness, and then can relate to the formal way and maybr use some patterns reasonably. But starting with those on toy examples just leads to what you describe: People going in with their trained knowledge and applying that to everything without rhyme or reason. What could be a simple five liner function already becomes a convoluted abomination. Just one concrete example, but this then on many levels of software engineering.

Who or what will save our future? :D No, it won't be ChatGPT. (Second lol btw, relative really showed me how likely all of then may use ChatGPT i. their online exams...)

Most people do not have the memory span to work with mutable state, so something simpler should be the default.

1) Proper FP languages with good enough semantics and necessary features to write functional code effectively. Those languages all fall into category of "too difficult" for the majority of industry, which seems to suffer from severe allergy to learning anything beyond "Intro to programming in Java 6" freshman class.

Even things like lambdas only recently stopped being too complicated for a typical programming shop that follows well-known best practices, such as "Programs must be written for inexperienced non-programmers with no prior exposure to the problem space to read, and only incidentally for machines to execute.", and "Would somebody think of the juniors?". Closures? Nah, the zeitgeist is still on the fence with this.

Yes, I have a sore spot for this. I've heard this reasoning many times in real life, and once was even target of it - an otherwise competent programmer and a good manager once asked me to stop using lambdas for event handlers in Java 8 (which the company had only recently adopted), for the sake of the juniors who won't understand. In fact, he meant specifically his favorite evergreen junior (who's been in that company a couple years longer than I was then). But guess what, couple juniors actually asked me to explain the how-and-why of lambdas to them and got it near-instantly. Said evergreen junior probably spotted the lambdas in the codebase too, or otherwise got interested in this, read up on the topic independently, and three months later was evangelizing basic functional programming in Java and otherwise upskilling his fellow coworkers.

But yeah, it's too difficult, your co-workers won't understand. Anyway, where was I? Oh right:

2) The other, arguably larger, kind of FP adoption: library-level and code style level implementation of FP features in languages with little or no natural support for them. This doesn't mean people understand FP extra well - they're barely aware of it at all. But every now and then some library gets popular - say, a sum type for error handling (typically known as Result, Either, Expected, or something similar) that's "obviously" "better" than exceptions. Except, that library is doing a half-baked implementation of a concept (e.g. monads) that doesn't fit the language, and the users don't understand what the approximated concept is, nor are they supposed to try (remember, FP is too abstract for a freshman after "Java 6 101" class).

The result is, as expected, an ugly mess in the codebase, displaying all the bad sides of imperative and functional styles, and leveraging none of the good sides.

Anyway, what is my proposed solution to this? For the industry to stop hating education and professionalism. To embrace the idea that the job of a junior is to learn, not to be the yardstick for complexity of the codebase. To encourage a culture of learning, of professionalism, at every level. To stop dismissing or banning techniques that reduce size and complexity of the codebase by large factors, just because one has to invest a couple hours of learning up-front to be able to use them. Etc.

</rant>

1. The developers who create the software.

2. The rest of the company developers work for, and possibly other people in other companies - who all make money selling or renting the software.

3. The actual users stuck with the software.

I'm not sure if Agile meant 3 > 1 and 2, or just pretended to, but actually meant 2 > 1, but your comment does read as advocating for 2 > 1 > 3, which is arguably the current industry standard.

I was dealing with having to support releases across multiple historical branches, and wanted to separate the core and business logic to allow for easier support on the older branches.

I needed the dependency injection in order to break up cyclic dependencies across modules (without which the core and client logic could not be broken into separate repositories).

Internally, the deployment tooling treated the profile as hierarchical; when I took over the project it was already using dynamically generated keys to store trees of data in the .properties files it was using for user profiles.

It only became more hierarchical as time went on and the internal representation fully so - using a tree structure internally allowed for very clean subtree selects (which helped to clean up an entire class of bug, introduced on mass, during a particulary ill advised (and strongly advised against) piece of work (completed (despite being given without notice) in a single iteration - when it really needed at least 2).

I would still have preferred to use an external format that allowed me to specify or generate schemas for the user profiles, and it also would have helped with supporting profile versioning (which would have been valuable for company internal users).

That's the thing about places like that.

My departure would've been less than ideal.

Either way, the company's haemorrhaged talent (now including long term CEO and long term CTO) and the stock price has plummeted since I left.

(In no way am I crediting myself for this; at best I was a canary in the coal mine - but I take my solace in schadenfreude.)

Whatever decision you made they yelled at you. I got yelled at once for not lying to a government official. When I quit, every sr. developer quit within 3 months of me; 7 total left and left behind 6 non sr. devs. This was 3 years ago. The company is hemorrhaging money but just keeps issuing more stock to stay afloat. It used the money from that stock to buy its competition in a very niche market and essentially just put them out of business. So they survive and lurch on.

/Softwarecompanies/Some companies/g

I wrote a paper for an assignment (maybe ten years ago, ask me for a link, and I'll see if I still have it), in which I called this phenomenon Corporate Inertia, and it isn't only applicable to software companies.

Sometimes the inertia is good - it allows the company to weather bad times.

Other times the inertia is bad, because the company cannot pivot in time to address new challenges or take advantage of new opportunities.

Mostly, the good outweighs the bad - if a company cannot pivot in time because it is so large, it merely buys the smaller company that verified the idea already.

+100.

Booking.com and JIRA come to my mind as good examples. Just about everyone complain about Booking, customers, engineers working there, hoteliers etc., Yet it continues on. I hear they've been trying to migrate to a modern tech stack for years but to no avail.

I am reaching 50 years old and am yet to see any ticket management system that is as powerful as Jira, while offering a relatively good usability story, and the integration story of having a ticket bind to documentation on confluence, SCM related history and CI/CD reports.

Likewise the competition to Booking usually fails to show as many properties, have as rich filter selection, and in many cases their prices are higher than what Booking offers, moreso if one happens to have Genie level status.

The irony of jira is the years it takes to realize how few pieces of software can soak into complexity like it can.

The pain of jira is how it largely can’t universally handle complexity out of the box or how you’ll find yourself in it - how it must be actually setup for your use case - how much it benefits from being setup by someone who has experiences with the pros and cons of cocaine one of the many ways of doing something.

The new project type (team?) is a step forward to start with from scratch. But a kiss of death if you need to outgrow it for some features that aren’t there yet.

Unless you're a mega corp - you're not getting a good work management solution.

I ask about ‘points’ in interviews, its an immediate stoppage for me

I asked him why he was so upset. Eventually he admitted that closing tickets in jira was how upper management measured how well our team was doing - and by extension him. I thought that sounded like his problem and still refused to put the issues in jira. So he did it - meticulously creating a couple dozen issues, making up a BS point estimation and closing them all as complete.

Whenever I think about bullshit jobs, I think about those silly cost estimation cards, that guy, and how deeply satisfying it was that day just diving in and making our product actually better.

There is some tiny marginal benefit to what you’re talking about (there’s a chance someone might later want to look at the corresponding issue). But you aren’t acknowledging the cost, measured in my sanity making a bunch of one-line issues in jira, and the team’s time doing cost estimation.

It’s nice to leave breadcrumbs for future code historians. But our job is shipping product. Cost estimation, jira, etc all only have value proportional to how much they help us ship. If the meetings are pointless, and jira isn’t needed then doing that anyway is a waste of time.

- Why the commit was made should be in the commit message

- What the change is inherently lives in the diff

- Why the code is how it is should be in comments adjacent to the relevant code item.

But when they become the subject of endless meetings it's a tell that maybe the company has hired too many people too quickly and is in the "bullshit jobs" phase where it's happy to waste precious salaried hours on meaningless debate.

You're lucky not having been discussing money budget per ticket instead.

I have had projects where sprints got money budgets assigned to them, which divided by the cost per hour of each developer got us the amount of hours available for the sprint.

Those hours were the budget for the sprint, no more no less, better chose wisely the stories that fit in.

To make it even better, some project activities only were given such kind of budgets in specific times of the year, then one of the external consulting partners would have the go at that specific sprint.

So not only were the tickets chosen to fit the budget, every few couple of months, the teams doing the work were always rotating.

I have had your experience with poorly organized PM and horror stories during sprints...

And I had the best planning sessions with a proficient PM, that made picking up tickets a marvellous pleasure.

that describes pretty much every tech company in the past 18 months ;)

First week was rebooting servers at random to try and get things working while I mapped everything out.

Eventually rebuilt everything and put together a good team.

Key was learning to tell management no. And having someone to keep upper management off our backs.

nobody will care if you leave, nobody cared about the last person that left, they have other priorities and software engineering is not one of them.

if its that much of a problem, their business will fail and some other business that does care will take over their market. but that might take decades because software engineering isn't actually core to a lot of business activity. youd be amazed at how bad some software is for major companies, and yet they are very profitable.

you only have control over yourself not other people.

I was in a similar situation once, I spent months telling management that I was the only one keeping them afloat technically and that it was a critical risk to the business. They responded by trying to pressure me into unlimited unpaid travel and overtime.

I left and they fared poorly.

> I was damned near suicidal from fighting for acknowledgement

I sincerely hope this was hyperbole. Please don’t ever allow anything work related to affect you to this extent. You are more than your job. There are other opportunities. There is much more to life than software.

Don't fight management, leave.

Maybe this is true in the general case, but there are so many exceptions to this that I feel it is not true enough.

Customers don't care about about using the best tool, they only care if it is good enough, and a computer/application/software that crashes once a day is actually good enough for most desktop workflows.

An OS that abuses the user might still be good enough that better alternatives cannot get a foothold. Same is true for web applications, enterprise systems, games, etc.

It's only when you get into the embedded space that you face the "quality or GTFO", because people have lower tolerance for malfunctioning appliances than they do for malfunctioning computer systems.

Primary thing that I learned being a consultant - you can leave and you don't have to suffer the disdain from the management. You learn to disconnect yourself and be replaceable.

It makes for a less frustrating life.

Sad that it happened to you. Don't invest yourself in someone else's child.

This reveals the secret goal of documentation: To build that model, not to document every last detail. The code can do every last detail, it's the model that is hard.

Set out with that goal and... yeah, I can't guarantee success for you either. It's a fundamentally challenging task. But at least you'll be off to a better start than if you think the primary purpose of documentation is to document what happens if you set this combination of flags. That has its place too, certainly, but if I can only pick one, I'd take the high-quality high-level overview over the stuff I can read the code for in 30 seconds.

Code documentation, in contrast, should be auto-generated from the code.

But yeah, totally into auto-generated (great example is dbt’s docs command: https://docs.getdbt.com/docs/collaborate/documentation)

But like you say it is challenging, even finding good names is hard. But basically, can you write a story about the software working, that would allow readers to understand it? Create user-stories but not (only) about the users but understanding that any module is a "user" of its components.

A junior dev who comes in without much coaching, might just reference an anti-pattern from the "mess" section and replicate it throughout the code base. And then do that enough times, and a senior dev who gets brought in might conclude that there's no design at all, and that what it really needs is a rewrite.

Maybe a good metaphor is to view the code as a simulation of something, and the documentation as the description of what that something is and how it behaves.

For instance say the software implements matrix multiplication. It is easy to document if its "units" correspond to the well-known concepts from the theory of linear algebra. But if the code implemented matrix multiplication but used random names for its modules classes and methods it would be difficult to explain what the code does.

Any explanation must assume that whoever reads it has a basic understanding of the basic concepts the software calculates and manipulates. But as said it is challenging to come up with such a theory and that must be done when the code is written, so the code itself reflects that "theory".

Documentation can describe a system state, and how to use or approach it, but in my experience it really lacks in capturing the full depth of experience when it comes to interacting with it. All the little shortcuts and workarounds. All the debugging steps. Seeing something, and realising you've seen it before and solving it in 5 minutes instead of the entire day it took you last time. I could write a story about that day, but it wouldn't do anyone any good - they won't learn from that experience like I did and it won't spring to mind instantly. Assuming they even know where to find a piece written about their specific problem.

So when someone says "How do I X?" all I can think about is all the incidental things I've learned when trying to do something else. This builds the bigger picture. I can answer your question, but you'll just paste the code in and be back to ask more tomorrow. I valued the coworkers who treated such things as more of a discussion and learning opportunity than a transactional question-answer routine.

This way you can incrementally automate various pieces of it so that gradually the user is doing less and less, but there's continuity on their end: run the script and do whatever it says to do next.

Although the example uses Python, rather than shell.

A tool like Jupyter Notebooks would be great for this - comments & code can be placed into a single document. I'm working on a project called unSkript (https://unskript.com) that is built on this idea.

This is the part that most people conveniently forget. Many people find it easier to create elaborate docs and processes than to regularly interact with the software system they've built.

I believe this is partially due to lack of training (technical writing rarely gets trained, and companies rarely invest in it as a skill), and sometimes probably also due to lack of exposure to high-quality documentation (which serve as role models you’d then want to live up to in your own documentation).

In addition, companies often do not reward expending time and effort on writing documentation, because it isn’t crucial in the short term. It’s similar to tech debt, just documentation debt.

While writing good documentation isn’t easy, the fact that it usually gets neglected is a cultural issue.

People "on the spectrum" are not great communicators, and also struggle to understand what it is other people know and don't know.

Also, documenting is genuinely hard, even if you don't consider that it changes all the time. Which it typically does.

Not that I disagree -- working in highly technical fields this long, I could categorize 80% of my co-workers as lacking typical social and communication skills and you never know the real reason. It's to the point where I see women get promoted to management roles more often because they're superior communicators.

The tests tell you what it's supposed to do, not how it does it.

Note that requirements change, and bad tests will break on no requirements. Telling the difference is an exercise for the reader.

1. Ask for forgiveness and do whatever you think is prudent regardless of org structure.

2. Find a better org structure elsewhere :)

They are when devs put all the code into those header files.

waterfall by any other name smells just as bad.

I think the reason is that senior management has no trust in the developers, which is the ultimate cause of the need to have waterfall. Each reporting level wants to show concrete evidence that those under them are "doing their best", rather than to have trust that they are.

Of course, it is understandable that people want concrete "evidence". But waterfall is the result, and you end up with big-bang designs, RFCs that are out of date because a change in circumstances necessitate a change that is not documented in an RFC.

It is a management problem, not a software development problem, imho.

This means if you want to buy a struggling or insolvent company it will most likely have bad documentation and tests.

But one of the most common steps when buying a struggling company is firing people (making the company fire people before buying it) and the selection of who goes and stays is rarely a good one. And it sets signals which make it mote likely for further important people to quit.

Usually, when I see tech companies purchased, it's because the company wasn't struggling: it was doing quite well, but was small, so a much larger company bought it up in an acquisition to add its capabilities to their own, like the Borg. The big company gets IP, customers, revenue, and an already-working business, while the founders get a big payout.

- A defined and documented process

- Project planning or organization changes

- Staffing, purchasing, or contracting

These are fine tools for a lot of business problems, but they are rarely helpful, and often harmful, for software development.

Just because it seemed obvious what you were doing and how things worked while you were writing the code doesn't mean it is, or that it will stay that way. Especially if that code ends up getting abstracted out. In line comments demystify code for the next guy way better than any amount of well written documentation.

As for what they're making you do with the RFC documents for changes, try writing your stuff as an ordered multi-tier list. The first tier should be the "manager" description of the change. The next level should be the more technical description of changes. The third tier is the location in the code where that change needs to be made (like a file path sort of thing, but you could use name spaces and even method names if you want). The forth and final tier if you need it would be cross references to other related items in the list and implementation notes/challenges specific to the file path. I've used this approach in the past myself. It's much faster, and it allows you to write a spec doc that isn't a huge waste of your time because it allows you to half write your code at the same time. It also provides a sort of semantic map from how the manager thinks about things to how you think about them, making it much easier for a manager to discuss it with you without having to write nearly as much.

Another benefit to this is you can often use stuff from the second and forth tiers to add in line code comments.

I'd suggest just using the documentation. Manuals get used for doing testing and implementation. Using the documentation is like testing for it, and finding those problems right away is the best way I've found to ensuring that the documentation is correct and up to date.

Documenting a terrible project is an excellent way to propose and volunteer yourself into a new project of your own design and ultimately get promoted into lead/architect/SME types of roles. I don't know anyone who has done it any other way.

That whole process sucks and I’m sorry you have to go through it. It’s probably the best solution of the time. My one question is… Isn’t that why we have github issues?

Write your RFCs as markdown in a git repository. Stick it to the man!

> The first is that it’s a partial explanation for bitrot, the phenomenon where working software seems to deteriorate even when the code is untouched.

I worked at an organization that had gone years without any updates: 10 year old dependencies that had never been updated, actively developed Python 2 code base in 2022, servers with 1000+ day uptimes and hadn't been updated since before the pandemic, on and on.

It made it difficult to get anything done because you had to wade through a dumpster fire to make the smallest change.

It certainly felt like bit rot to me.

> the phenomenon where working software seems to deteriorate even when the code is untouched

If the code is touched, and fails to keep up with dependencies, I might argue that's more a case of bit neglect. But maybe that's semantics.

I do agree that bitrot can be real though. Links rot off the web. If you're going to leave a project sitting on git without maintenance, you should probably capture the build context. Like shrink-wrapping a boat and storing it on land rather than just leaving it rusting away in the water for years.

It’s something that just breaks down without any changes in use case. If a wooden bridge rots away, nobody means that they were unable to add extra lanes. They mean it ceased to do what it previously did.

“Code calcification” is likely a better approximation of what you’re trying to convey.

“Bit rot” would be better fitting to actual storage loss, but this industry chose poorly.

https://en.wikipedia.org/wiki/Subsidence

At least thrice I joined teams where all original developers and the manager was long gone. The situation for some reason looked very similar in all cases. The team is complaining about supposedly bad design that is preventing them from getting work done. But the system is also large and complex and they are unable to fix/rewrite it.

Lo and behold, in two out of three cases I found that the original developers actually did pretty good job designing the system but in all cases any one of new developers understands only small part of the design and is unable to contribute changes across the entire system, there is a core of the system that nobody understands even though everybody uses and the team has made changes that are against original ideas of the system quickly creating more problems and exploding complexity.

In all cases developers are only able to comfortably do some kinds of changes to the system that are routinely done (like adding new fields to forms, messages, etc.) that do not change how the system functions. Changing the basic structure of the system is a taboo.

In essence, developers treat "legacy" code as if it was some kind of framework or language runtime -- to be used but never to be studied to have its internals understood. (I will never understand this approach to development)

Making "non-standard" changes (yes, one team used this exact term!) pretty much always ends up with something resembling long archeological expedition with ancient traps, diseases and wild animals.

I was baffled by this as well for 15+ years until just last year I had a key insight: most developers don’t like to read. In fact, most people don’t like to read. Most people are intrinsically lazy and since growing your skills involves hard work, they don’t do it unless pushed to do so, either by life circumstance or necessity.

Time and time again people have asked me: “how do I get really good at $tech or software engineering?” I recommend a few books or send them my standard list of must-reads. In 999/1000 cases the person approaching me gives me that defeated look that I know means: “Oh. I was hoping you’d say something not involving working hard.” Most people are polite enough not to say it to my face.

When folks have problems with HTTP I ask them if they’ve read RFC 2616, and if not, I offer to bring them a nicely printed and bound copy. It typically goes unread though some use it as a reference. Only once did someone follow my recommendation and read it cover to cover.

Folks seems to lack a kind of basic economic perspective. Spend a week, month, something along those lines growing your skills (or even just 30 minutes reading the docs on that thing you just spent 2 hours declaring idiotic). Yes, this means probably giving up your favorite Netflix binge, video game, or drinking on weeknights in favor of study. But once these skills are internalized they pay back for a lifetime with good returns. Yet for some strange reason people prefer to keep thinking short term and so they never learn what they’re doing. It used to make me sad but I’m making peace with it.

I knew a developer who said of such people “they spent six months in the lab to avoid two weeks in the library.”

I think at least to some degree people sort themselves into projects. Typically, the better ones that are also happy and capable with reading other people's code command ability to place themselves into projects that seem like more intellectually rewarding -- which means they are much more likely to form the original team for a new project than to join a project that is already legacy and is not meant to be rewritten.

What strikes me is that it actually is easier to write new code than to properly maintain/refactor old, foreign code. Working with large amount of old code requires a lot of hard, real planning and decisionmaking and learning to live with and adjust to decisions made by others in the past.

This hits "maintenance teams" twice -- not only they face actually more difficult task but they are also less likely to get good devs to help. It does not help that management is usually less likely to give resources to teams in maintenance mode than to teams that are developing new toys.

Right on the money. Everyone loves greenfield because you get to build things up from scratch, try out your own ideas, and there's this feeling of tremendous velocity. You feel like you're just tearing through code, kicking butt, delivering wins. Then the difficult requests start to come in, which usually means feedback from real human users. That's when the rubber hits the road and the grumbling begins.

And like you said, once the project is seen as maintenance it is 'not sexy' despite the fact that all successful software definitionally makes the most money while in the so-called maintenance stage.

But I must admit as I get older, I don’t have as much time or patience to devote to this and I’m getting to terms with getting good enough to make a living rather than really understand $new_technology. I think this is partly from getting old and not wanting to sink so much of my own time and mental energy on computer stuff, but also from realizing how little it matters to most PHBs/middle managers who basically hold your fate in their hands. I’ve become much more cynical after a decade in the industry and I’ve realized how much of my own energy and health I have wasted in trying to get a deeper understanding of programming concepts. It was personally satisfying for a while, but it doesn’t pay off for most people. Most orgs want it done fast and cheap, they don’t want squeaky wheels, PHBs think in quarters and only want the thing to not blow up until after they get promoted in a year or two.

Developers who deliver that kind of low cost code fast (albeit low quality) have much more stable careers, better promotions, and get bonuses.

The rest of us ruin our mental health, paint targets on our backs, and don’t get as far. But man do we know our shit.

What I've found is that good companies and outfits do exist out there, but usually they can't pay as well. Or they are boutique consultancies where everybody really cares about the craft...but then you have the realities of consulting to deal with (i.e., dealing with really difficult people).

I am not sure what to say other than: please don't give up hope. Things can get better but you may have to change your expectations or value system to find a place that works for you. Or if you have the stomach for it, try to build a product on your own leveraging your expertise.

For myself, a difficult but necessary change has been not wrapping so much of my self-identity up in 'being a great programmer.' Yes, I know some things in tech, and that's cool, but there is more to life than technology. But just to re-emphasize: I hear you, and it's tough. Wish you the best.

I agree it’s sad but I also have to look out for myself. There is no point in exhausting your energy trying to fix something where no one wants it fixed at best or is actively fighting you to fix it. It’s not even your money that’s being set on fire then, but it is your sanity.

Curious if you’ve found either a better company, or created your own product? It’s difficult to come to terms with mediocrity and needless toil, I went into this industry because I like well designed and efficient systems.

But I don't think I've read any books on that. What I do is read the code!

I think can do this mostly because I'm interested in it. SW projects die because the code isn't maintained properly and becomes unworkable. So I work on being the counterforce to that, keeping the project alive.

Most engineers are into writing new shiny code, and hate "cleaning up other's mistakes". That's also often rewarded. The results are well known :)

You’re right that there are very few books available showing people how to do this. Usually “Working Effectively with Legacy Code” is recommended but it’s just one book and so can’t cover everything. It’s an area where on the job training and experience definitely makes a difference.

Rather than follow suit, he went in the opposite direction and did what no one else wanted to do… he started a septic tank cleaning company. He said that it was hard work, but lucrative… and that after a while, he woke up one morning and realized that he was passionate about other people’s shit.

AI and crypto are very cool right now. They'll change the world and make some people billions.

But I bet you'll on average have a better career working on sewage plant management software :)

Well, that sounds like an utter waste of time. I just wanted to know if this endpoint responded to a GET or to a POST request.

EDIT: to clarify, when I say “problems with HTTP” I mean actually understanding the protocol. Many people for example are unaware of the power of the Cache-Control header and all the things it can do. I used to ask people in interviews what the Content-Type header is for and enough senior engineers didn’t know that I simply stopped asking not to make them feel bad. It’s truly shocking how many people will go decades operating on muscle memory, never bothering to understand what’s actually happening when their code runs.

Thankfully basically everything is utf8 these days.

(I recently looked into getting a copy of a certain specification printed/bound, but found that the cost would be a lot higher than I cared to pay)

Or, if you’re into arts and crafts, you can get good home binding machines that are not very expensive. It’s actually pretty fun IMHO to punch and bind the paper but YMMV. :)

My reading list at any point is tied to whatever I am doing in my private or professional life. If my boss asks me to help grow the company my reading list will contain a bunch of stuff about company growth. If I have money to invest but no knowledge of investing -- you will see me listening to investing books while cooking, taking walks, commuting or long distance driving.

Basically, whenever I face a problem I look around for literature to help me with it.

I really hope to have an original problem in my life. Until then I assume that whatever problem I have there is somebody who wrote a solution to it and if I can't find it it is just because I wasn't searching hard enough.

People / Soft Skills:

* How to win friends and Influence People - Dale Carnegie

* Secrets of Consulting - Gerald Weinberg

* Peopleware - Tom DeMarco & Timothy Lister

* Nurtured by Love - Shinici Suzuki

* Turn the Ship Around! - David Marquet

* Never Split the Difference - Chris Voss

http://dhemery.com/articles/resistance_as_a_resource/

http://dhemery.com/articles/managing_yourself_through_change...

Macro Skills:

* Toyota Production System - Taiichii Ohno

* Workplace Management - Taiichii Ohno

* Thinking in Systems - Donella Meadows

* Fifth Discipline - Peter Senge

Agile:

* Extreme Programming, Explained (2nd ed.) - Kent Beck (the annotated bibliography is a goldmine)

* Chaos: Making a New Science - James Gleick

Programming:

* Implementation Patterns - Kent Beck

* Test-Driven Development, By Example - Kent Beck

* Refactoring (both editions) - Martin Fowler

* Refactoring to Patterns - Joshua Kerievsky

* Practical Object Oriented Design - Sandi Metz

* Functional Programming in Scala - Chiusano & Bjarnason

* Growing Object-Oriented Software, Guided By Tests - Freeman & Pryce ("The Goose Book")

* Modern Java in Action - Raoul-Gabriel Urma

* Java Concurrency in Practice - Brian Goetz

* Grokking Algorithms - Aditya Y. Bhargava

Architecture:

* Righting Software - Juval Lowy

* Release It! - Michael Nygard

* Site Reliability Engineering - Beyer, et al. ("The SRE Book”)

* Domain Driven Design - Eric Evans

* Introduction to Permaculture - Bill Mollison and Mia Slay

* Fun with a Pencil - Andrew Loomis

Design:

* All books by Edward Tufte, especially “The visual display of quantitative information”

* The Elements of Typographic Style - Robert Bringhurst

* Understanding Comics - Scott McCloud

* Design of Everyday Things - Don Norman

* Notes on the Synthesis of Form - Christopher Alexander

* The Timeless Way of Building - Christopher Alexander

Also, the realization that people skills are actually no less important than tech skills in a tech setting.

Yes, and this is hard to accept: they are more important than technical skills. I have worked with many truly brilliant engineers who didn't see their ideas implemented because they didn't have good social/conversation skills. The most common version of this is the engineer who gets miffed/irritated/pissed off if people don't accept their ideas after 5 minutes of explanation. Meanwhile, the hustler whose code totally sucks gets the win because they know how to schmooze the business folks (or literally just tell the best lie).

Hard technical skills are not enough for most people to be successful. There are the outliers with "super programmer skills" (and here I'm thinking Linus Torvalds / John Carmack level status) who are just so indisputably expert that they should be heard. But even then they may not get the win. You have to know how to talk to others.

So, the way I think about it instead is that I’m just another type of system that can be maintained, improved, evolved. And my communication with others is simply an interface. And, well, any engineer can grasp the idea that easy to use and well thought out interfaces are valuable and worth working on.

I can’t tell you how valuable it’s been at times to understand how packets flow over a network. When ECONNREFUSED first rears its ugly head many developers tremble. Over time you learn it’s just a slog of debugging. Emotional control while programming is invaluable.

Specs are extremely terse and dry, and I never liked reading them. I pushed myself to read a few and now I find myself craving that kind of technical documentation in my work. The WebAuthn spec is pretty interesting and also has some even more interesting drama on GitHub issues.

"I think a good place to start is through the service design lens — “This is Service Design Thinking/Doing,” Jon Kolko’s books, and IDEO’s handbook (all available on Amazon). These are all practical, business-oriented works, and will give you the language and tools you need to interact with service designers or conduct your own explorations.

In the tech side, I still think there’s a lot of value in reading through Booch and friends, even if you don’t go on to use UML in a formal way, and even old, deprecated approaches to modeling (think Shlaer-Mellor) can be valuable as an introduction to more modern (and generally more complicated) approaches to domain modeling.

For more academic and less practical works, look into philosophers of language and communication (Rorty, Habermas, Latour, etc) — if you don’t have a background there, try the “Very Short Introduction” series, which gives a good grounding before moving into further secondary or primary literature. What you get from these writers is a way to think about language, communications, and complex systems, so you’re building a conceptual toolkit. (Analytic philosophy, which is another, “mainstream” branch of philosophy, also has applications in the formal comp-sci route.)

Sociology and STS (Science and Technology Studies) kind of occupies a practical middle-ground between philosophy and design; I’ve often said that when we do service design, we’re really just doing folk sociology. There’s no shortage of textbooks, but I’d recommend looking into academic field guides for ethnographic studies (a fancy term for “going out and talking to people about what they do”) and ethnomethodological studies (a fancy term for “going out and talking to people about what they believe”). While the service design books will give you practical tools for conducting workshops and research programs, sociology books will give you a firm grounding on why those tools are used, and how to theorize effectively on what you uncover. I also like measurement theory, which is a branch of psychology concerned with understanding how to define measures and metrics — great tools for clearly defining data and reporting.

Finally, in addition to all this, consider digging into a corporate finance text or two; the three legs of the stool are product and service design; technical and operational architecture; and finance and sales. If you can read your company’s balance sheet and cash flow statements, you’ll be in a much better position to understand where you need to target investments, what the likely return on investment will be, and how much cash (or debt) is available to make things happen. Start building up a good knowledge base in all those areas, and you’ll have the tools necessary to know the “where, how, and why” to apply the “what” of technology to your organization’s problems. "

I agree with most of what you said, but this seems a bit ironic.

Your suggestions almost exclusively involve large investments of time with little established proof that they are efficient. Do you genuinely believe reading RFC 2616 "cover to cover" is an efficient way of solving the specific problems they came across?

I would wager most developers wishing to be "really good" actually have a concrete desire like a greater paycheque or employer in mind. If that is true, I doubt reading someone's booklist necessarily is their fastest pathway, and their economic perspective is exactly what stops them doing so.

But over time this person will be getting better and better and better and at some point they will be able to do their job on the fraction of their time while still keeping momentum and learning more.

This has been my experience.

I have been spending 3/4 of my working hours learning new things for the past about 17 years after I realised this. The actual work takes a very tiny part of my day.

Software development really is a knowledge sport. Writing code is actually pretty small part of the task -- knowing what to write is the important part. If you are able to "write down the answer" so to speak, without spending much time getting there, you can be 10 or more times productive than everybody else, easily.

I think you missed the spirit of my writeup.

Especially since reading is great, but actively engaging with what you have just rea, d by trying to solve some related small scale problem, is even better !

https://news.ycombinator.com/item?id=30651273

They're not about any technology per se, but mostly about exercises in how to tackle complex systems.

This is key. Most engineers want to build frameworks instead of applications. But leave the frameworks to the framework builders unless you were hired to build a framework.

I can jump into a 10 year old rails code base and be productive right away if they stuck to the framework defaults. But invariably someone read some blog posts about how "rails doesn't scale" and built some rats nest of service objects/decorators/portsAndAdapters wtfs and it takes some serious code spelunking to figure out what's going on (hint - the end result is always creating or updating a database record, or making a single api call).

I would argue microservices also encourage bitrot. You can leave some service running on some ancient libraries for years it doesn't need changes. But if you have a monolith where you constantly keep the dependencies up to date - everything stays up to date.

But this in my experience is truly important -- from the very start plan how a new member of the team is going to get really proficient at the system. Ideally, they should be able to immediately recognise/reuse patterns/idioms they already know.

People bitch and moan about boilerplate of Java applications. But when I join the project and I see controllers, models, repositories, entities, services, etc. I am pretty much at home as I know what all those things are doing and how they are connected to each other.

Whenever developers decide to run with their intellectual desires it ends badly.

I try to explain to people that typical business backends applications do not have custom problems. Most of those REST services are just supposed to receive query, run a piece of business logic, translate it into database call and then map response into specified format.

Mostly I’m thinking of more generic services rather than pure ‘business logic’ systems. These are the sorts of problems that Twitter might have in the future. Some of these are problems that Facebook might have gotten to discover in their last big outage.

Many probably noticed that very old, relatively simple machines, cars, tractors, printing presses, etc. can be maintained almost indefinitely. Many businesses do prefer old machines even when there are superior new alternatives and that is because the new versions are just too complex.

To keep maintaining something indefinitely you need to understand the machine enough to know every single piece that needs to be maintained and how to do it properly. Everywhere you need to lube it or clean it or to look from time to time.

Make something complex enough and nobody will have this knowledge. Things start deteriorating. Maybe the data in the database will start accumulating defects, maybe those defects start breaking processes, maybe the processes will not be fixed properly but the errors just get ignored possibly causing other data inconsistencies, and so on.

When you have people who wrote it and understand it fully they can very likely maintain it properly because they understand how to work with it, where to look, how to address problems. Loose them and if the system is complex enough the new crew may not be able to acquire this ability. And then it is just the question of how fast the system is going to be deteriorating.

At the expense of dependencies constantly breaking you, which itself contributes to bit rot by pissing off developers who find workarounds to pin their dependencies and/or switch to work on different products.

But.. that is a dead-end job. There is no career path being the maintainer of a large legacy code-base. It was pretty good if you wanted to rest-and-vest, I could do whatever I wanted. But there were no great features, no demo's, no selling of the work. Even small changes were painful.

Moved to a different team where everything is new and building from scratch and suddenly opportunities open up.

As a fresh developer out of college I had the mindset that frameworks were something you use and they always work and all you need is the documentation. My first job had a lot of "core" code written by another team in another office we would use with exactly 0 documentation. I was trying to get our UI to do something specific but couldn't figure out how to do it by just looking at the functions available, so I ask my manager and he immediately knows what to do, makes 2 small changes (in my defense it was pretty hacky how it was set up imo) and I get what I want.

I remember being moderately pissed because I was already feeling the heat of asking my manager too many simple questions and I just apparently asked another one, so I say something along the lines of "How the hell would I ever have known how to do that, there are no docs available to say that is possible and that is not obvious at all."

He just looks at me and says "You read the core code" and I remember having my mind blown at missing something so obvious for so long. After that I voraciously read almost all "framework" code I could get my hands on, I asked more intelligent questions and my relationship improved my manager and eventually came to be considered a generally good developer.

Once I had this breakthrough I was seeing it in younger developers all the time, and now any time someone asks for general advice on becoming a better developer one of the things I say is "read a lot of code", and more specifically "read the code in your frameworks"

- Was the original team slowly replaced due to cost cutting measures and thusly the people who joined aren't compensated well enough to dive super deep into the system to understand it?

At some point people need to be compensated to care is my theory. Engineers who inherit large systems are either scared or simply don't care, possibly a mixture. But compensate them enough in both salary, time, freedom to understand the code which is clearly critical in maintaining the long term viability of the system.

Thoughts?

If I feel underpaid, I'll do the strict minimum to not get fired and spend my energy looking for another job.

If I feel underpaid, I will do my normal work as best as I can and start looking for another job.

I always try to leave on best terms. You really don't risk anything, you actually improve your possibility of fixing the problem without leaving. And in my case it led to at least couple more opportunities.

Not to mention the world seems small enough and on many occasions I joined the company only to learn I will be working in vicinity of somebody I worked with in the past.

This. This is how I felt trying to understand legacy core code. I kind of like that though. The idea of being able to improve/rewrite a decrepit piece of code excite me. But you won't be productive, and getting a (somewhat) complete and precise understanding of what it does and doesn't, take a lot of extra time.

My currently company only did a specific niche in digital marketing but for sales goals needed to sell outside of that niche. Unsurprisingly, the product didn’t work for those use cases.

and

> Team stability is vital for software development. Each team needs to be composed of a majority of first-generation developers.

How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever...

To take this seriously, internal churn (even if just someone being promoted into management, or specializing on expertise in a small part of the original project, or switching projects when they get bored) is almost as costly as external churn. Five, ten years later how much is their domain knowledge still that relevant to the people working on the code day to day today? It could be harmful if they've been out of the loop and make outdated suggestions, even!

Assuming and desiring your employees will stay in the exact same role indefinitely seems like a huge cause of churn.

What I learned from that project is that being indispensable sucks. That lesson has recurred in my career either for me, or when I got wiser, for other people.

It's not enough to write reasonable code that 'works' or document it as it is now, there's a whole unmeasured and thus under-encouraged aspect around maintaining a legible edit history. I often am trying to get into the head of the bug author to figure out why they thought this was a feature in order to preserve functionality while fixing bugs. They aren't always around, and they aren't always fun to talk to. If you have the history, the person is free to work on other things. To gift them to people they mentor, or otherwise re-home them. If you don't have that, if it's all in their head, they have to leave to get away from the project.

If I can figure out what you're doing I can figure out how to change one or two conditional blocks to fix a problem. Your code stays intact. If there's no history anyway, and the code is sketchy, I'm far more likely to take the code away from you by rewriting it. We can't have black boxes on the team. That's a ball-hog, and ball-hogs get benched.

If you don't show your work, then you're going to find you have fewer and fewer interesting responsibilities. Because "how we got here" is also a "why" and we ask Why all the time.

I try to have them clean it up but often the best thing I can get is a squash.

It's simple, makes reviewing easier and works for 90% of the cases.

Maybe I'm missing something, but I find it painful to rebase into a neat history, and my colleagues would probably hate me to police them around to do it as well.

So it becomes a fight between the two.

Grow a garden, not a tree. Let's disambiguate the word "project" from the word "system" (not "product" -- that's sales). Let's call a project a time- and scope- limited effort to create or modify some system(s) in your suite (your garden). I think it's a common mistake to mix these two up. If projects just drag on forever, then yes, employees can feel stuck. By this definition, I certainly would not want to work on one project forever.

What about working on one system forever? The problem there is that to stay busy, you're forcing attention on a system that might not need attention. At the end of a project, a system should be fulfilling its purpose, and it should do so indefinitely, forever. That purpose may expire, but the system shouldn't. There should be no urgency to keep messing with a working system.

So let's work in one garden forever. Systems will grow; we'll prune and re-plant them; we'll give the healthy ones the barest maintenance it needs; we'll give the dying ones the sweet release of death; we'll shape and braid them together. All your institutional knowledge about how System A works will be great when you're writing System F that depends on System A, or when we're splitting System A into Systems A1 and A2. I would absolutely love to work at a company where I can work in one garden for the rest of my life. That's my dream job.

Sidebar: do not make the mistake of assuming "system" means "microservice"! That's only one of many ways to define system boundaries.

If you can grow people into being the ones who know how the components of the system work together at a layer above how the code for any given component works, you can have a retention plan and keep institutional knowledge. (Or even then move people into a layer above that, as the system grows, etc).

You still won't retain everyone, some people want to jump between gardens, but you will be able to retain a LOT of people, and it's much better than telling people they have to own all their individual lines of code forever.

Wonderfully stated! Losing sight of this is I think where projects start to run into trouble with scope-creep and the ”fix” vs “enhancement” issue. It also shifts the perspective of what is a satisfactory state for a finished project vis-a-vis maintenance plans, dependencies on aging systems, etc. Your garden metaphor is also beautiful and IME accurate.

A big component of what this article is saying is basically that knowledge transfer in the way you're describing doesn't work, and if you dig down to the theory this article is based on, new devs adapt and create their own theory and solutions was specifically called out as a problem. In case 1 of this paper, it's described as second generation programmers submitting patches to the first generation programmers that don't make use of the structure of the program, and add things that are trivially done by the existing systems. Then, a third, disconnected generation of programmers takes over the maintenance from the second generation, and they've added so much cruft to the system that the original powerful architecture can still be seen, but isn't in use because of all the additions.

[0] https://pages.cs.wisc.edu/~remzi/Naur.pdf

Knowledge transfer doesn't work at all :) It's not a thing that's possible. The best we can do is for someone with knowledge to attempt to explain it, hope they do a good and complete job of it, and then hope that the explainee is able to internalise it and come to the same/similar conceptual understanding that the explainer had in their mind.

It doesn't tend to be done very well at corporations. It's a skill that good teachers have developed after much practice, which isn't something we tend to hire for.

And it requires favourable conditions, and one of those conditions is not "it's my last day tomorrow and the boss asked me to bring you up to speed on what I've been doing these last six months. You only had 30 minutes free in your calendar, so... let's go"

Is that an improvement?

(I'd also argue that I've worked on systems where I've developed a better mental model of the problem than the first-gen people I inherited it from, because I as a maintainer was under less pressure to get anything that worked out the door initially.)

EDIT: and looking at the paper you linked there, https://pages.cs.wisc.edu/~remzi/Naur.pdf , I'm in complete agreement with "the designer's job is not to pass along "the design" but to pass along "the theories" driving the design" - it was always easier as that maintainer to take over a project where people could tell my why not just what.

A lot of the time they don’t want to understand the system. Maintaining some old legacy system is no fun. Programmers want to do new things their own way. And if there’s no one around to guide them then they will.

This is a management and cultural problem. Enterprise software resembles failed open source projects - no one wants to work on them so they die, get forked, or re-written and replaced.

Contrast this with successful open source projects. A vibrant project will integrate new developers into the culture (or show them the door), and conversely developers who want to change things will have to prove themselves.

Personally, I don’t our ability to learn is that bad. Given time to come up to speed and motivation to read the code and understand it, and some good (consistent) coaching from people who fully understand the code, I believe new developers can be integrated into a complex codebase that they haven’t written.

But this really draws a stark relief from the way software engineering on existing code is treated by every manager I’ve ever met, which is likened to reading a book once in a language you already know. One read through is enough to grok it. If we land somewhere closer to this source material on the spectrum I think that’s a win.

I also look at my own thoughts on existing code. I’ve had the strong urge to refactor out old code I didn’t like… and that is interesting to me. Did I not like the style or just not understand it and want to make my own mental model my way?

The idea that we’re just replacing engineers and they are just re-writing the same code with the same outcomes because we think of them as Lego blocks is interesting as well.

If you try to fix the theory knowledge/documentation/practice problems purely through insisting the "first gen" stays where they are, they're going to leave you for more interesting work, and then you're in exactly the situation you wanted to avoid. The only way out is "more effective practice of theory and knowledge transfer" but all the author seems to have to say about that is "This is a team management issue that can only be addressed by a manager who understands the fundamental nature of software development. There is no magic development process that will prevent these problems. It takes skill."

E.g. that's the real hard part yet they completely punt on trying to dig into it.

To me this is the a huge incentive for building strong teams. Nobody should be indispensable; it's a disservice both to the company and the person. Pair programming, collective code ownership, intentional cross-training, good automated tests, continuous deployment, and good automated monitoring have all been very helpful to me there. And vacations! Everybody should take long vacations where they are out of contact. By managerial force if necessary.

Decision makers for American companies tend to use "Time spent at work" as a proxy for "Productivity". That used to work just fine when the job was pulling levers on an assembly line but turns out to not make sense when you are talking about software development.

Do you actually think of yourself as a capitalist though?

Alternatively: just promote and pay your people as they develop, even if they're in the "same role".

The first project is maybe a bad example but if we say the "nth" project instead then:

- not always changing members doesn't mean never changing, companies should just not want you to jump between projects all the time (except maybe as junior programmer) and you shouldn't want this either as it's hard to build up proper deep expertise this way (instead of a lot of shallow expertise)

- if you are still in the same company and the company is healthy then it should be fine if you get asked questions about your previous project for some time after you switched (through not arbitrary, just as an "emergency" thing)

The problem is that the concept of carrier and seniority is seriously messed up in the software industry. Often shallow knowledge about a lot of things is more valued then actual expertise in the things needed. Similar you will find a surprisingly lot of people believing that a senior engineer which doesn't try to get into team lead/people managing positions is doing so because they "surly are incompetent and shouldn't be hired" (IMHO completely rediculus but I have still seen it often enough, just most times in a less directly phrased way). But in the end senior software development skills and team lead skills are distinct skill sets. Implicitly pressuring someone who is good at solving technical problems into a team lead position (or the other way around) is just a pretty stupid decision from a human resource utilization POV. (This depends a bit on the definition of team lead and yes team leads also need to have technical skills but most important they need to have the skill to listen to their team and not get stuck on their opinions, etc.).

When TSHTF and we need some expertise from the original builders or operators of a system, it's a lot nicer to have them on another team than at another company. If they still work for us, somebody can make the case to them (or their boss) that borrowing them for a few hours to fix it is in everyone's interest. A lot of times they are in some kind of management position now and they kind of smile and say "Wow, I haven't thought about that system in a long time. Sure, I can take a look." and it's even kind of a fun diversion from whatever budget meeting they were in.

If they work for somebody else, the answer is often "sorry, I'm busy" or just a hangup when you explain who you are and why you are calling.

Reasonable amounts of churn are manageable if you do a good job of acculturating new people into the team. This has actually been one of my concerns about the switchover to WFH. I know a lot of that acculturation used to happen by osmosis and people working next to each other. I think my team doesn't do as good of a job maintaining it when we're all working separately. We're a lot more efficient now, but I think we haven't figured out a good way to share and transmit context like we used to and I worry what happens if we hit a period of flux in our company. I worry our team culture is a little more fragile now.

I used to think this too, but I think we need to be cautious of magical mechanisms for knowledge transference. I think the real mechanisms are code reviews, discussions of issues+code+project, consults / shared problem solving, design reviews. Just sitting next to someone doesn't work, there needs to be a shared task or problem for knowledge transfer to happen.

I think all of these shared tasks happen with remote workers just as effectively, as long as you have good team comm hygiene.

Talking to your teammate who sits nearby in person is both lower friction and higher bandwidth than sending them a chat method.

"Adhoc conversations" don't magically lead to knowledge transfer, the probability of any adhoc convo being long-term-meaningful is low, but they greatly increase the cumulative probability.

By documenting your work and/or mentoring a junior employee to pass on your tribal knowledge.

This part rings very true, as someone who's been through plenty of rewrites / "make module X not suck" projects. A side observation here is that usually the original author (sometimes long-departed or departing) understands very well the limitations/flaws of the original system/implementation, but can be excluded from the design of the replacement for any number of misguided reasons. Trying not to bruise the ego of an early employee, thinking that they wouldn't agree with any changes, whereas those are often the people in the best position to guide a rewrite, and often would love the chance to fix some of the decisions they had to make originally.

I relate to this so much. I'd offer an open letter to some poor anonymous soul who inherits one of my dumpster fires. Because I'll certainly never be given the chance to revisit anything.

Yep, my shit's broken. It was written long ago, under different constraints, for a different audience. I made the best trade-offs I could. I documented them! Though you'll probably never find my writings, and if you do trip over them, you won't understand them yet. Today, I'd scrap most of it if given the opportunity. But to whomever is now taking on the rewrite, you know that little nugget at the core which perplexes you so? The one that doesn't fit your mental model of any type of programming you've touched in your career? That there is the core value proposition that makes it all possible, and if you throw that out, you've doomed yourself to learning my wisdom the hard way. If you're at all up to the task, then the next few years of your life and the remainder of your days at this company will be spent fixing the same problems by reading the same papers and obtaining the same obscure domain knowledge. No, it can't be abstracted away. Yes, it's fundamental. No, no one else will really understand it either when you try to explain it, because no one else without the accountability you now have will sink enough of their time into building the mental model you've had to construct. Good luck.

My own efforts don't quite earn me "the big bucks" that I read about at companies I'll actually never work for. Just a respectable east coast income that I shouldn't complain too much about. Don't know that I'd be happier anywhere else anyway. The bills get paid while I'm making the world a better place.

On a bad day, I'd say we do it because we're suckers for punishment. It's in our nature.