This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them.

Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development. We have an existing system. This needs to be maintained and extended because it's our core business. We cannot just say "oh it's become too complex. Let's go and spend the next few years creating a new product, so we'll have a multi-year feature freeze".

The only feasible way I see to do anything even close to this is to split up the software into domain-like silos slowly and incrementally. At that point you can start considering rewriting silos as they become too complex/cumbersome/big. However, at that point you're already doing complex, lengthy and generally risky refactoring, and speaking of it being a write from scratch is very obviously not true.

Complexity and technical debt are our problem, and the customer could not care less about how we solve it, or even whether we solve it. As long as we are struggling with an old piece of sh*t, that's good for us: it means the product is selling and the effort to maintain it still makes sense!

Selling without complexity and technical debt would be much better, I think we all agree on this, but it's a receipt whose success I have never personally witnessed.

The amount of times I have advised clients to not use something like Wordpress to just end up having to use a page builder like Divi hurts my soul.

At the end of the day the majority don’t care about things like carrying bloat/legacy code, rigorous testing patterns, using modern tech stacks or typesafe codebases. They simply want you to “make thing go brr”.

I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities.

I guess the main thing from a business perspective is that I’m getting paid but, at the same time, ouch, my feels.

I think part of the problem here is the "market for lemons" thing: The client can't tell the difference between the short-term-cheap-long-term-expensive option and the short-term-expensive-long-term-cheap option, so can't trust that the expensive option will actually save them money in the long term, rather than being a short-term-expensive-long-term-even-more-expensive option.

Consider getting quotes to re-plaster your house. One builder quotes you £2000, another £4000. Now, is the first builder giving you a fair price and the second builder charging you extra because you don't know what a fair price is? Or is the first builder a "cowboy" who's going to do a terrible job you're going to regret, and the second a craftsman who's going to do an excellent job you'll be happy you went with? Maybe you happen to know enough, but that's not where I was when we were renovating our house.

And consider that the client isn't sure they'll even need a website in 3 years; maybe this new venture will crash and burn well before then. Then we'll be doubly glad they didn't invest in a website with the equivalent of a 10-year guarantee.

Until more client gets burned with a cheap option that becomes expensive to maintain they won't care. Many clients (across the whole industry, I don't know your clients which could be very different from average) will not get burned as most problems will never get that complex. However most clients have a lot of small problems that are quick to solve meaning that most developers spend their time on the small minority of clients that have complex problems where this matters.

Persistent refactoring is one of those things where the cost is clear and well known (expensive developer time) while the payoff is big, but very diffuse, very hard to explain and often gets attributed to something else - sometimes even by the developers who did it.

It's not the only thing like that. Plenty of companies skimp on cost centers and fail spectacularly as a result - sometimes without even ever realizing that skimping on the relevant cost centres was the reason why they failed.

If I could know for sure that a piece of software would exist for 50 years in production I could make a better case for refactoring. Likewise, if I knew for sure that a piece of software would get completely replaced in 1 year I would not bother with refactoring.

Never in my career have I been able to accurately predict how long software would live in production. I have had throwaway code live for years and things I thought would last forever get replaced almost instantly.

I agree that it's a waste on throwaway code, throwaway code is common and what ends up being throwaway code is hard to predict, but I don't think it's impossible to get a sense of the likelihood of the code being tossed, and that sense can be used to tweak your refactoring budget.

But if I spend a month refactoring and only get a weeks worth a benefit before the project is scrapped it wasn't worth it. If I get a month back before the project is scrapped it wasn't worth it (I could have been working on something else instead). Unless/until we know how to quantify how much gain will be got back, and how long it will take to get that gain back and so we cannot do a proper economic analysis if it is worth it - but that is what we need.

A response of "but what if my investment decision doesnt pay off??" kind of suggests you might have missed that point.

That happens sometimes when you make investment decisions. You dont know how much you will make back or even if you will make back anything at all.

Some people dont invest their savings because they can't stand the uncertainty of not being able to quantify the payoff but I dont.

Proving the value is the hard problem. It's almost impossible to prove the value of small changes today. Product wants things to do X and they decided the value tradeoff was good, not the engineers who are tasked with making it so.

Or impossible, even.

Thats why I just do it. If somebody wants proof that my years of experience applied in a particular way will provide a quantifiable benefit that is impossible I will just avoid doing that thing.

Or more likely, move somewhere which can appreciate those non quantifiable benefits.

How can you find such a place? This definitely resonates with me.

There's so much value in doing things right, but it's impossible to prove. I can do things that make my team go faster, but there's no way to say that having a X% better tests resulted in us being able to release Y% more features.

I'm not going to pretend it always worked for me, but it certainly got easier once I could demonstrate a wealth of experience.

There's a lot of luck, too.

In terms of refactoring, I have an iron rule that I never ask for permission. If I think it needs doing I do it. I don't justify it to product or ask for their opinion. Technical issues and priorities are on a need to know basis and they don't need to know. All the work gets baked into feature work and bugfixes.

Oh yeah customer cares. If I leave my car in a workshop, I care whether mechanics have one big mess there or not, I am looking for signs of a good routine, clever process, some smart choices they made for themselves. If they did a fix for me but have a mess, I will never go back to them. Most of customers do such choices.

If your solution is the best compromise, it is more likely to be selected.

- product must be purchased because its vendor is somehow connected to the upper management or the investor

- product must be purchased because the person who has chosen the product is at the moment perceived as having some kind of holy cow status and can't be criticized, so people around put that to the extreme

Happening at most of companies everywhere.

Not necessarily. Some otherwise good systems might have one bug in each of those categories.

And, most other poorly designed and poorly executed systems just chug along fine without the customer realising anything.

That's not the norm though and I hope we can all acknowledge that.

All the examples you provided miss the point. Revenues are the point; all the rest is ancillary at best.

In your car, you can tell, in software you cannot.

Add in the fact that most times even cowboy-coded systems are once off and will be replaced (and not maintained) down the line, and you really cannot tell if the system was written as a huge mess just waiting to be exploited, or as a well-crafted and resilient system, because the majority of systems are in-house only and face no adverserial threats other than the employees.

I can tell about bad development habits from using the software it because code smell (and process smell) stinks.

Besides, I was telling not about car repair per se, but about the process around car repairing, which is visible when you visit car workshop and talk to the guys, and is even more visible when you use the software.

Concrete examples:

- installation/upgrade process involving interactive scripts means that somebody thought dumbing down such process for a tiny percent of less-tech users is ok, disregarding the majority who automate everything (and interactive scripts are obviously harder to automate); it also can be a sign that vendor's decision maker (product owner) is disconnected from reality

- poor documentation; a distinct case here is when part of the product is an API and of course "there is" documentation, generated by Swagger - but the crucial part of it are some subtleties of arguments passed which are themselves undocumented, making the whole API simply unusable. The vendor has an excuse "but we have documentation" - heck yes, it's because your employees hate their superiors so they gave them something to f*ck off; and this is given to the customer. Very common practice actually, I can give you a list of vendors who notoriously do that

- painfully slow webapps are a sign of a process smell that no one thinks about the real world operations; sometimes it's not even about scalability, but bad choice/usage of libraries; i need to give here names, in rot13 so it's not googlable: Argfhvgr has a HR product for corporations where every single damn click on any active field in the browser - and you click it a lot because it is fscking calendar - triggers a series of round trips to the server, each taking over 2 seconds. The usability of this product is so braindead at so many levels that I can see a whole bunch of teams failing to stop this sh*t from being released & sold. Long story but a good example that just by using the product you can see how rotten the vendor must be

The question as a developer is do I want to work on such a system? For me, the answer is a resounding no. If I agree to work on such a system, I will essentially be enslaved to the bad design decisions made before I joined the company. It will never be truly satisfying to work on that system. I am speaking from experience.

But that is why companies pay a salary. The established company mindset is not the startup mindset is not the hobby project mindset. It's up to you to figure out what is best for you at any time and that choice may change as life circumstances change.

A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand.

That last part is also the most common reason why large system replacement projects fail, go over buddget, or are not even attempted.

Often the people controlling the money do not have the technical knowledge how much complexity the old system hides. The safe bet is to keep it going.

Yes. It can happen. If you can't do it with a 99.9% succes rate you cannot make it a general business practice.

> A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand.

Beyond my general disagreement is shouldn't need full parity (man our clients would be pissed off if we cut functionality "because it was easier to rebuild a system that was functioning quite well on your end", and they have guns!), I don't think this takes into full account how much can be hidden in years and years of commits and code. We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely". This, in spite of all handwringing about how this would never have happened if people maintained proper documentation, will happen in any legacy application. We're talking about a hundred people working side by side over decades. These things will slip through, or 'best practices' change so much that things that were a good idea then are entirely unknown now.

Even something as non-intrusive as attempting to stranglevine this will take up a lot of time and effort, if it can be done correctly.

Interesting - when thinking of systems that I've personally worked on which I could never imagine being replaced, the first one that came to mind was one with exactly the same problem: a massive cache of SQL stored procedures at the core that nobody seemed to fully understand. This particular company had even implemented stuff like HTTP auth token validation and all access control with SQL scripts without any centralized access control architecture. It was outright terrifying to touch the SQL.

When I started at the company the manager advertised they "really have no upper limit" on how much they compensate developers who stick with them for years.

I guess a big problem with implementing large parts of the application logic with SQL is that it's simply not a tool fit for the purpose. Incrementally replacing, or even reasoning about, logic splintered accross assorted SQL scripts is very difficult. So nobody has the full picture of what are all the things the monolith does.

I think that what often happens, is that the manifestation of an old system crumbling under the weight of its complexity is that a "new kid on the block" (startup, etc.) eventually takes over. As long as the new thing only forgoes the old cruft very few care about and add stuff many care about, there's a possible takeover.

This seems to be the a key to the issue and it's a people problem.

You snipped:

> [...] a lot harder than it sounds – more on that in a different post.

I don't see a newer post in that blog as of now (May 29, 2024).

The entire idea of "well let's rebuild this before it ever gets bad". That only works in the one tiny niche corner of greenfield development that nearly no one works in. To say "oh yeah just don't ever let software get complex" is like saying "well have you tried excising any oncogenic mutations before they cause cancer"?

I’ve worked on well designed systems that stayed well designed. For example one I remember that was very solid: How? The team had enough support and autonomy from management to:

1) allow us to hire the right people (no secret sauce - just a mix of not too fast growth and a healthy mix of seniors and juniors)

2) not rush new features, and spend enough time on design and tech debt.

It probably helped that the system was a core e-commerce platform. If it descended into chaos it could have had a material negative impact on the top line.

When reading articles like this positing “laws” remember many people live in a bubble and may have only worked in dysfunctional orgs. There are tons of counterfactuals out there when the system and complexity was critical to the business. (And also tons of examples where despite that it still descended into chaos- but that isn’t a law)

In my experience I think the larger any team gets the more likely the team is to hire "regular" engineers and the quality regresses to the mean. So it's only a small set of specialized teams with unusual luck and control over hiring that can become "exceptional".

If 99% of projects end up in a way that's not a consequence of a bad engineering culture.

Even then though I'm not sure it'll ever really improve. Even beyond the human element I feel like the incremental building approach cannot and will not lend itself to ordered systems. Unless we go back to the ecosystem of objects that Kay talked about back in the 80s we'll always create frankenmonsters, and even if we do: we just create another kind of frankenmonster.

My frustration is that there is economic incentive, but it's diffuse and difficult to measure. And in today's culture across a broad swath of contexts, benefits that can't be measured are benefits that aren't valued.

This isn't unique to software development, but it is something that seems to be plaguing American thinking as a whole these days. Public education is extremely valuable, but it's a large cost and increasing numbers of people are skeptical of the benefits. Fixing potholes is expensive but it's surely less than the diffuse cost of fixing flat tires and broken axles. Public transit is expensive but the reduction in traffic and QOL benefits are large and widespread. Money spent on at-risk youth typically pays for itself many times over, but years later and in a difficult-to correlate reduction in policing and prison needs.

More and more we narrow our focus on short-term expenses at our of long-term detriment and it's becoming more and more apparent as a long, slow-moving rot growing from within.

I also don't agree complexity is necessarily a moat. Sometimes it drives customers away to simpler, less expensive solutions; it depends where the complexity is as to whether it creates real lock-in or just friction and annoyance.

= Software consultants and free lancers. I don't like them because I am jealous that they get to do greenfield projects and make all the architectural decisions but never actually have to maintain their own creations (and therefore never really learn how to design a system for changeability and maintainability).

> What can we do about this state of affairs?

1st: be aware of these laws and that keeping a 20 year old system in production is a non-trivial task and to be proud of every additional year I can keep it running.

2nd: Instead of seeing it as a 'DoS attack on myself by previous developers', I try to see it as a neverending Sudoku that challenges and stimulates and pays me every day.

This is how it is for me, I've had a great time with several business systems initiated by amateurs or mediocre developers that took hold and had been going for 5-10 years when I arrived. Yes, the WTF:s are everywhere, changing these systems is really hard and takes a lot of discipline and care, but every time you put something in production you'll also have made other parts of the application better, more robust and faster.

To me that is really rewarding, and not even noisy, butterfingered management can stop you from littering the commit log with lovely little improvements that incrementally turns the turd into a much more stable income for them, and possibly yourself (and if not you jump jobs after a while and get the raise then). I don't care whether they appreciate my work, first I care about practicing my craft well, second I care about how it is perceived by my peers in the craft.

Of course, this isn't a reason to make crappy software on purpose, but this kind of just happens when you on the cheap test an idea against a market and it flies really well and customers flow in.

Agreed. Many years ago I spent two years on a greenfield project that got slowly but surely worse, which was very frustrating. Now I'm working on something that's seven years old and has been through a lot of hands, but every small improvement feels good, and when I need to do the occasional hack, I don't feel bad because on the whole it's better than it used to be.

If I can extend it a bit: sometimes when you start a new job, the board state you inherit is actually subtly invalid (i.e. it will lead to an unsolvable solution), and you have to figure out the smallest number of numbers to "erase" to end up with a valid board state.

You get to realistically use the eraser maybe once per year, because nobody likes erasers. So you just compensate for it with lots of notes!

It's kind of entropy: I wouldn't say that a badly designed system is "stable", it is just as unstable as a well-designed system and keep evolving, but there are many more ways to be "badly designed" than to be "well designed" so without actively fighting entropy, it'll be unstable but consistently "bad"

At TableCheck we have a Ruby monolith app we've been working on for 11 years, and TBH it feels better/cleaner today that it did 8 years ago. Granted we haven't had a lot of personnel turnover and spend a lot of time on maintenance tasks.

There are. But also in software, the article says that ordered states are "easy to change" and disordered ones "difficult to modify". So the disordered states have more "friction" i.e. from there it is harder to to safely move to an ordered state. But possible to step further into disorder and yet more friction.

I have observed this in cases where past a certain point people give up on refactoring and just add code to workaround - they just wrap the code that does most of what they want in a special case for their specific needs. Which in turn makes it harder to understand and refactor. After a while, it's a big pile of nested special cases and workarounds.

Fighting that tendency is harder. It is going uphill, against the path of least resistance.

It's not possible to safely move further into disorder. It's not even all that safe to stay without change.

I agree that there's more friction, though...

> A system that inevitably degrades (to a significant degree) over time was badly architected. Most systems aren't badly architected but most systems that *succeed* are. Good architecture requires time (usually an unprofitable amount) & most successful systems prioritise velocity.

In summary: the demands of profit necessitate bad system design.

Of the two examples I've seen in my career: one was by a larger-than-needed/accidentally over-resourced team in a very large corp, who were operating on an undervalued background product with no real OKRs. The second was a government-funded contract (these often skew badly in the opposite direction due to corrupt tender processes but barring that there's often breathing room for quality if there's a will).

Complexity is aided by having sturdy mental models - what you intuitively understand. And to see the truth clearly.

To have humility: my code is probably ugly to other people, but I understand my code faster than reading yours.

I can't pretend to know that the system I build would be better than yours.

Be wary of pride.

It's a good habit in and on itself, but there still are objective heuristics to evaluate software quality.

Simplicity for example: if to solve the exact same problem, with no (impacting) performance penalty, one solution is considerably simpler (meaning, more straightforward) than another, then there's a clear winner.

The amount of automated tests is another objective way of gauging software quality.

I think I am contradictory when it comes to software : I don't enjoy maintaining something that breaks all the time: dependencies, system upgrades, deployment scripts and things that aren't 100% working reliably every time.

So my ideal system is to run a simple binary against a file or SQLite database and have it work reliably everytime. Not a complicated micro service architecture with lots of indirection and keep things running on network.

But balancing this with my hobby of designing multithreaded software.

I have a friend like that: loves pure functional programming languages; day job? jack of all trades doing buses communication systems ¯\_(ツ)_/¯

Not so clear: some people say `foreach` on arrays is simpler than single line map/reduce/comprehensions. Others say the opposite.

Some people say a monolith is simpler than microservices. Others say the opposite.

Some people say Python is simpler. Others say Java is simpler.

I mean, an earlier response to an earlier comment of mine on a different story was from someone who thinks ORMs are simpler than SQL. I think the opposite.

Until we can get everyone to agree on what 'simple' means, it's not that useful a metric.

What I had in mind was things like removing 20/30 lines of literally useless code, or avoiding sophisticated OOP patterns when a few bare functions on ``struct``s are sufficient. I've saw both cases in practice (eh, I'm guilty of it myself as well): they're often the result of overthinking, or "trying to make code reusable."

For the micro-service vs. monolith, I don't think they are comparable as-is, with respect to complexity. Once the factual requirements are known, and once the usage of each pattern to fulfill those requirements is known, then we can compare. But before, it's kind of a "C++ vs. Rust" type of debate: what is the real situation?

Regarding ORMs vs. SQL, I tend to agree with you: we often can't pretend ORMs are perfect ("non-leaky abstraction") black box: in some use-cases, it's true, but as soon as you shy away from toys, you tend to have to understand both how SQL and the ORM work. Way more work than just dealing with SQL.

Same goes for dependencies in general. But they are also essentially mandatory dependencies (e.g. a TCP stack, disk driver, an HTTP lib, etc.).

I disagree with this, it is certainly possible to improve the state of some system without starting from scratch.

For example, once you have a good grasp of the codebase, and an overview of the future requirements, you can perform low-risks, local refactorings, so as to ease implementing both current and later features/bugfixes/etc.

The requirements aren't systematic though. Meaning, as a dev, you're not always, at least explicitly, allowed to get a bird-eye view, nor to act on it.

In more normal words - the codebase will fight your changes. And that means that small incremental changes may not be enough, and you will need at least a few big leaps.

- https://web.archive.org/web/20240529081236/https://maheshba....

"It only takes a match to burn down a forest"

"One bad apple spoils the bunch"

Keeping a system well designed requires good engineering culture (as other people have said), and a great alignment across engineering, management, sales... the whole company.

In control systems, you typically handle these situations with some sort of feedback mechanism. Said another way: you need to "feel" the problem if you want to be able to adjust quickly enough.

It'd be interesting to know if this kind of "feedback loop" exists for human societies / companies. Or it is just what it is: it exists, but it has a very large delay, which causes the system to eventually collapse and restart...

This one is cool to me because (as I understand it) a healthy forest experiences multiple natural minor burn events that clear out underbrush. The resulting ash help regrowth by acting as a fertilizer.

However, when people with bad policy get involved, then these minor burn events are prevented until the underbrush accumulates to such an extent that a single match can burn down the entire forest.

The analogy to refactoring, bad development methodology, and project failure really speaks for itself.

This subject always reminds me of something someone said, possibly Professor Alain April, "software is the only system where maintenance tends to degrade it."

Someone hasn't met my shoddy electrician/handyman

To me this is the axis of "generic" vs "specialized". As systems get increasingly optimized (for anything: resource usage, bandwidth, latency, cost, etc.) they increasingly need to make choices that limit other potential choices.

This has nothing to do with being badly or well designed. A well designed system is one that does it job well, which in part includes being able to evolve it over time (or not). This might includes staying more generic, or becoming more specialized.

6 years after left the role, I got a call from a sysadmin. He told me the machine that hosted the system went down but they managed to recover the HDD and restore it into a VM. I was surprised because I thought they had decommissioned it long ago, when the vultures who wanted to pad their KPIs tried to write a better version. He said multiple efforts all failed and no one was able to replicate what that system does. The reason he gave me a call is because after they loaded the HDD into a VM and turned everything back on, the system came back online and was working perfectly without any effort on his part. He was astounded and just wanted to talk to the person who was able to build a system that literally wouldn't die.

It wasn't the most elegant or best designed system, but I designed it not to die and that's exactly what I got!

This is often what I see junior programmers do. The author has some experience, much in research, and I’m curious what they practice here and how effective it has been.

I don’t see poor design as inevitable myself. It can be hard to battle against the tide of developers with good intentions and wrong incentives. All too often we are pushed to accept fast and cheap over good and well done.

Update: spelling/grammar.

>Let’s say that a system X is well-designed. Someone comes along and changes it – by definition, quickly and easily – to a different system X’. Now X’ either continues to be well-designed; in which case it can be quickly and easily modified again to a different system X’’; or it will enter a badly designed state and hence be difficult to modify.

???

> Software has a Peter Principle. If a piece of code is comprehensible, someone will extend it, so they can apply it to their own problem. If it’s incomprehensible, they’ll write their own code instead. Code tends to be extended to its level of incomprehensibility.

Personally, I have never thought about that.

It's possible that they are smarter, and use mathematical language as a matter of course in their thinking and speech.

Not only is the author correct about what he said, he expressed it just fine. get off your high horse.

Initially it only ran on X86 PCs and most of the hardware it supports today didn't exist. Today it runs on many architectures from small embededded systems to super computers.

Some of the initial code was pretty rough but the modern stuff is pretty good and cleanups and refactors are done all the time. Indeed long term maintainability is one of the major considerations for maintainers deciding to accept or reject patches.

Why does this work for Linux? Because companies are used for funding (the majority of kernel developers now do it as part of their paid jobs) without giving them technical control of quality. The companies paying people to work on the kernel get a say in what is worked on (presumably things important to them) but not about how it is done.

The classification of "complexity" is a function of the observer, not the observee.

This is good. Rebirth brings vibrancy and you cannot have birth if you don't have death.

Memento mori, the inevitable entropy, constant decay, and disintegration. But also, vive ut vivas, live that you may live - there is an opposite force, often called negative entropy or "negentropy", but I prefer to think of as creativity. It's what organizes things, flowers, designs, buildings, systems.

Also, the second law: "complexity is a moat", seems to be contradicted by "worse is better".

> Rate limit for this resource has been exceeded

…

nods thoughtfully

[0] https://youtu.be/s-CTkbHnpNQ?si=KYwllK4NJY1bjRa3

Such refactoring is an essential part of maintaining software, regardless of how old or bloated it may be.

Back in my day it was via McConnell and the PragProg guys. Jeff Atwood covered it in a blog post from over 15 years ago:

https://blog.codinghorror.com/tending-your-software-garden/

As soon as you are ready to give up the income for those things you want to trim, please go on!

This rings false for established systems that are locked to their APIs.

The vendor of a large, complex set of APIs that large, complex applications depend on is not going to give the application users everything in a new API for fear they will migrate to another set of APIs.

If we are to survive, we need to slow down and instead of making "more" make "better". Follow Unix philosophy. Embrace "good enough" and learn to stop.

Who am I kidding, we're doomed.

Almost all programming jobs will be on too complex systems. If a system is well designed and not too complex, then they wont need to hire you!

>> Essential Complexity is inherent in, and the essence of, the problem (as seen by the users).

>> Accidental Complexity is all the rest — complexity with which the development team would not have to deal in the ideal world (e.g. complexity arising from performance issues and from suboptimal language and infrastructure).

>> Note that the definition of essential is deliberately more strict than common usage. Specifically when we use the term essential we will mean strictly essential to the users’ problem (as opposed to — perhaps — essential to some specific, implemented, system, or even — essential to software in general).

The best skill I've learned, and continued to practice and improve, is the ability to strip how we talk about problems we want to solve with software down to what's truly essential to the problem. Making a habit of doing so helps clarify the contours of the problem itself, and improves discussions around solving because the boundaries of what's truly essential become clear—and then everyone involved knows that every choice we make from that point is additional, accidental complexity we are adding to the problem ourselves.

Far too often I have seen even greenfield software quickly ratchet up the overall complexity because the people making choices don't take the time to really isolate the problem from the software—but instead frame the problem within the context of languages, frameworks, architecture, infrastructure, and so on, and then just start slinging code at the problem.

If you haven't yet read Into the Tar Pit, it truly changed the way I look at and think about software and problem complexity. You may find value in it, as well.

[0]: https://curtclifton.net/papers/MoseleyMarks06a.pdf

Now what do we do about it? What I'm doing about it is

http:/github.com/civboot/civboot and http:/github.com/civboot/civlua

(both use Unlicense)

When I boil down the task to its nature as a data transformation, the solution flows from my understanding of the problem, and I’ve found that my choice of tools flows transitively from there pretty easily. The problem is “isolated” from the software as you said which makes it so much easier to reason about things.

I sadly have not gotten much traction when I try and advocate for this mindset in our industry.

As an aside: It reminds me of a funny point from secondary education. Did you take AP tests in high school? If you did, you might remember as I do a consistent refrain that teachers used to beat into students preparing for the tests: “Answer the question” Over and over we heard this ad nauseam until it became second nature, whether for AP English or AP Physics - and it was good advice! Because the number one mistake students make on those exams is not actually answering the question asked, which even when couched in the most wonderful prose, results in a failing mark.

I think software engineering is often pretty similar. Even the best, most sophisticated tools will not produce a working solution if you don’t understand the problem.

Yeah, I know what you mean. It's become a bit of a primary signal for how I evaluate a company's engineering culture. I've been lucky to work with some fantastic people who really get it, and I've also struggled and suffered through working with those who do not.

> Even the best, most sophisticated tools will not produce a working solution if you don’t understand the problem.

I'm sure we've all seen some awful monstrosities—or created them ourselves—that we could call a technically working solution to a given problem ... but it doesn't mean anyone wants to work on it. Keeping complexity at bay requires finding the simplest solutions that isolate essential and accidental complexity. Simplicity is hard, and it requires doing this well, constantly. It is [ahem] essential to spend the time required to isolate the problem and articulate it clearly. If you can't isolate and articulate the problem without referencing your tech stack and tooling, or your explanation gets all muddy and convoluted, you haven't actually identified the essential complexity of a problem. You're still stuck in accidental complexity territory. And that's a horrible place to be designing and architecting your software from.

It's also critical to note that over the lifetime of any piece of software, as new things come up—new bugs, new features, etc—you have to keep re-engaging the same process, and evaluating/reflecting on how new things fit (or don't!) within your existing architecture/design. Failing to do so is what drives toward infinite complexity and endless "tech debt" in poorly designed software. Well-designed software isolates and encapsulates all the accidental complexity into its own space(s), leaving open avenues to adjust and expand the software. Well-designed interfaces allow you to refactor, reshape, and grow the internals of a problem domain in isolation from its callers. This requires discipline from a software team—and its leadership—to take the time necessary to adjust and refactor as priors change. Such work should always be moving the needle toward greater team velocity and individual productivity.

> Did you take AP tests in high school?

Yep, sure did! I definitely remember what you're describing here.

100%. I don't have much to add but I've really enjoyed our discussion.

Have you seen this talk by Mike Acton? If you haven't, it might really resonate with you. https://www.youtube.com/watch?v=rX0ItVEVjHc

https://archive.ph/4A53o

Elegant code and clean architecture are no match for the chaos of the human mind.

Elegant code and clean architecture are no match for the chaos of the human mind.

Note how even the author thinks that nonsensically complex systems like Kubernetes and Zookeeper are the best possible designs.

the actual article was kinda disappointing.