I like Agile, but I think the industry needs lot more Waterfall. Imagine building a house with Agile, you would start from the roof (rain is the most pressing customer's problem), and the foundation would be built last (after, 5 years into the future, a crack appeared in the house wall).
I also think what's problematic is this debate goes too much around project management, rather than actual methodology of building software (software engineering). I have an idea how we can reframe the debate to this effect.
When software engineers plan and write software, they actually build two things - the product itself and the infrastructure/tooling for it. From the project and product management perspective, the focus is on the first and the latter is often neglected as unimportant (and waste of time). This sort of jives with the human intuition, we want to build a house, while foundation and scaffolding are only incidental to it.
However, I can't help but notice that the software products that are actually most "agile", and easy to change, are written in a way where there is a huge amount of generic infrastructure and tooling with a tiny veneer of product ("business logic") on top of them (and in fact this goes all the way down). For a canonical example, Emacs, to the point where people even joke about it.
But also, this is completely different to how the commercial industry builds software. The focus is on very specific features, pieces of business logic, instead of generic infrastructure, that makes delivering changing features easy. That is a very short-term view, which actually becomes more expensive as the time goes on.
I think the way we should build (commercial) software products is the first one, lots of infrastructure and tooling which only incidentally happens to create a product. See also functional core / imperative shell, or Unix philosophy.
That's why I think Agile failed in the industry, because the project management people who pushed for it, didn't understand the above distinction. This distinction is also related to Eric Raymond's the Cathedral and the Bazaar distinction, although his is more about social organization around building software than the actual software architecture. (The relation manifests in Conway's law.)
To conclude, I think we need more Waterfall in the sense that we need to think (plan) more deeply about the infrastructure and tooling that we need to build our products. Jumping directly into building a product that solves customer problem will have detrimental effects on long-term productivity.
I would also add I see role of open source only as a relatively small part of all the infrastructure and tooling. As you get closer to the actual product, more and more infrastructure is specific to your business domain. The open source will not save you, because it's too generic. Emacs is not just a generic OS or a Lisp interpreter. It has huge amount of infrastructure around text editing specifically.
I think scrum and waterfall are both just procrastination. Instead of building the app, you do all this other stuff instead. Like when you clean your house instead of doing what you're supposed to be doing. Nobody wants to do real work, so productivity theater ensues. It's really all just productivity theater.
The most productive way is kanban because it's like sending orders to the kitchen in a restaurant. There's no due dates, here's the stuff that needs doing, just do it.
Yup. I think this was a sideways swipe at my analogy so I'm going to double down now because I'm right. You would never get your food before closing at a restaurant if the cooks had to do sprint planning and provide estimates and deadlines. For some reason this is not only acceptable but preferred in software industry.
couple of points. building software is so far from building a house that's not really worth comparing. (just bringing this up because the analogy is used quite a lot). second, the signatories on the agile manifesto may have done some management but most, if not all, were strong developers.
While I think you can write good software with other methodologies I do think agile is a good fit for something that is supposed to change a lot which was the original idea behind software (as opposed to hardware) existing. the core idea is that things will always change, set yourself up so you have early knowledge and can change tack when you need to.
Another thing is depending on your experience you may have been exposed to differing version of agile. I have seen many places where it has been distorted into a KPI / mini deadline / micromanagement framework, which was never the point. From my point of view agile was always about developers teaching managers how they can manage us effectively, given the uncertain nature of software estimation and process of trying ideas and learning new things feeding into working software. it's part of our professional duty to explain how to do these things properly.
> building software is so far from building a house that's not really worth comparing
I disagree. I think the comparison is between engineering part of building a house, i.e. creating blueprints. The actual "blue collar" work in SW engineering is making sure the whole thing compiles, runs and is tested.
> the signatories on the agile manifesto may have done some management but most, if not all, were strong developers
Yeah, that's what I am suggesting with my post, maybe it worked for them, but it was later misinterpreted, or they thought that the secret of their success lies elsewhere.
> is supposed to change a lot which was the original idea behind software (as opposed to hardware) existing
I disagree here. Software is easy to copy first and foremost (you don't need much materials). I think easy to copy does not mean easy to change. For example, DNA sequence is also easy to copy, but difficult to change. (And that's why people wish for rewrites from the ground up.)
> agile was always about developers teaching managers how they can manage us effectively
I disagree. That implies you have a much bigger problem - useless managers. Managers simply have to understand the domain they are managing (and ideally have hands-on experience). There shouldn't be any "teaching" going on.
But I also agree, in a way. I think who needs to listen to SW engineers are product and project managers, that the software is not just a bunch of features to be built, just like a house is not just what you see on the promotional render. That's what my post was about.
I find that many analogies that seem not to hold water actually hold a lot of water, frequently including some the purveyor hoped it wouldn’t.
“Low hanging fruit” is about as stupid an idea if you understand how to make performant systems as it is if you own a mature fruit tree. Going for the low hanging fruit on a real tree grants you the biggest mess. Neither domain works the way idiots pretending to be enlightened think it does. And you can paint a sane picture in both if you know wtf you’re talking about, even a little.
Similarly, what the home improvement industry has taught me is that you can build a house with siding and a roof for about 10% of the cost of the finished house. This house will keep you dry. It will not keep you warm or clothed and feed you. For that you need plumbing, wiring, ceilings, walls, insulation, and finally flooring. These are intensely labor intensive jobs, and if you 1) learn a few of them and 2) know the right people, you can build things to code and pay someone only to sign off on them. Drywalling doesn’t even require that, but it’s a shitload of work.
So yes, in fact you can build an MVP of the house. It’s just the bones and a couple of the most obvious features. And if you live in an old enough house, it’s been refactored and retrofitted so many times that you can’t tell that it used to look a exactly like five other houses within a block radius of your house. And so did all you’re neighbors’.
I also think what's problematic is this debate goes too much around project management, rather than actual methodology of building software (software engineering). I have an idea how we can reframe the debate to this effect.
When software engineers plan and write software, they actually build two things - the product itself and the infrastructure/tooling for it. From the project and product management perspective, the focus is on the first and the latter is often neglected as unimportant (and waste of time). This sort of jives with the human intuition, we want to build a house, while foundation and scaffolding are only incidental to it.
However, I can't help but notice that the software products that are actually most "agile", and easy to change, are written in a way where there is a huge amount of generic infrastructure and tooling with a tiny veneer of product ("business logic") on top of them (and in fact this goes all the way down). For a canonical example, Emacs, to the point where people even joke about it.
But also, this is completely different to how the commercial industry builds software. The focus is on very specific features, pieces of business logic, instead of generic infrastructure, that makes delivering changing features easy. That is a very short-term view, which actually becomes more expensive as the time goes on.
I think the way we should build (commercial) software products is the first one, lots of infrastructure and tooling which only incidentally happens to create a product. See also functional core / imperative shell, or Unix philosophy.
That's why I think Agile failed in the industry, because the project management people who pushed for it, didn't understand the above distinction. This distinction is also related to Eric Raymond's the Cathedral and the Bazaar distinction, although his is more about social organization around building software than the actual software architecture. (The relation manifests in Conway's law.)
To conclude, I think we need more Waterfall in the sense that we need to think (plan) more deeply about the infrastructure and tooling that we need to build our products. Jumping directly into building a product that solves customer problem will have detrimental effects on long-term productivity.
I would also add I see role of open source only as a relatively small part of all the infrastructure and tooling. As you get closer to the actual product, more and more infrastructure is specific to your business domain. The open source will not save you, because it's too generic. Emacs is not just a generic OS or a Lisp interpreter. It has huge amount of infrastructure around text editing specifically.