I tend to be skeptical of statistical analyses like this.
At the end of the day, it's usually someone who is paying or funding a task, and someone who will perform the task. The person performing the task wants to please the person funding it (in order to secure that funding). This commonly involves minimizing difficulty and giving optimistic estimates.
People who give straight answers (something like: this is really hard, it will take a lot of time, there are lots of unknowns, and might not be achievable given the budget) won't get the award. Those who overdo the bullshit and give transparently falsely optimistic estimates will not receive it either. So those who fall in the middle do end up getting the award.
TLDR being, model this effect Game Theoretically, where the "blowup factor" is derived from a Nash equilibrium between counterparties playing optimal strategies seeking to optimize their payoff.
Finally, take three companies:
Company A delivers a project in 50% of the time, and 50% under budget.
Company B fails to deliver the project, after consuming 100% of the budget.
Company C completes 90% of the project with 100% of the funding, and only needs 20% more money to finish it. The customer will most likely top them off, because they are so close, and it's a sunk cost.
Who makes the most money? Company C. Who is the biggest loser? Company A (despite being the best). Anyway, for my next trick, I'll explain why this same effect causes agile to make everything take longer, cost more, and make everyone more miserable.
Company (or team, or individual) A "the best" is perceived to not work hard. They are perceived to be underutilized. They miss out on so much potential revenue they don't make it to the next round.
Company B is obviously incompetent, and doesn't make it to the next round.
Meanwhile, company C, barely competent, is perceived to make best use of resources and remain fully utilized. They are received to work really hard. They get not just the initially agreed upon budget but a top off in addition. They have the most profit to compete in the next round.
If company (or individual A) makes it to the next iteration, they know how to manage expectations, stretch the budget, and make a theater of working hard.. thus turning into company C.
After enough iterations everyone kind of ends up like company C. Just good enough to complete the task with estimated level of effort some double digit percentage over the original estimate, but not incompetent enough to flop altogether. It is to this I attribute why software is usually behind schedule and over budget to a reliably predictable degree
Would love to hear your take on agile. You should write a "blog" post about it...
Company A would eventually get a good reputation and win in the long run.
In an idealized world consisting of perfectly rational econ agents company A would win out. However we live in a world of people full of cognitive biases and driven by fear, greed, sentiment, and a whole slew of not perfectly rational drives.
I've seen individuals and small groups that would fit in category A loose out to C's. People that quietly get on with their work without problem whilst a bunch of drama queens get all the attention for heroically fixing problems they created in the first place.
A team that crams agile sprints with tons of "tickets", makes features that aren't fully thought out (necessitating more "tickets" to fix them), and has steady influx of general bugs cropping up is perceived to work super hard. People like to see noses to the grindstone and people stressed out.
The team that works carefully and deliberately, does things well the first time, and seems to be moving at a relaxed pace is perceived to be lazy and inefficient. People are offended by this. This hit me hard the first time I was a project lead -- the customer was delighted, the product was steady and reliable, but my staff were super uncomfortable and insecure -- because the project was successful and they did their job well and like professionals, they realized that they weren't super busy. And when they aren't super busy it means they may be on the chopping block - it was terribly depressing. No amount of reassurance that, well, this is what life is like when you're good at what you do really settled anyone.
So, consciously or not, we fill our time with "work theater". We work really hard to write new code that then requires more work to get right and to fix, and with a backlog of work, it looks like we're working really hard. Being always a little behind, we then make the case for more funding, more staff, more resources, etc etc etc... And the cycle continues.
At the end of the day, it's usually someone who is paying or funding a task, and someone who will perform the task. The person performing the task wants to please the person funding it (in order to secure that funding). This commonly involves minimizing difficulty and giving optimistic estimates.
People who give straight answers (something like: this is really hard, it will take a lot of time, there are lots of unknowns, and might not be achievable given the budget) won't get the award. Those who overdo the bullshit and give transparently falsely optimistic estimates will not receive it either. So those who fall in the middle do end up getting the award.
TLDR being, model this effect Game Theoretically, where the "blowup factor" is derived from a Nash equilibrium between counterparties playing optimal strategies seeking to optimize their payoff.
Finally, take three companies: Company A delivers a project in 50% of the time, and 50% under budget. Company B fails to deliver the project, after consuming 100% of the budget. Company C completes 90% of the project with 100% of the funding, and only needs 20% more money to finish it. The customer will most likely top them off, because they are so close, and it's a sunk cost.
Who makes the most money? Company C. Who is the biggest loser? Company A (despite being the best). Anyway, for my next trick, I'll explain why this same effect causes agile to make everything take longer, cost more, and make everyone more miserable.