It's difficult bordering on impossible to judge even the success or failure of a given software project.
This I disagree with. Causes of success and failure are hard to tease out, but people know when software sucks. Also, if people can't use it, then all the cleverness in the world (in optimizations, for example, or in feature set) doesn't matter. It still sucks.
But the broad middle is a muddle. People with talent that has been squandered by circumstance, people without talent who have succeeded due to accident and luck, etc.
That's very true.
This is part of why I think Valve's self-organizing open allocation is superior. Management rarely knows who the best programmers are. The group of programmers, if they're good, usually can figure out an appropriate leader on a per-project basis. This doesn't generate the permanent, entitled leadership that management wants to see, but it gets the job done.
You also need to create room to fail. If the software project can't be saved, let it die so people can allocate their talents and energy to something that has better odds. Give the architects respect for trying and ask them to write up the challenges they encountered, so as to keep the knowledge in house. If you fire people for failing projects, then you lose that knowledge and will probably fail in the same way again.
This deserves a longer response but a short one will have to do for now.
You talk about software that sucks. However, one of the quirks of software is that sometimes even when it sucks it can succeed and even when it doesn't suck it can fail. For example, technologically google wave didn't suck, but in terms of actually providing useful features for people that justified its use, it didn't have a leg to stand on. Then look at wordpress, which started out sucking but because it was open and because it had developed a strong community around it ended up getting better and better to where it was finally sort of decent. Or look at PHP. As a language it definitely sucked at the start, and there's a strong argument to be made that it still sucks. But it is perhaps the most popular language for web development in history.
Most software is even more difficult to determine success or failure with because even though a software project might not be a success immediately it could be a success down the road. Another case in point would be the Mozilla Project. At the outset it sucked, but eventually it became pretty awesome. How much of the awesome of today's firefox is rooted in the code from the early days of mozilla and how much is due to subsequent dev. work? How do you tell the difference between software that sucks because it is rotten through to the core and software that sucks because it has a layer of crap on top of awesome internals?
And then how do you track everyone's contribution to software? Sure you can keep track of commits, but that doesn't track inspiration and ideas. Sometimes the fundamental design or mechanism for a given piece of software will mostly be due to a different dev. than the one who implemented it in code, and often there is no paper trail whatsoever that that's the case.
Ultimately there's no objective way to measure either talent or success except in the extremes. Some people's subjective estimates can still be reasonably accurate though, but usually it takes a talented and experienced dev. to be able to judge another dev.
This is a good point. There's a time behavior to software that is hard for most people, and especially non-engineers, to understand. Software that's superficially weak or seems to provide functionality no one wants might be a hidden gem.
So, if people have an unfailing sense of good and bad software, which they might not, it would be eventually consistent at best.
You're also right that individual talent is, for the most part, impossible to measure.
This I disagree with. Causes of success and failure are hard to tease out, but people know when software sucks. Also, if people can't use it, then all the cleverness in the world (in optimizations, for example, or in feature set) doesn't matter. It still sucks.
But the broad middle is a muddle. People with talent that has been squandered by circumstance, people without talent who have succeeded due to accident and luck, etc.
That's very true.
This is part of why I think Valve's self-organizing open allocation is superior. Management rarely knows who the best programmers are. The group of programmers, if they're good, usually can figure out an appropriate leader on a per-project basis. This doesn't generate the permanent, entitled leadership that management wants to see, but it gets the job done.
You also need to create room to fail. If the software project can't be saved, let it die so people can allocate their talents and energy to something that has better odds. Give the architects respect for trying and ask them to write up the challenges they encountered, so as to keep the knowledge in house. If you fire people for failing projects, then you lose that knowledge and will probably fail in the same way again.