This deserves a longer response but a short one will have to do for now.
You talk about software that sucks. However, one of the quirks of software is that sometimes even when it sucks it can succeed and even when it doesn't suck it can fail. For example, technologically google wave didn't suck, but in terms of actually providing useful features for people that justified its use, it didn't have a leg to stand on. Then look at wordpress, which started out sucking but because it was open and because it had developed a strong community around it ended up getting better and better to where it was finally sort of decent. Or look at PHP. As a language it definitely sucked at the start, and there's a strong argument to be made that it still sucks. But it is perhaps the most popular language for web development in history.
Most software is even more difficult to determine success or failure with because even though a software project might not be a success immediately it could be a success down the road. Another case in point would be the Mozilla Project. At the outset it sucked, but eventually it became pretty awesome. How much of the awesome of today's firefox is rooted in the code from the early days of mozilla and how much is due to subsequent dev. work? How do you tell the difference between software that sucks because it is rotten through to the core and software that sucks because it has a layer of crap on top of awesome internals?
And then how do you track everyone's contribution to software? Sure you can keep track of commits, but that doesn't track inspiration and ideas. Sometimes the fundamental design or mechanism for a given piece of software will mostly be due to a different dev. than the one who implemented it in code, and often there is no paper trail whatsoever that that's the case.
Ultimately there's no objective way to measure either talent or success except in the extremes. Some people's subjective estimates can still be reasonably accurate though, but usually it takes a talented and experienced dev. to be able to judge another dev.
This is a good point. There's a time behavior to software that is hard for most people, and especially non-engineers, to understand. Software that's superficially weak or seems to provide functionality no one wants might be a hidden gem.
So, if people have an unfailing sense of good and bad software, which they might not, it would be eventually consistent at best.
You're also right that individual talent is, for the most part, impossible to measure.
You talk about software that sucks. However, one of the quirks of software is that sometimes even when it sucks it can succeed and even when it doesn't suck it can fail. For example, technologically google wave didn't suck, but in terms of actually providing useful features for people that justified its use, it didn't have a leg to stand on. Then look at wordpress, which started out sucking but because it was open and because it had developed a strong community around it ended up getting better and better to where it was finally sort of decent. Or look at PHP. As a language it definitely sucked at the start, and there's a strong argument to be made that it still sucks. But it is perhaps the most popular language for web development in history.
Most software is even more difficult to determine success or failure with because even though a software project might not be a success immediately it could be a success down the road. Another case in point would be the Mozilla Project. At the outset it sucked, but eventually it became pretty awesome. How much of the awesome of today's firefox is rooted in the code from the early days of mozilla and how much is due to subsequent dev. work? How do you tell the difference between software that sucks because it is rotten through to the core and software that sucks because it has a layer of crap on top of awesome internals?
And then how do you track everyone's contribution to software? Sure you can keep track of commits, but that doesn't track inspiration and ideas. Sometimes the fundamental design or mechanism for a given piece of software will mostly be due to a different dev. than the one who implemented it in code, and often there is no paper trail whatsoever that that's the case.
Ultimately there's no objective way to measure either talent or success except in the extremes. Some people's subjective estimates can still be reasonably accurate though, but usually it takes a talented and experienced dev. to be able to judge another dev.