I've actually had that question for management in general. I'm sure that much of what management--reasonable management, anyhow---does is valuable, but is it that valuable? I really don't know.
For programming, at least, I think it's partly just historical. That's how it's always worked. In a different setting, like a factory, this makes sense as no factory worker is going to have a big positive effect on productivity while a good manager might. But for programming, the ratio of productivity between programmers and managers is much less biased towards the latter.
I guess more progressive software development companies actually are changing this, paying programmers relatively more and reducing the amount of management. So perhaps it's just a matter of the industry adapting slowly.
I think there are going to be many parallels between professors and programmers, so the two questions probably overlap a fair amount. On the other hand, there are also many differences, so perhaps that sort of work is more (or maybe less) important in a university than in a software company.
I think it comes down to being a manager basically sucks. A lot of people, all other things equal, would rather not do it.
Think about it, a manager is fundamentally someone who has to deal with everyone's problems all day long. The one reason many people do it is because it is an obvious path to better pay in many companies. Otherwise who would voluntarily deal with other people's problems all day long?
That isn't to say I think they are necessary but that if you have a crappy position the obvious way to fill it is via higher pay.
For programming, at least, I think it's partly just historical. That's how it's always worked. In a different setting, like a factory, this makes sense as no factory worker is going to have a big positive effect on productivity while a good manager might. But for programming, the ratio of productivity between programmers and managers is much less biased towards the latter.
I guess more progressive software development companies actually are changing this, paying programmers relatively more and reducing the amount of management. So perhaps it's just a matter of the industry adapting slowly.
I think there are going to be many parallels between professors and programmers, so the two questions probably overlap a fair amount. On the other hand, there are also many differences, so perhaps that sort of work is more (or maybe less) important in a university than in a software company.