You're describing how most IPMI controllers are implemented. This sounds great and all, until you realize the vendors don't bother to keep things up to date, and run all sorts of extra shit so they have a bigger feature list.
Yes, they use Linux, but, as you say, they run a bunch of other crap that's old and buggy, and the implementation of the protocol itself is not stellar.
How many bugs were found in IPMI implementations vs. ssh over the last 5 years?
We don't need IPMI, we really only need ssh, and a console that allows setting things up (like Open Firmware on RISC machines).
We don't need a new protocol for remotely accessing our machines, when we can remotely export the plain old machine console securely.
Some vendors do work the way I described, at least for their RISC offerings, although, for example, the Oracle ILOM runs some Java web server crap by default. But at least you can turn it off and use pure ssh! No IPMI.