I spent double my budget on my last road bike when I decided to go all-carbon, and my rationale was I'd be more likely to ride it if I spent way too much money on it. That was CERTAINLY the case, at least for the first few years.
Depends on what you like. I ride a Cannondale Supersix for sport and love it. I used to race and still enjoy training hard, so my priorities are lightness, rigidity, and handling. A carbon frame beats steel in all of those.
I think the reputation for harshness is undeserved. I rode an aluminum frame (the gold standard of harsh and rigid) for a long time and IMO the difference between frame materials is way smaller than say 23 vs 25mm tires.
That said, I also have a 70s era steel frame as my commuter. It's a fun, springy ride that is respectably quick, and I enjoy riding it to work. I would not want to do thousands of feet of climbing on it though.
Heh, all the arguments over frame materials, and what do people actually feel when riding? Their tires, and whether or not the cables slap and vibrate depending on how they are routed.
The supersix is what software should try to be. Better than it's competitors when released, and it's been refined year over year, ignoring fads, to become a phenomenal machine.
Also, frame geometry and tubing size plays a larger factor in how the frame bends moreso than just the materials. Granted, to a certain extent, frame materials will dictate the geometry (e.g. skinny aluminum tubes would be a horrible idea).
Harsh? You're kidding. I love my bike (2008 Trek Madone 5.7, I think? The one with the integrated bottom bracket, the 'lowest' model with that but is made in the US).
The first time I rode carbon, I knew it had to be all carbon. It's so smooth, like butter. Not even a contest vs aluminum. I haven't ridden steel in ages, but based on weight alone, yeah, carbon.
You don't really need one (despite the obvious danger if taking this in the literal sense). There are tests that can determine pretty much every metric a bike is trying to nail. It becomes more of a "what's the best choice for X purpose" than trying to pick the best overall.