Military aircraft that are capable of supersonic are quite small in comparison to airliners. The boom is considerably less. But it still generates noise complaints; the military only does it with impunity in war zones or hotly contested borders (Golan Heights).
The noise complaints and subsequent laws were one of the things that killed Concorde. It wasn't allowed to fly supersonic over populated areas, so it ended up only flying trans-Atlantic routes (didn't have the range to cross the Pacific.)
The other thing that killed Concorde was the fuel usage. For the same amount of fuel as a 100-passenger Concorde ocean crossing, you can fly a 400-passenger 747 round trip. As fuel costs have continued to climb, the economics have made less and less sense for the never-profitable Concorde.
It doesn't help that they had that bad crash outside of Paris, grounded the fleet, and then finally started them flying again on the morning of September 11, 2001 -- thereby immediately facing the worst market for air travel, and the highest oil prices, in decades.
(I don't have sources readily available. This is all stuff I learned from fellow museum staff working with G-BOAG at http://www.museumofflight.org/concorde )
Well, Concorde wasn't allowed supersonic over the populated areas of the USA, to be exact. Was never a problem elsewhere, and some of the contemporary subsonic airliners were actually louder than the Concorde.
As you say, catastrophes didn't help either, although they occur regularly with vanilla airplanes as well. Fuel efficiency is indeed a problem; however there was no further R&D put into the programme since the 1970s, so no wonder. A modern 747 is certainly way more fuel efficient than the original mid-1960s project.
0) Get to my destination faster.
Where are all those hypersonic planes we were promised?