Yup, RPI's internal audio sounds rather dull, which you might not notice until you buy an external DAC :D.

The RPI can do great audio out of the box without USB, just don't use the 3.5mm jack. Instead, remember that HDMI does uncompressed bitstream audio!

You can connect the RPI to a dac/amp with HDMI input or a digital audio extractor. The best quality will be found by connecting to an HDMI to SPDIF extractor and then running that into a good quality DAC.

The nice thing about this solution is that you're not dependent on the horrible and CPU-sucking USB offered by the RPI.

How to connect a single hdmi cable to a monitor AND an amp?

HDMI amplifiers, DACs, and audio extractors (which give a clean digital bitstream on SPDIF) provide an HDMI output. The output HDMI will continue to carry the video portion of the signal and usually (if not always) the audio stream as well.

That is a nice hack! I'm going to have to plug the HDMI port into my stereo amp and try this out.