Latency is a problem for me. Maybe for some users, who already have very slow internet, or who live in areas with more TOR relays, it is not so noticeable, but for me the speed difference is about 10 times. Mostly not because TOR would be unbearably slow (it is not slower than regular internet 8 years ago), but because regular internet is very fast where I live.
I mean, you'll notice my strategy is to never torify anything where I'd have to actually wait for it. Except IM, which is small enough that it doesn't matter, anything I've torified is downloaded in the background anyway.
