I usually do use scp, but it becomes problematic when there are symlinks hanging around. scp just follows them and ends up copying the same file many times.
This is especially bad if you have a directory structure that references a higher-level directory.
In my experience, using rsync to do just a raw copy is not faster than the ssh+tar approach. rsync wants to poke around and optimize, and that's incredibly awesome, except when there is absolutely no chance of optimizing at all, in which case it's just dead weight. I don't know of anything that can beat ssh+tar (possibly with the encryption tuned cheaper, as mentioned above); it especially blows raw scp out of the water for lots of small files.
This is especially bad if you have a directory structure that references a higher-level directory.