> I always just used `--prefix="$HOME"` so that everything went into `~/bin`, `~/lib`, `~/man`, etc...

You can't just use rm as a blunt instrument. ~/bin will contain lots of binaries from lots of packages. You want to uninstall only one package. How do you know which files correspond to that package?