I mean, you could probably get away with just keeping the `Makefile` around. But for the stuff I installed from source, I was often interested in keeping the source code around anyway, for curiosity's sake. And hard drives are big, while source code trees generally aren't - comparatively speaking.
Fair enough. In that job I was working on a remote Linux system that was quite outdated. So everything I wanted to install (newer version of Emacs, etc) required me to build so many libraries, as the system ones were too old (Emacs alone required 50-100). I didn't want the hassle of keeping all the source code around.