Finally someone with a setup similar to mine (though I use Vim instead of Emacs, WeeChat instead of irssi, and am planning to move from Screen to tmux).
Using a terminal multiplexer like Screen or tmux as your "IDE" is one of the most amazing things I've discovered. It's completely changed how I work - I can start/stop working at any time from any machine that has an SSH client. Apparently with tmux, it'll even preserve your tiling layout.
My main development "box" is a 512MB Linode running tmux, weechat, vim, and some zsh consoles. Efficient, consistent and reachable from all kind of machines and places.
How much does that cost you? I was thinking of doing something similar, except through Amazon EC2. It could be quite a bit cheaper than having a home server, and definitely wouldn't be so loud.
Using a terminal multiplexer like Screen or tmux as your "IDE" is one of the most amazing things I've discovered. It's completely changed how I work - I can start/stop working at any time from any machine that has an SSH client. Apparently with tmux, it'll even preserve your tiling layout.