pretty soon the hard-drive is just thrashing like crazy, essentially freezing the entire system, for minutes at a time
As a side note, this is why I never configure swap space in a server. If a server starts swapping, it's a fail. It should have enough RAM to do the tasks it's supposed to do, without ever running out of RAM. When it runs out of RAM, either the software has to be reconfigured to work within the available RAM, or the RAM has to be increased; or some tasks should be allocated to other servers.
As a side note, this is why I never configure swap space in a server. If a server starts swapping, it's a fail. It should have enough RAM to do the tasks it's supposed to do, without ever running out of RAM. When it runs out of RAM, either the software has to be reconfigured to work within the available RAM, or the RAM has to be increased; or some tasks should be allocated to other servers.