I answered a very similar question on stackoverflow here:

http://stackoverflow.com/q/1229241/25981

In this case the user wanted the program to run out of memory and the best solution I could come up with was to use ulimit, to limit the amount of memory available to the process.