The core issue is that Rubygems adds the `lib` directory of every gem to your path, so you end up calling stat a zillion times.
I am admittedly a bit out of the Ruby loop these days, so it's possible that this has been improved. The fastest way of course is to not load a zillion gems...