I don't agree with that. In fact, I wear a helmet whenever I am near a wall because I have seen way too many formerly solid looking rocks fall, even on granite.
Heck, a few years ago, a 2000 ton slab of rock fell off of Half Dome, which is a famous granite dome in Yosemite. I can't find the article now (of course), but I read a great article about how rock expands and contracts with the heat of the sun, water gets into cracks and plates will break off.
This article is okay, though it isn't the one that I'm thinking of. If I can the correct article, I'll edit this answer.
You're missing a very important distinction though: for a solo climber the main thing out of their control that's different from normal rock climbing is holds breaking unexpectedly. For that risk, having lots of people climbing the rock helps increase the chances that such unexpectedly weak holds will have been broken already by someone else; obviously loose rock isn't so much of an issue, for the simple reason that you know it's loose and can climb appropriately.
Rock fall meanwhile is dangerous to all climbers, whether or not they're using ropes. Sure, it's more dangerous to a solo climber - you might get lucky and just get knocked out, something a rope can save you from - but we're talking about a moderate risk vs. high risk comparison, not zero-risk to high risk. That 2000 ton slab you gave isn't a very good example, because something that big has a pretty good chance of killing you rope or not.
FWIW, I do a lot of cave exploration, and in caves the "already broken off" effect is a very powerful one. Virgin passage that's never been explored before can be a very scary place to be because everything can be loose and ready to fall. This is particularly true in areas that undergo freeze-thaw cycles, such as upper parts of alpine caves - I've been in places where every single bit of rock, including the ceiling, is heavily fractured due to frost. Breakdown piles are also a big problem - they're huge piles of rocks and boulders, generally due to ceilings gradually collapsing, and you have to be really careful around ones that haven't been visited frequently because pretty much anything you stand on could be unstable. Lots of cavers have been trapped in breakdown piles that shifted unexpectedly.
The underlying cause of this problem in caves is simple: caves can be very static environments with no weather, increasing the chances that the force you exert on a rock will be the largest force it has ever experienced in that position.
Heck, a few years ago, a 2000 ton slab of rock fell off of Half Dome, which is a famous granite dome in Yosemite. I can't find the article now (of course), but I read a great article about how rock expands and contracts with the heat of the sun, water gets into cracks and plates will break off.
This article is okay, though it isn't the one that I'm thinking of. If I can the correct article, I'll edit this answer.
http://www.livescience.com/51490-yosemite-half-dome-rock-fal...