It's not that people blame Google for the site looking crappy, but they do blame Google if the top search results aren't useful -- and that includes the site being unusable. Google also makes a huge amount of its revenue off of pay-per-click display advertising, and if it's referring customers all day long (through organic search) to sites that don't convert, then it's bad for everyone.

This is nowhere near the first pagerank modifier based on site performance -- it's well-known that Google penalizes sites that are slow to load, because users click away from those sites more frequently.