Interesting read. I work with Ruby on Rails myself but I'm not that knowledgeable about scaling it.
It seems to me they didn't really know what setup they were running if they are wondering what a 'mongrel' is.
I hope they weren't trying to serve the site on only a couple of mongrels.
My first thoughts,
Benchmark a bit, use a tool like fiveruns to find out what's really happening. I wonder what the real bottleneck is.
Of course they shouldn't use a shared database server, but I'm wondering if they are they using caching? What I'm reading about that site I think they could cache the hell out of it.
I've only used the builtin Rails cache methods but a tool like memcached should help out on all the database requests.
Thankfully there are numerous parties out there that will handle the scaling issues for you (heroku.com, engineyard.com) so you can concentrate on what's important, the app itself.
If the application is performing idiotic operations such as "[querying] MySQL 1500 times with 1500 separate queries (instead of one query returning 1500 rows).", merely moving to a better provider or using more powerful servers won't help. A modern web-based application is very complex -- the programmer has to understand how the database and application communicate, how HTTP works, how to cache data, etc.
As a RoR developer I have often been hired into optimize various services. One of the biggest issues I have seen is this exponential growth of queries.
I have routinely seen requests at clients with 2-3000 queries (even with query caching). Most of them are small, but at that amount it doesn't matter how small and efficient each query is.
I love AR to death as an OR library, but it is extremely easy to get into these kinds of issues when you iterate over a large dataset and then call associations of associations without thinking too much about it. I don't think it is only an AR issue, conceptually I think it is true for all ORMs.
They can be a PITA to unravel and are often very hard to do in a clean AR like way in more complex data models. Normally you end up doing some fairly un AR like pre loading like you would do in a pre ORM app, which while ugly works.
Agreed, I do all my dev with query trace in order to make sure I'm not looping back into individual queries.
to those that don't know, Rails doesn't load the children of AR objects by default, so if you do something like
Select * from books
and then iterate through the books and get books.author_name where the author data is a relational table, you're going to get a separate query for each row.
If you're operating a top site/service, scaling is the most important issue you have. It's naive to think that you can get someone else to handle it for you flawlessly.
It seems to me they didn't really know what setup they were running if they are wondering what a 'mongrel' is.
I hope they weren't trying to serve the site on only a couple of mongrels.
My first thoughts,
Benchmark a bit, use a tool like fiveruns to find out what's really happening. I wonder what the real bottleneck is.
Of course they shouldn't use a shared database server, but I'm wondering if they are they using caching? What I'm reading about that site I think they could cache the hell out of it. I've only used the builtin Rails cache methods but a tool like memcached should help out on all the database requests.