+1000 I'll tend to look at the speed coming off the server instead. At least I know that I can control 100%. If the user is on a dial up modem, not my problem ;)
There are many things you can do to improve the performance of your website, even to people with dial up - or worse - Mobile!
Compress data with gzip is probably the big one for modem users, but also batching scripts and CSS together, reducing image sizes, re-arranging code to reduce blocking during rendering.
These are all things under your control, and with current mobile browsers, these are things you really must care about
Getting user perceived page load time down is vital for most sites. You'll see your bounce rate drop, pages per visit rise, and conversion rates go up. A better experience for your users is always better for you.
You can do something to make like better for those on mobile and dialup with server settings, image compression, and the like. You can't change their connection speed, but you can focus on their experience.
