Getting user perceived page load time down is vital for most sites. You'll see your bounce rate drop, pages per visit rise, and conversion rates go up. A better experience for your users is always better for you.

You can do something to make like better for those on mobile and dialup with server settings, image compression, and the like. You can't change their connection speed, but you can focus on their experience.