There are a lot of reasons this may be. Your Internet may be faster than your average user's. You may be closer geographically to your server. Different browsers will show different load speeds. The site may be cached for you most of the time.
I know you weren't actually asking, but I figured it's worth stating this for someone else who might experience the same thing. Load times do vary greatly. Digging down into the Analytics can show you where/why they vary. And as you said and as always with Analytics, look for trends more than exact figures.
+1000 I'll tend to look at the speed coming off the server instead. At least I know that I can control 100%. If the user is on a dial up modem, not my problem ;)
There are many things you can do to improve the performance of your website, even to people with dial up - or worse - Mobile!
Compress data with gzip is probably the big one for modem users, but also batching scripts and CSS together, reducing image sizes, re-arranging code to reduce blocking during rendering.
These are all things under your control, and with current mobile browsers, these are things you really must care about
Getting user perceived page load time down is vital for most sites. You'll see your bounce rate drop, pages per visit rise, and conversion rates go up. A better experience for your users is always better for you.
You can do something to make like better for those on mobile and dialup with server settings, image compression, and the like. You can't change their connection speed, but you can focus on their experience.
I know you weren't actually asking, but I figured it's worth stating this for someone else who might experience the same thing. Load times do vary greatly. Digging down into the Analytics can show you where/why they vary. And as you said and as always with Analytics, look for trends more than exact figures.