I think the intent of that comment is genuine. There are poor working multiple jobs to make ends meet and their are unemployed or underemployed poor that have tons of free time. There are middle and upper class people that work 100 hours a week and their are middle class people that sneak by on barely 40 hours a week. I do genuinely notice a spike in joggers in more well to do areas.

It may be that people with more money get to spend their time a bit higher on Maslow's hierarchy of needs, but I don't think it's strictly that all poor are busier than all middle or upper class.