>... accused of having "a lot of spare time". I find this rather offensive.

isn't strong enough to me. Essentially when someone tells you you have "too much free time" they're saying:

"I judge the way that you're spending your time to be completely worthless."

I walk away at that point. We obviously have different priorities and values, and I don't have enough "free time" to want to discuss it with them.