Have you spoken to men who have joined yoga courses? Same thing there: it's uncomfortable when you're the odd one out, and assholes who are going to asshole are most egregious towards those who are "other."

The underlying reason for the disparity can still be "group A people are somewhat less interested in that thing than group B people" and not "group B people actively drive out group A people." Everything else just follows because humans suck.