I was also labeled as gay in milieu for turning down a woman.

I was hit on by gay men in an extremely uncomfortable fashion more than once.

I was also assaulted by complete strangers for no reason at all.

I was also isolated and bullied by my schoolmates in kindergarten and primary school (boys and girls).

And if I go and complain about it I will be labeled as less than a man. So why should I care?

p.s.: Disclaimer. Since I grew big and strong enough to do it I also disallow any kind of improper behavior in my presence.