I don't understand how men wearing dresses works as a comparison to women wearing pants, as there hasn't been an equivalent attempt by men to adopt dresses as everyday attire.
Regardless of the history behind the current situation, the fact remains that women are more free to wear "male" clothing than men are to wear "female" clothing.