Replace "men" and "women" with "blacks" and "whites". Is it reasonable for white people to feel a need for caution talking to black people in an elevator?
If they're using racial slurs in their conversation, yes, it is. It entirely depends on the topic being adequate for the venue or not, not on the races or genders of the participants.
