If you turn "I only have sex with women" into "I only have sex with white people", suddenly it sounds pretty offensive. Can we judge those two statements equivalently? It's not a good argument at all.