That's exactly what happen to my SO and I can see how this can be an issue to many people. The unexpected and unwanted convo with a particular person happen just because he had mobile number saved in phone's address book and despite of not giving Signal access to contacts, the presence of SO was announced.