I run private matrix server and all the ux problems it has when you want encrypted chat compared to Signal are just massive turn off for most people. I hope they fix it one day.
Also it's two pretty different things. Signal is mostly personal 1on1 and Matrix is big chat rooms.
I am not sure it is so clear win. It has chance to replace work chat platforms like Slack but i am still doubtful about casual users which is much harder and probably also more important.
The big killer feature of Signal is that once i "convert" someone to use it they always have some people from completely different social bubble that also uses it and that app immediately becomes more useful.
This wouldn't be possible without the phone number. I've used to share your view but nowdays i think Signal way is actually genius deliberate decision and one of the main reasons why it's picking up steam. Encrypted messaging with my buddies has never been an issue I need to replace Facebook Messages. I've realized this when some random Airbnb host started to naturally communicate with me over Signal instead of SMS without even mentioning it.