I mean, this is intentional. They even allow you to set your meeting ID to a well-known number, like your company's published phone number.

If you want to join the all-hands meetings of a company I used to work for, you only need to go their website and lookup their primary phone number. That's the Zoom meeting ID.