No, that's the problem, a hacker can clone your SIM. it's not trivial, but it's not impossibly hard, as in there are known attacks and if your fortune is protected by SMS 2fa, you'd better hope you don't draw attention from a motivated attacker. SS7 attacks and others are not theoretical.