Because you cannot verify your phone is not compromised at either a software or hardware level.

You would need independently verifiable hardware and all software running on a closed system (ie, no third party modifications to running software which would mean at most a trusted sandbox for other applications outside the proven path) to be able to trust it to reliably take your vote.

Thats on the order of correctness provability that NASA puts into launch vehicles but NASA doesn't have to contend with hostile actors seeking to undermine their software and hardware.

TL;DR: hardware security, software security, authentication of voters, and the tech literacy of the average person.

Because now instead of securing centralized voting locations and machines you somehow have to create perfectly secure software running on you Aunt Flourence's machine with 51 tool bars and 3 different bot nets installed and also make sure she can use it properly and securely. Oh also now you're accepting votes as bits over the internet giving nation states probably the juiciest target and the widest possible attack surface (see securing every voters computer).

Even using something like the IME and secure enclaves to take the computation outside the the range of your average exploit it's still vulnerable to attack.

Then even if you've perfectly secured the hardware and software you're just left with the largest login/key infrastructure problem of all time with the average voter having to understand how to not be tricked into not actually using your secured software and hardware environment...