I do the same for one-off tasks. I use ^a as the escape key locally and leave the remote server(s) to use the default ^b because I might need to login to random servers with no shared home or login as a common user and they often have restrictions on adding/changing custom dot files.

If it gets too annoying, the remote session is openend in a new terminal tab and I temporarily remap the escape key to ^a on remote tmux.