I previously used WireGuard and for a bit tried just having an SSH tunnel with autossh, but in the end just settled on using Tailscale, because it doesn’t ask me to manually manage the keys and also doesn’t drop around every 30 minutes for a bit and doesn’t need weird hacks to expose ports for my Docked network traffic.
That said, what messed with me greatly was the fact that Tailscale seems to have an MTU of 1280 whereas Docker by default had 1500 which lead to inexplicably dropped overlay traffic with nonsensical log messages in my reverse proxy web server.
It was quite the mess. I have no idea why Docker couldn’t just figure out that it needs the smaller MTU by itself, cause it listens for the Swarm on an interface that’s related to Tailscale and it can see what MTU that has.
Its due to some strangeness in general with tcpip layers that don't forward PMTU discovery ICMP messages. You'll see the same thing in some cell networks, and wireguard is particularly fragile here, because wireguard itself doesn't have a PMTU discovery mechanism.
Or, to be more exact, wireguard currently doesn't have a method to 'bubble up' a PMTU process to the inner wireguard interface from mtu-impacting events in its outer layer.
In applications where I've had to deal with this (wireguard over cellmodem networks), I tool my network setup to poll whatever the cell network mtu happens to be and then set the wireguard MTU appropriately.
This gets really painful though if you think you wanna do something like run a network that really wants a >1280 MTU over tailscale. It's pretty much not doable, and it is, in fact, my biggest gripe with tailscale. Yes, its suboptimal for the 'whole-internet' usecase, but I really do want my wireguard links to be 9000 MTU.
Maybe wireguard will get that in the future, since it is an acknowledged problem. I bet someone in the conjunction of secure networking and HPC spaces could even justify paying the wireguard team to implement it.
That said, what messed with me greatly was the fact that Tailscale seems to have an MTU of 1280 whereas Docker by default had 1500 which lead to inexplicably dropped overlay traffic with nonsensical log messages in my reverse proxy web server.
Basically, I had to delete docker_gwbridge and recreate it with some specific options: https://docs.docker.com/engine/swarm/networking/#customize-t...
It was quite the mess. I have no idea why Docker couldn’t just figure out that it needs the smaller MTU by itself, cause it listens for the Swarm on an interface that’s related to Tailscale and it can see what MTU that has.
Still, Tailscale in of itself is pretty nice.