This is no better than ssh in a loop, which is trivially done by a shell script - no systemd needed.
However, when you have shitty NAT routers (SonicWall, any AT&T fiber device, for instance), the connections will be timed out or will die and there'll be long periods where you're waiting for the next iteration of the loop, and/or sometimes it'll get stuck and never try again.
autossh deals with this by actually passing traffic and taking action if traffic doesn't move.
Today I learned that some people make mistakes, but I already knew that ;) ServerAliveInterval doesn't do this properly and consistently.
I've used my own autossh type script for two decades now. It's mostly used to give access to machines behind shitty NAT, and/or that have addresses that constantly change, and/or for systems on CGNAT, like Starlink.
If ServerAliveInterval works so well and negates the need for something like autossh to exist, then why have sessions created by my script, which has ServerAliveInterval (and ServerAliveIntevalMaxCount) gotten hung up where the script needs to kill the old and create a new ssh connection now and then? My script logs each timeout, each session hang, and each new connection, and depending on the network, it can happen often.
Please read the bit where it's explained how autossh sends test data back and forth. Do you think you just magically and cleverly discovered ServerAliveIntevalMaxCount and that the autossh people have no idea that it exists?
Or perhaps they know it exists, they know it's not perfect, and they used another mechanism to make up for the shortcomings of what ssh offers out of the box?
> For example, if you are using a recent version of OpenSSH, you
may wish to explore using the ServerAliveInterval and
ServerAliveCountMax options to have the SSH client exit if it
finds itself no longer connected to the server. In many ways
this may be a better solution than the monitoring port.
Just to clarify that we're talking about the same thing in case I misunderstood something: autossh (style) scripts do these things:
1. fake data to keep a connection "fresh" for shitty middleware
2. detect connection which are stuck (state = open, but no data can actually round trip) and kill them
3. restart ssh when that happens
Is that what we're talking about here? I think people are saying that points 1 and 2, but not 3, are covered by SSH's ServerAlive* options. And that's also how OpenSSH advertises and documents those options, and apparently even how autossh talks about it in their own readme.
You're saying that those options don't actually solve points 1 and 2, while (your/their/etc) autossh does properly detect it.
Correct so far?
If so that seems like a bug in OpenSSH (or whatever implementation) which should get appropriate attention upstream. Has anyone reported this upstream? Is there a ticket to follow?
PS: I think we're all in agreement that option 3 is out of scope for stock OpenSSH (regardless of what other tools do)
I haven’t revisited this issue in years but on a project for thousands of similar devices we found autossh much more reliable.
I believe the issue is that the connections often fail or get wedged in other network layers; the only way to be sure that your ssh tunnel isn’t: a) lossy enough to “keep alive” but too lossy to send data, or b) isn’t just always waiting on TCP retry backoff, or c) etc, is to use the tunnel to transmit actual data at the application level.
> is to use the tunnel to transmit actual data at the application level.
Isn't that exactly what ServerAliveInterval does? The man page says: "ssh(1) will send a message through the encrypted channel". A plain TCP keepalive wouldn't count as being "through the encrypted channel".
I don't know if I would call it smoke and obfuscation, at the time systemd was not widely deployed and the ssh functionality was not as developed, so it made sense to use autossh. Now it sounds like it doesn't make sense anymore. It happens.
You summarized things well. #2 is the primary reason that ssh in a loop doesn't work as well or as reliably as autossh (the program discussed here; it's just coincidental that my own automatic ssh script is also called autossh).
This approach works very well. I've had dozens of extremely remote systems hooked up this way for about 8 years. The only problem I've seen is that occasionally the server ssh process will get stuck, so you have to log in to the server and kill it. It seems to happen when a remote goes offline and reconnects without closing the old connection first.
If I were doing it now, I'd probably use wireguard, probably. This is simpler to set up and works great.
> The only problem I've seen is that occasionally the server ssh process will get stuck, so you have to log in to the server and kill it.
You also need ClientAliveInterval on the server side (in addition to ServerAliveInterval on the client). In other words, both the client and the server need to be configured to monitor the connection. With this setup I had no issues with reconnections.
systemd's RuntimeMaxSec should help in this case but I've never had trouble with sshd personally
To add more context I use the above service to ssh from my phone to my laptop via my desktop PC. The service runs on my laptop and binds port 22 of my laptop to port 7070 of my PC but wiregaurd would probably work similarly
closing ssh doesn't close the ports if they are being used, at least with ControlMaster. You need to run something like this to force the ssh daemon to close the port
ssh -O cancel -L 4102:localhost:4000 pc
but if ControlMaster is stuck maybe autossh is better in that case, or use this:
This is not very dissimilar from how the RIPE Atlas software probe (debian package) maintains a persistent SSH command/control session to the anchors and RIPE infrastructure. As I recall it installs itself as a systemd service.
No passphrase for the key?
What about spotty connection? Doesn't WantedBy block startup on this starting properly? (I'm pretty sure I've been soft locked out of my computer when Comcast decides to do Comcast things.
Ok now how to tell if the connection is running in the systemd status output? Cos this will show as active even when the connection is down or trying to reconnect.
exits (and so restarts) every 20min, e.g ensuring there's no hung sshd on the other side for longer than that.
IIRC if there's an active connection on the forwarding thingy, that ssh command won't exit until the forwarded connection is closed, so this won't interrupt an active forwarded connection every 20min.
It doesn't by default, but you can set the AUTOSSH_GATETIME environment variable to 0 so that autossh retries even if the first connection attempt fails.
14 years ago, i was using auto ash to keep SSH tunnels up; but at some point (quite far back - perhaps 2016?) ssh gained everything needed to do this internally except the restart.
At this point I configure all of the keep alive and retry options in ssh_config and sshd_config, and use
While true; do ssh user@host ; sleep 10; done
To get the same effect, but with much more flexibility - e.g. alternating connection addresses on a multihomed host, add logging, run from daemontools or systemd unit instead of a loop and let them track the process and restart, etc.
mosh is for interactive sessions, to keep them working when the connection is flaky.
autossh is for keeping unattended ssh tunnels alive, if the connection is flaky or one end is only intermittently available. So for using tunnels for the sort of thing you might otherwise use a VPN for.
If your concern is to have secure tunnels between hosts, you should probably use spiped rather than SSH, since it uses a separate TCP connection for each pipe -- this avoids the "connection dropped" problem and also the "multiplexing many connections over one TCP connection" performance hit.
Also, spiped is way simpler and more secure than SSH. (On my servers, I tunnel SSH over spiped, to protect the sshd from attacks.)
The last time I used autossh it was on a client site to keep 2 layers of ssh tunnels open to jump through their network isolation hoops.
In general, when flexibility is possible, such a use-case nowadays would often be better served by deploying WireGuard. Grouchy, out-of-touch corporate net admins probably don't even know what it is and insist on their antiquated Cisco VPNs.
I used to be a happy user of `autossh` until 2023. I used it on Cygwin on Windows and was quite happy how reliably it set up my tunnels (upon tunnels) in a flaky corporate network.
`autossh` worked reliable compared to `ssh`s many timeout options.
I’d recommend https://eternalterminal.dev/, compared to mosh(poor colors support), this is the only thing that manages to consistently keep up my ssh sessions.
I love ET. Some discussion here of its advantages over mosh: https://news.ycombinator.com/item?id=21640200. Beware that ET does phone home: depending on how it's packaged for your system, telemetry is enabled by default in /etc/et.cfg.
Wouldn’t ssh with systemd or auto ssh be a more secure means of remote access to apps (like http/https apps) than the zero trust network access solutions (like Cloudflare Tunnels which terminates the TLS) or even Tailscale (which should be a trusted third party)?
You set up public key authentication with SSH to a reverse proxy, a persistent tunnel, and a socks proxy. In a Firefox profile, you set localhost:port. Done! All your services are available in that browser all the time.
Autossh with a reverse ssh tunnel can also be used to expose an internal service to the Internet through a VPS.
SSH has been very secure over the decades. A good feature of SSH is that it can jump from host to host, unlike VPN.
SSH protocol does not protect against weak configuration, e.g. password authentication without brute force mitigation. Zero-trust can be misconfigured too, so it depends how well either of them is configured.
Not 100% the same use case as autossh was built for maybe, but I'm now simply throwing tailscale on every box i need to interact with. Does away with all the port forwarding stuff, it's absolutely delightful.
How much reliance on third party am I subjecting myself by using Tailscale? What happens if I make a local connection to a machine/service running on Tailscale, does it still go out of the local network? If so, is the bulk of the payload transferred locally? Is there any advantage on using it if the machine/service is easily accessible over ipv6?
It will route directly over the local network when possible.
It will be encrypted through the VPN, so there will be some overhead. But will be as direct as it can be. It only routes through tailscales servers as a last resort, when it can’t find a direct route at all (usually because NAT holepunching fails somehow). Their “DERP” relay servers just relay the encrypted connection. I think you can use your own relay servers, but I don’t know if that feature can be disabled entirely.
Headscale can be entirely self-hosted. It still uses the tailscale client applications- but is compatible.
That's what Tailscale is built for, when it can it sets up a P2P connection, it only ever sends data through Tailscales servers if you are in a restrictive network environment (i.e. corp network that controls all inbound and outbound traffic)
good questions, pretty well answered by other commenters. if you are happy with the level of encryption you have on your 'plain' ipv6 connection, sure, use that.
additionally the acl/auth system, their dns and service discovery thing is nice, though not essential.
If "orthodox" means "vpn traffic cannot be established through third-party cloud infrastructure", then tailscale and any other cloud-hosted ZTNA solutions wouldn't be accepted in that kind of enterprise.
Sometime back, I had a rapsberry pi connected to wired network of a coworking space. I remember using autossh to keep a tunnel open with one of my VPS. Mainly used it as a torrent box. I added magnet links through qbittorrent webui installed on raspberry pi. Qbittorrent was configured to only run at night time to not cause issues for business work. Downloaded all sort of things easily reaching thousands of GBs throughout my time there. They never found out. Or they didn't care to look. Good times.
Rather than using AutoSSH for port forwarding and such, I just create a systemd unit with a restart policy. Then you don't need autossh at all, just use ssh.
I used to use autossh to set up a SOCKS proxy to tunnel my web browser traffic via my home network and it worked really well. Also had a ControlMaster on the tunnel which made SSH connections to my server instantaneous.
Nowadays I use wireguard an a dedicated SOCKS proxy. The upside is that I can access everything on my home network directly without having to tunnel.
Nice tool, but I'm getting tired of using port numbers for everything instead of more descriptive strings. My system has more than 10 tunnels and servers running, and since I only do sysadmin work once every half year or so, the port numbers are very cumbersome to deal with.
I believe these days SSH is willing to forward a UNIX domain socket to a remote TCP port, or a local TCP port to a remote UNIX domain socket, or any combination of the two families really. You could use names locally, if your client tools are willing to do AF_UNIX!
And if you are wondering, if you can just point your browser to a local unix socket (without setting up a proxy - which will listen on... local tcp port), then no, but maybe some day?
The nice thing about this is that, with filesystem permissions on one end and a check for SCM_CREDENTIALS or SO_PEERCRED on the other, you can effectively get user-based access control working between two machines.
I think this is the one remaining advantage of ssh tunnels over using a VPN.
NB if you're doing this sort of thing, you probably want to add `StreamLocalBindUnlink yes` to the ssh options.
Agreed, I have so many services that all want to run their own webserver, db, elasticsearch, etc. I have to start using non-standard port numbers and it’s a burden to have to keep track of them.
I use this to set up reverse tunnels, for example to set up MongoDB replica sets which sync through SSH. It kind of simplifies the security aspect of replica sets a bit, since then MongoDB does not need to be exposed to the internet and no VPN setup is needed.
Way back redis didnt have passwords at all. That got added but there was no secure transport support.
So I ran redis in a higher memory box at rackspace separate from my db and my app server. I used autossh to forward 6379 from localhost on the app server(s) to the redis server. Worked like a charm and never caused any issues.
Other commenters are right in that wireguard is a great modern solution to this!
In good conditions you can go months without sending a single byte of traffic between an SSH server and client and both will pick up the connection just fine when it's time to communicate again.
You could cut off traffic between them for any amount of time and they would be none the wiser as long as the network connection is back to normal when they finally try to send traffic again.
(I had SSH sessions in a QA lab persist as if nothing had happened after the connection between the endpoints was down for almost a week while we replaced the aggregation layer routers. They never saw a link state change since the access layer switches were up the whole time. They never attempted to communicate while the connections between those were down, so there was never any problem as far as they were concerned.)
The keepalives and connection checks and so forth are mostly to account for things like stateful network gear (firewalls, NAT routers, etc) between the endpoints that will cease relaying traffic between them if they are quiet for too long.
reply