Create a reverse SSH tunnel for remote access to a restricted Linux machine

I did not find a clear modern description of this set-up so have written this post to help others.

Say you have a Linux machine to which you want to ssh to but that machine is behind corporate or other firewalls etc which you have no control over, and thus you can not forward an external port for ssh etc. Let's call that the restricted machine and we assume you can make changes to it, e.g. locally at that machine. This post describes how you can set up a reverse ssh tunnel initiated and maintained from that restricted machine to a port on a Linux server elsewhere on the internet that you control. With the reverse tunnel in place, you can simply ssh to that port on your server to get a tunnelled ssh session back to the restricted machine through the restricted firewalls. Of course you should always check you have permission to do this with whoever controls the restricted network.

Note that autossh was previously often used for this (and nearly all existing online tutorials about reverse ssh tunnels use autossh) but autossh is redundant nowadays since modern openssh versions can monitor link health and exit if the link fails, and systemd can then be used to restart the ssh tunnel. So autossh is not required, we can implement an automatically maintained ssh tunnel using standard ssh and systemd alone.

In summary the steps are:

1. Set up the restricted machine.
2. Set up the server.
3. Final set-up and test.

Set up the restricted machine

On the restricted machine, create an ssh key pair in /etc/sshtunnel/ with an empty/no pass-phrase:

$ sudo mkdir -p /etc/sshtunnel
$ sudo ssh-keygen -qN "" -f /etc/sshtunnel/id_rsa

Create/paste the following systemd service file in /etc/systemd/system/sshtunnel.service and then edit it as needed for your specific server and ports:

[Unit]
Description=Service to maintain an ssh reverse tunnel
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
ExecStart=/usr/bin/ssh -qNn \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o ExitOnForwardFailure=yes \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  -i /etc/sshtunnel/id_rsa \
  -R 9001:localhost:22 \
  sshtunnel@server.net -p 443
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

You may likely want to change some of the unlabelled ssh parameters above so they are described below:

Set up the server

On the server, create an account for the sshtunnel user but restrict it so that shell logins and remote commands are not allowed. The server will only allow the remote sshtunnel user to set up port forwarding:

$ sudo useradd -m -s /bin/true sshtunnel
$ sudo mkdir -p ~sshtunnel/.ssh

Copy the public ssh key you created above on the restricted machine (/etc/sshtunnel/id_rsa.pub) to the sshtunnel user's authorized keys file on the server (~sshtunnel/.ssh/authorized_keys).

Ensure the owner/permissions are strictly set as ssh requires them.

$ sudo chown -R sshtunnel:sshtunnel ~sshtunnel/.ssh
$ sudo chmod 700 ~sshtunnel/.ssh
$ sudo chmod 600 ~sshtunnel/.ssh/authorized_keys

Final set-up and test

On the restricted machine, enable and start the service:

$ sudo systemctl enable --now sshtunnel

Check the status and logs with:

$ sudo systemctl status sshtunnel
$ sudo journalctl -u sshtunnel

Then on your server test the tunnel to the restricted machine with the following:

$ ssh -p 9001 localhost

To diagnose problems, check the sshd logs on both the server and the restricted machine:

$ sudo journalctl -u sshd

External public access via server (optional)

For most users, the above configuration is completely sufficient. The tunnel is available by making an ssh connection to the port on the server localhost interface. Either you connect to this locally from within that server, or if you are connecting from another machine then you use the ssh ProxyJump configuration keyword to automatically redirect to the restricted machine via the server proxy, again via the localhost interface.

Some users may want to make the tunnel port also externally accessible on the server public network interfaces, e.g. for use by users who don't have ssh access to the server itself. Be aware you are opening a public port directly connecting to the restricted machine if you do this so this approach should be avoided, or at least disable password authentication and only allow ssh key authentication on the restricted machine.

There are 2 extra steps to add this:

1. On the server, enable GateWayPorts yes in /etc/ssh/sshd_config. Then sudo systemctl reload sshd if you changed this.

2. On the restricted machine, prepend a colon (':') on the port number in the /etc/systemd/system/sshtunnel.service file, so the ssh option is -R :9001:localhost:22 rather than -R 9001:localhost:22. Then do sudo systemctl daemon-reload and sudo systemctl restart sshtunnel.

Removal

To remove all the changes described here do the following.

On the restricted machine:

sudo systemctl disable --now sshtunnel
sudo rm /etc/systemd/system/sshtunnel.service
sudo rm -rf /etc/sshtunnel

On the server:

$ sudo userdel -r sshtunnel

Disable GateWayPorts yes in /etc/ssh/sshd_config if you added it then sudo systemctl reload sshd.