Simplified Persistent SSH Tunnels

Tutorials

I have written previously on this topic and I recently discovered a simplified way of doing this.

Instead of editing the existing article, I thought I’d leave it as a reference. Previously I used AutoSSH – which is a great program – but I found a far easier method where systemd can manage the tunnel.

Requirements:

  • Machines starting the connection must be using systemd
  • Understanding how to set up SSH key authentication (not covered in depth)

Goals:

  • Create persistent SSH connection from one machine to another
  • This connection will start a remote port forward
  • The remote port is the SSH listen port on the originating machine

Imagine the following scenario: you manage several Linux machines at various sites. These sites can be people’s houses or small businesses – a wide range of network configurations. Maybe some networks you manage and some networks are managed by the someone else. You want an easy way to access all these boxes but there are a few problems like not managing the firewalls, not having static IPs, or not wanting to use a VPN in general. There are only two things you control: the machine on your side and the remote Linux machine you manage. The machine on your side can be at your house, if you manage the DNS and/or firewall properly, or a cloud server. In my examples below, the [A] machine is the remote host and [B] is the one you own.

All of the [A] machines will reach out to the WAN and connect to your singular [B] machine. Since you control the DNS and/or firewall on your side, that’s not a problem. Since the [A] boxes are going outbound, nothing needs to be changed with their network or firewall. My example shows one machine, but this method can be used for several [A] machines to connect to one [B] machine with various reverse listen ports, all of them lead back to the 22 (SSH) port on each [A] machine.

[A1$] ssh -R 5000:localhost:22 user@bmachine
[A2$] ssh -R 5001:localhost:22 user@bmachine
[A3$] ssh -R 5002:localhost:22 user@bmachine
[A4$] ssh -R 5003:localhost:22 user@bmachine
[A5$] ssh -R 5004:localhost:22 user@bmachine
---
[B$] ssh a1user@localhost -p 5000
[B$] ssh a2user@localhost -p 5001
[B$] ssh a3user@localhost -p 5002
[B$] ssh a4user@localhost -p 5003
[B$] ssh a5user@localhost -p 5004

Each [A] machine individually reaches out to one [B] machine, opens up reverse ports forwarding to their local SSH port. This allows a user on the [B] box to connect to any [A] box, completely disregarding any normal challenges like no static IPs, no open firewall ports, no dynamic DNS, etc.

[A]  Firewall)   ~~~~~~~WAN~~~~~~~   (Firewall [B] (port 22 open)
Outbound SSH ------------>------------> b.host.com:22

Outbound SSH set to open reverse port forward 5000:localhost:22
localhost:5000 on [B] now leads to localhost:22 on [A] (via tunnel)

[B]  Firewall)   ~~~~~~~WAN~~~~~~~   (Firewall [A] (no ports open)
Outbound SSH ------------>------------> localhost:5000 (via tunnel)

The tunnel needs to stay up 24/7, needs to heal itself when it goes down, and needs to terminate itself in the event of port forward failure. This is required so that systemd can automatically restart it.

Below, there is a reference systemd service file.

#Template systemd service for a auto healing (reverse) SSH tunnel.
#/etc/systemd/system/ssh-callhome.service
#Edit the ports, remote user+host, and the key file.
#In the authorized_keys file on the other system, use something like this:
#<restrict,command="echo 'No Commands Allowed.'",port-forwarding,permitopen="localhost:22",permitlisten="localhost:5000">

[Unit]
Description=SSH Callhome
After=network-online.target

[Service]
ExecStart=/usr/bin/ssh -T -N -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -R <5000:localhost:22> <user@host.com> -o IdentityFile=</root/ssh.key>
ExecStop=/bin/kill -HUP $MAINPID
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=60

[Install]
WantedBy=multi-user.target

For a detailed explanation of the SSH options, see the original post (at the top of this post). This systemd service file will start a SSH client session with those specified options, which are made specifically to forward a remote port. The options will not allow a console session to start, and it will automatically kill the service if the remote server fails to respond or if the port forwarding fails. The systemd portion will automatically try to start the tunnel again in 60 seconds.

There are three components to this:

  • systemd service on the [A] machine
  • SSH key on the [A] machine
  • authorized_keys file on the [B] machine

Setup steps:

  1. Already have a working SSH key authentication configuration, and be able to SSH from [A] machine to [B] machine with the key to a specific user you want to use
  2. Create file /etc/systemd/system/ssh-callhome.service
    • This file goes on the [A] system (in my example), the one initiating the service
  3. Edit the following parts
    • <5000:localhost:22>
      • Remove brackets, change the 5000 if needed (you cannot use this port more than once on the same host)
      • I am using localhost:22 because I need access to port 22 on the [A] machine (for SSH)
    • <user@host.com>
      • Remove brackets, this is the username and hostname (or IP) of the [B] machine
        • (If the [B] machine listens for SSH on another port, you can use -p xyy right after this)
    • IdentityFile=</root/ssh.key>
      • Remove brackets, this is the location of the private key for the [B] box
  4. On the [B] machine, under the user you specified earlier, there should be a ~/.ssh/authorized_keys file (prerequisite for using SSH keys, not covered here)
    • At the top of the main code block, there is another line
    • Add this right before your key on the same line (before the ssh-rsa part)
      • restrict,command="echo 'No Commands Allowed.'",port-forwarding,permitopen="localhost:22",permitlisten="localhost:5000"
    • Only change the permitlisten port to match the one you used earlier, the permitopen is a security restriction so no other local port forwards can be used
  5. On the [A] machine, run the following commands to enable, start, and check the new service
    • sudo systemctl daemon-reload
    • sudo systemctl enable ssh-callhome.service
    • sudo systemctl start ssh-callhome.service
    • systemctl status ssh-callhome.service

You should have started with a existing SSH key authentication configuration between the [A] -> [B] machines. On the [A] machine you add the systemd service, specify the host – reverse port – and the key file. On the [B] machine you add the restrictions to the authorized_keys file so the key cannot be used for anything but opening this port.

If you followed this guide and used the examples you should be able to run the command on the [B] machine: ssh user@localhost -p 5000 (use the user for [A] machine)

If it all worked you should be able to ssh from [B] -> [A] via the localhost reverse port. This will bypass the [A] firewall, and it should re-connect if the WAN goes down.