Persistent SSH Tunnels

April 04, 2019

There are various uses for having a persistent SSH tunnel, similar to why you’d want a VPN but less complicated since network routing is not in the scope of these tunnels.

This tutorial will cover how to use the tool autossh along with a systemd service to keep it running. Examples will be for CentOS, however only package installation will be different.

Prerequisites:

EPEL repository installed (covered below)
Password-less ssh authentication between two machines (not covered here)

Goals:

Create persistent SSH connection from one machine to another
This connection will include a remote port forward
Machine A (remote) will be able to open a remote listen port on machine B (local)

Practical Application:

There are many, however the one we are focusing on here is the ability to access a remote socket on machine A from a local port on machine B. One of the most common reasons for this would be a “poor mans VPN”.

Specifically, this guide will cover a remote machine [A], which is behind a firewall which we cannot open, and a local machine [B] which we can open the firewall on. [A] will start a ssh tunnel with [B], and open a remote listen port on [B] that becomes the normal ssh port back on [A].

[A] ssh -> [B], opens remote port 5001 which leads back to [A]’s port 22.

[A]  Firewall)   ~~~~~~~WAN~~~~~~~   (Firewall [B] (port 22 open)
Outbound SSH ------------>------------> b.host.com:22

Outbound SSH set to open reverse port forward 5000:localhost:22
localhost:5011 on [B] now leads to localhost:22 on [A] (via tunnel)

[B]  Firewall)   ~~~~~~~WAN~~~~~~~   (Firewall [A] (no ports open)
Outbound SSH ------------>------------> localhost:5011 (via tunnel)

Now a user on [B] can ssh to localhost:5001, and that will actually take them to [A]’s ssh socket. No firewalls need to be opened from [A].

——————– Important ——————–

The machine that initiates [A] the SSH needs to have all of these steps done, the machine on the receiving end [B] only needs an SSH key set up (which we don’t cover here).

Installation is pretty easy, you’ll need the epel-release repository if its not already installed.

$ sudo yum install epel-release
$ sudo yum install autossh

There are two ways to do this, and I’m going to briefly explain both. Either most of your configuration is done via the systemd service file, or via the ssh config file. I find the ssh config file to be easier to update and I’m used to making most of my changes there so that’s how this guide will do it.

Time to create the systemd service, I’m calling mine callhome-ssh.service.

$ sudo vim /etc/systemd/system/callhome-ssh.service
-----------------------------------------------------
[Unit]
Description=Service for autossh
After=network.target

[Service]
ExecStart=/usr/bin/autossh -M 0 -t -N b.hostname.com
ExecStop=/bin/kill -HUP $MAINPID
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=60

[Install]
WantedBy=multi-user.target
-----------------------------------------------------
$ sudo systemctl daemon-reload
$ sudo systemctl enable callhome-ssh.service

A few things to note here:

After=network.target
- This tells systemd to only start this service after the network system has come up
ExecStart=/usr/bin/autossh -M 0 -t -N b.hostname.com
- This is the command that will be executed
- -M 0 tells autossh to disable its monitoring feature (we will compensate for that)
- -t -N are passed to SSH, these stop the creation of a terminal session (we will be blocking the terminal later)
- host.com, this is your remote hostname
ExecStop=/bin/kill -HUP $MAINPID / ExecReload=/bin/kill -HUP $MAINPID
- These tell systemd how to stop and reload the process
Restart=on-failure
- This tells systemd to restart this process if it fails
RestartSec=60
- The default systemd restart time is 100ms, this tries to restart the service every 60 seconds

Now we will configure the actual SSH part. We are going to use the root user, and I am assuming that key files have already been generated for both boxes.

$ sudo su
$ mkdir -p /root/.ssh
$ chmod 700 /root/.ssh
$ vim /root/.ssh/config
-----------------------------------------------------
Host b.hostname.com
    HostName      b.hostname.com
    User          mfox
    Port          22
    IdentityFile  ~/.ssh/keys/b.hostname.com.key
    RemoteForward  5001 localhost:22
    ServerAliveInterval 30
    ServerAliveCountMax 3
    ExitOnForwardFailure=yes

Lets explain a few things:

The first three commands make the .ssh folder and set its permissions, even if it exists already this wont break anything
Host b.hostname.com
- This is the host that will be referenced from the ssh command – it can match the actual hostname, or it can be a short nickname instead
HostName b.hostname.com
- The actual hostname or IP for the remote box [B]
User mfox
- Username for SSH to use on the remote machine [B]
Port 22
- Port on the remote machine [B] to use for SSHing to
IdentityFile ~/.ssh/keys/b.hostname.com.key
- This references the key file for SSH to use, not covered here
RemoteForward 5001 localhost:22
- Tells SSH to open remote port 5001 [B], and map it to local port 22 [A]
ServerAliveInterval 30
- This tells SSH to send a keep-alive packet over the tunnel so it dosn’t timeout
ServerAliveCountMax 3
- This is the amount of times that the keep-alive packet is sent, if no response after three then the session is closed
ExitOnForwardFailure=yes
- Session is closed if the port forwarding fails

I feel like these options need further context, just so you know exactly why we are doing it this way. Technically the ServerAliveCountMax isn’t needed because the default is 3 already, but I felt like it was good to keep the option in there in case it needed to be changed.

Other then the port settings, the last three options are crucial to keeping a steady connection. The ServerAlive options make sure that the SSH tunnel is talking back and forth. If its not talking, we want the tunnel to close ASAP so it can be re-established. If there are any errors with port forwarding, either a actual problem or perhaps another session is open already holding these ports, we also want the tunnel to close. It’s important to keep the tunnel up as long as possible, which the ServerAlive does, but also to close it as soon as possible if its not working, which is also accomplished by ServerAlive.

You should be able to test SSH now by just trying the following command:

$ ssh b.hostname.com

The prerequisite was to already have a working password-less authentication set up so this should have worked before and it should still work

Optional step – securing the remote key if its not trusted. Again this is assuming you already have the authorized_keys file configured on [B] – which is not covered here, you just need to append this to the key you’ll be using:

command="echo Key for port forwarding only",restrict,port-forwarding

Adding this to your authorized key will stop anyone trying to use the key to get regular command line access back to your machine. You should not be using this key for regular access, as it will no longer work once you add that. Be sure to test this and make sure the key doesn’t work for normal SSH sessions. Again this is not needed if you trust the remote machine where your key will sit.

At this point you should be able to start the systemd service on [A] and see if it works:

[A]$ sudo systemctl start callhome-ssh.service
[A]$ sudo systemctl status callhome-ssh.service
-------------
[B]$ ssh user@localhost -p 5001

Troubleshooting:

[A]$ sudo journalctl -u callhome-ssh.service
-----or-----
[B]$ sudo tail /var/log/secure

Those two logs should give you some insight into what happened.

It seems like a lot of steps and looks confusing, but it’s really not. If you boil this down, it condenses to this:

SSH key files created and put on each box
autossh installation
Systemd service file creation
SSH config file creation
Edit working SSH key file (optional)
Testing