home June 15, 2021

HAProxy tunnel SSH through HTTPS


Creating an encrypted, HTTP/2 TLSv1.3 Tunnel for SSH

Due to draconian ideology, some organizations block outgoing ssh (port 22) to the internet and their network monitoring appliances complain when ssh is used on alternate ports. One solution is to create a secure tunnel with a commonly acceptable encrypted protocol like https. We can configure OpenSSH on the client side and HAProxy on the remote server to allow ssh to tunnel through an encrypted https connection to the remote sshd server.

Advantages of tunneling ssh through https include:

NOTE: We are going to assume the following conditions are true:

Install HAProxy on the home machine

Use your OS's package manager to install HAProxy on the home machine; the size of the install is only a few megabytes.

Install HAProxy

# FreeBSD 13 
pkg install haproxy-2.3.10

# Ubuntu
apt install haproxy

# Redhat or CentOS
yum install haproxy

HAProxy Configuration

The location of the haproxy.conf file will vary upon your OS. Search for your haproxy.conf file using "find / -name haproxy.conf" or check the HAProxy man page for details. We are using FreeBSD 12, so the default location is /usr/local/etc/haproxy.conf . Copy and paste the following to the haproxy.conf file.

Our HAProxy server uses the strongest, most strict https settings. The following config only accepts the binary HTTP/2 protocol, TLSv1.3 encryption and the CHACHA20_POLY1305 cipher. HAProxy also requires clients to send the correct domain name when connecting which is enforced by the "strict-sni" directive. All other SSL/TLS connections are denied. Because we are using TLSv1.3 make sure that your version of HAProxy is built against OpenSSL v1.1.1 or later. You can check your build options with "haproxy -vv" and look for the lines starting with, "Built with OpenSSL version".

Pay special attention to the "bind" line. In the example, HAProxy is bound to port 443 on all network interfaces, but you can specify a single ip as well. You can also define any port, but 443 and 8443 are commonly used for encrypted https communication so network monitors will not see encrypted https traffic on those ports as suspicious. We have included two(2) commented access control lines (ACLs) which can limit the source ip addresses allowed to connect to HAProxy. Any ip not in the ACL whitelist is rejected and the connection immediately closed.

Following this section we will talk about defining your SSL Certificate from a public Certificate Authority like Let's Encrypt.

#
## moneyslow.com - /usr/local/etc/haproxy.conf
#          
## ssh through an https tunnel , TLSv1.3 , HTTP/2
#

global
   daemon
   user  daemon
   group daemon
   chroot /var/empty
   ssl-default-bind-ciphersuites TLS_CHACHA20_POLY1305_SHA256
   ssl-default-bind-options force-tlsv13

frontend https
   bind *:443 ssl alpn h2 strict-sni crt /USER/.acme.sh/example.com_ecc/example.com.haproxy
   default_backend sshd
   mode tcp
   option tcplog
   tcp-request inspect-delay 5s
   timeout client 1h
   #
   # ACL - ssh ip whitelist
  #acl network_allowed src 11.22.33.44 192.168.0.10
  #tcp-request connection silent-drop if !network_allowed
   #
   # ACL - verify ssh banner
  #acl ssh_banner req.payload(0,7)  -m str "SSH-2.0"
	#  --OR--
  #acl ssh_banner req.payload(0,17) -m str "SSH-2.0-OpenSSH_8"
  #tcp-request content silent-drop if !ssh_banner

backend sshd
   mode tcp
   server localhost-sshd 127.0.0.1:22
   timeout connect 1h
   timeout server 1h

Hostnames and SSL Certificates

A hostname, like example.com, is needed to retrieve a public SSL certificate from Let's Encrypt. Many companies offer free hostnames if you do not already have one, like No Ip.

In order for an https server to accept an encrypted https connection, HAProxy needs to have an ssl certificate. If you already have an ssl certificate, that will work. If you do not have a certificate you could create a self signed certificate, but we recommend Let's Encrypt who issues valid, public certificates for free.

Using your domain name, like "example.com", take a look at a tool like the ACME Shell script: acme.sh to issue a valid SSL Certificate from Let's Encrypt.

Once you have a valid ssl certificate two(2) files need to be combined for HAProxy; the full certificate chain and the private key in that order.

Combine both the full certificate chain and private key

cat fullchain.cer > example.com.haproxy
cat example.com.key >> example.com.haproxy

After the combined certificate file is created, define the full path to the example.com.haproxy file on the "bind" line after the "crt" directive in haproxy.conf . The ssl certificates can be owned by root with restrictive permissions as HAProxy will read the certificate file as root before switching to the unprivlidged user, daemon.

OpenSSH Client Side Configuration

On the client side, like the machine at work, both the OpenSSH client and OpenSSL should be default installed on Linux and FreeBSD. We are going to assume you do not have permission to edit the main /etc/ssh/ssh_config file normally owned by root. So, create an unprivileged, user defined ssh client config file in your home directory called ~/.ssh/config . Edit the ~/.ssh/config file and add the following Host and ProxyCommand lines. Change the hostname "example.com" to the hostname you registered with the home server.

user@work$  vi ~/.ssh/config

Host example.com
  ProxyCommand openssl s_client -connect example.com:443 -quiet

This ssh client configuration says that when the client tries to ssh to "example.com", ssh will use the ProxyCommand directive to create an https connection to example.com on port 443 using openssl's s_client binary. The OpenSSH client will then tunnel its ssh connection inside of the established https connection.

Initial Testing...

At home, make sure HAProxy is started and listening on the ip address and port you defined. If your home machine is behind a router or firewall then verify the port forwarding configuration of port 443 from work to the internal LAN connected home server. We recommend limiting the ip addresses allowed to connect to port 443. You should be able to define your work source ip address or the work source subnet in the port forwarding settings.

OpenSSL can be used to quickly test if the https connection can be established. Here is an abbreviated example output executed on the work machine. Notice the last line of the output displays the ssh banner of the sshd server at home. In this example, we see OpenSSH version 8.3 on FreeBSD which is correct version of sshd on our home machine.

user@work$  openssl s_client -connect example.com:443

---
read R BLOCK
SSH-2.0-OpenSSH_8.3 FreeBSD-20200715

Making an SSH Connection

On the client side, execute ssh to the remote home server. The ssh client will create the https tunnel using openssl's s_client command to the home machine answering at example.com . After the https connection is established, openssh will tunnel an ssh connection through the https connection to the remote home server. HAProxy on the remote home server will stream proxy the ssh traffic to the sshd daemon listening on localhost, port 22. This is what the output of the ssh client looks like. Notice the "verify return:1" lines confirming the public certificate authority "Let's Encrypt" and a valid https ssl certificate for example.com before we enter our ssh passphrase.

user@work:~$ ssh example.com

depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = example.com
verify return:1

Enter passphrase for key '/home/username/.ssh/id_ed25519':

user@home: 

Finished. The connection from work to home is complete. The full flexibility of ssh can now be used. You can scp or rsync files to and from home, you can setup a SOCK5 tunnel for a web browser, and you can even create a Reverse SSH Tunnel allowing ssh connections initiated from home back into work.

Questions?

Do you have a working HAProxy chroot configuration ?

Yes, check out our tutorial for an HAproxy https in chroot.

How is the https tunnel double encrypted ?

The initial https tunnel created from work to home is encrypted using the ChaCha20 Poly1305 cipher negotiated by the openssl s_client and HAProxy. The tunneled ssh connection uses AES-256 GCM encryption inside the encrypted https tunnel.

How much latency does HAProxy add to an SSH connection ?

HAProxy seems to add 20 milliseconds to 50 milliseconds of latency to the ssh connection. Not a large impediment, but enough of a delay that you will notice the connection is not as responsive as a direct ssh connection on a low latency network.

How can I create an HAProxy http to https redirector ?

HAProxy can be used to redirect all http requests to https. This setup is useful on a dedicated server at the network edge in front of an https only web server farm. HAProxy does not care about the hostname or URL, HAProxy simply redirects all traffic to the https scheme.

#
## moneyslow.com - /usr/local/etc/haproxy.conf
#          
## http to https redirect
#

global
   daemon
   user  daemon
   group daemon
   chroot /var/empty

defaults
   timeout connect 1m
   timeout client  1m
   timeout server  1m

frontend http
    mode http
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }

The following is the output a client will see. No extraneous information like server names or extra headers. HAProxy sends the redirect back to the client and immediately closes the connection.

user@home:~$  curl -kIL http://moneyslow.com

HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://moneyslow.com/html/webconf/