A common issue we encounter while working on deployed Anvil! clusters, and Linux systems in general, is reliable and secure access. Whether it is access to high latency deployments in remote areas like cargo freighters, navigating client firewall policies, or access to specific tools within the remote site, being able to quickly assist with critical issues depends on ready access including backup methods if main routes fail. A primary way to access client sites is via remote desktop access. In the context of Linux, VNC is a common tool, however, it isn’t the most performant method of access. VNC and other remote desktop solutions are also dependent on the target system operating as expected; if your endpoint or its applications are not working, VNC won’t help. If your goal is to access networked services (such as LOM interfaces or other troubleshooting tools) via the endpoint, this can be an unnecessary roadblock. Enter SOCKS proxies via SSH.
Proxy vs. Tunnel
Proxying over SSH differs from a tunnel in that it allows for broader network resolution beyond the tunnel endpoint1. Where a tunnel links a local port to a port on the destination, a SOCKS proxy creates a tunnel, then asks the endpoint to make further network requests on your local host’s account. This can be extremely useful in situations where your desired endpoint doesn’t listen to ssh requests but your primary access to the site is ssh-dependent, or where you need to appear to be making network requests from a specific IP space. Put differently; a SOCKS proxy over SSH is a quick and dirty VPN solution for when a more comprehensive solution (like my own favourite VPN, wireguard) is overkill.
SOCKS and You
It is extremely simple to start a SOCKS connection over SSH:
# ssh -D $port user@destination
Pretty straightforward. The -D flag asks our ssh client to create a SOCKS proxy as user at the destination system. Our port could be anything we know is unused on both ends, though by convention, port 1080 is used by SOCKS servers.
Once initiated, we can then set our system or web browser to use the port specified above on our localhost as our proxy location. In firefox, this can be set as below, choosing 127.0.0.1 (localhost) and the port set above as our SOCKS host.
From there, we can navigate as if we were on the destination network. In the context of an Anvil! deployment, if we proxy into the Striker dashboard we would gain access to its management interface, the IPMI interfaces, all of the power and networking stack, as well as virsh connections to the nodes themselves: very useful! The only thing to remember is that we are not the destination host, so localhost connections will still point to our own machine. Use the IP of the destination instead for anything hosted on the endpoint.
Digging Deeper: Multi-hop SOCKS proxies using SSH port forwarding
In a previous post, I covered SSH tunnelling and its myriad uses. SOCKS proxies can also be tunnelled over multiple hops!
In doing so, it’s important to note that only the final hop needs to use the SOCKS protocol. Up until that hop, we can tunnel as normal. For example, if we have a single midpoint we could go:
Localhost -> SSH Tunnel -> Midpoint -> SOCKS Proxy -> Endpoint
In practice, this would look like the following series of commands:
# ssh -L $ourPort:localhost:$ProxyPort user@midpoint
# ssh -D $ProxyPort user@endpoint
Then we would set our SOCKS host port to whatever we chose for $ourPort. Any number of midpoints could be added this way. For instance,
# ssh -L 8080:localhost:10001 user@midpoint1
# ssh -L 10001:localhost:1080 user@midpoint2
# ssh -D 1080 user@endpoint
would create a tunnel from port 8080 on our system to port 10001 on midpoint1, then from 10001 on midpoint1 to 1080 on midpoint2, and finally connect 1080 to the endpoint as a SOCKS connection. Instant multi-hop VPN!
Naturally, there are way more uses for a proxy than just accessing HTTP/S content, but if you need to access an intranet for emergency maintenance, it’s hard to beat the speed of an SSH proxy.
Happy networking!
Sources:
Special thanks to tatanus2 for their quick how-to on GitHub which inspired a client solution, and the seed that sprouted this article.
1 https://ma.ttias.be/socks-proxy-linux-ssh-bypass-content-filters
2 https://gist.github.com/tatanus/da204a0529a96ffc007d
by Alex Bruneau