This article discusses how to set up SSH tunnels to run the HashiCorp Nomad CLI remotely from a Windows desktop.

I recently built a single node HashiCorp Nomad cluster in dev mode on IBM Cloud. This is a single Virtual Server Instance with Ubuntu 18.04 installed. I then set up Nomad and Consul. The security groups for this server only allow traffic from my network to the VM. I then set up two SSH tunnels so that the Nomad CLI works from my desktop.

An added benefit is that the Nomad GUI also works over an SSH tunnel.

Nomad Network Details

Nomad requires three ports to work properly for Nomad Servers and two ports for Nomad Clients.

  • Port 4646. HTTP. This port is used by clients and servers to serve the HTTP API. Only TCP is used.
  • Port 4647. RPC. This port is used for internal RPC communications between client agents and servers and for inter-server traffic. Only TCP is used.
  • Port 4648. Serf WAN. This port is used by servers to gossip with other servers. This port is not used by clients. Both TCP and UDP.

To run the Nomad CLI, access to ports 4646 and 4647 over TCP is required.

SSH Port Tunneling – Local Port Forwarding

SSH local port forwarding lets you connect to a local address and have the network traffic forwarded to another system. A local port is mapped to a remote address and port. The remote address can be a DNS hostname or IP address. The network traffic is secure and encrypted by the SSH tunnel that is created.

Why?

When Nomad is run in developer mode, the only network ports opened are for localhost. This prevents Nomad from being accessed over the network and from the Internet. To access Nomad remotely in developer mode either requires a proxy or a TCP tunnel. In this article, I will show how to use TCP tunnels to forward traffic securely to the system running Nomad. In another article, I will show how to use an Nginx proxy.

Is this Secure?

Yes. Provided that you have the SSH Server configured correctly and implement good practices for managing usernames, passwords, and SSH keys, TCP forwarding is secure. The network traffic travels over an SSH tunnel which is encrypted. Only those with the correct username and password or SSH private key can connect to the SSH server to set up tunneling.

In this article, I assume that you are just getting started with Nomad and Consul. You are the only person accessing the Nomad server, which is running in developer mode. This means your Nomad server is running both the client and the server. For Nomad production deployments, review Nomad Security to set up TLS, ACLs, and Secrets.

I want to disable SSH Port Forwarding

For secure environments, SSH port forwarding should be disabled. The exact options are SSH server dependent. For the SSH server installed with IBM Cloud Ubuntu (and most installations), you can control this thru several options. Also, consider the situation where a user logged into an instance can launch another SSH server instance with port forwarding enabled. Block that situation by controlling access permissions to the executable and the ports opened in the security groups.

The simplest method to disable port forwarding is thru the AllowTcpForwarding option in /etc/ssh/sshd_config. Set this option to no. Another option is PermitTunnel.

Also, consider PermitOpen and authorized_keys to provide more granular control.

Review the documentation for your SSH server.

Requirements

  • Windows 10 Professional
    • Other Windows versions might work.
  • Windows OpenSSH
    • Microsoft’s version of SSH is free.
    • Other SSH programs might work.
    • To install Windows OpenSSH, read this document.
    • The OpenSSH program path is:
      • c:\windows\system32\OpenSSH\ssh.exe
    • I recommend adding the following directory to the environment PATH variable.
      • c:\windows\system32\OpenSSH
    • IP Address for your VM running Nomad
    • Port 22 open in the VM Security group.
    • The full path to your SSH private key or the username and password.
  • Nomad CLI
    • Download the Nomad Windows CLI from this link.
    • Unzip the download.
    • Extract nomad.exe.
    • Copy nomad.exe to a directory in your environment PATH or create a new directory and add the directory to your environment PATH variable.
  • SSH TCP Forwarding enabled in the SSH Server

Step 1: Open Command Prompts

Open two Windows Command Prompts.

Step 2: Setup the Tunnel for Port 4646

Execute this command in the first command prompt. This command will start an SSH connection, open a shell terminal window, create a tunnel, and open port 4646 on your desktop. Connections made to port 4646 will be forwarded to hostname on port 4646. This command leaves the SSH shell terminal window open and you can execute normal Linux commands to the Linux server. When you exit the shell terminal window, the tunnel closes.

This example requires a username and password.

Replace username with your username. Replace hostname with the IP address with your VM.

This example requires an SSH private key.

Step 3: Setup the Tunnel for Port 4647

Execute this command in the second command prompt. This command will start an SSH connection, open a shell terminal window, create a tunnel, and open port 4647 on your desktop. Connections made to port 4647 will be forwarded to hostname on port 4647. This command leaves the SSH shell terminal window open and you can execute normal Linux commands to the Linux server. When you exit the shell terminal window, the tunnel closes.

This example requires a username and password.

Replace username with your username. Replace hostname with the IP address with your VM.

This example requires an SSH private key.

Step 4: Execute the Nomad CLI

Get the status of Nomad nodes:

The output will be similar to this:

Get the status of running Nomad jobs:

The output will be similar to this:

Step 5 – Open a Browser to the Nomad GUI

The Nomad GUI is served over port 4646. Open a web browser to http://localhost:4646/.

Summary

This article discussed the Nomad CLI, networking requirements to run the CLI remotely and how to set up SSH tunnels to forward IP Ports.

With the setup provided in this article, you can now check on the status of Nomad, the status of Nomad jobs and run Nomad jobs remotely from your desktop. You can also use the Nomad GUI securely from your desktop without exposing the Nomad GUI to the Internet.

You can also extend this setup to create another SSH tunnel for the Consul GUI which is accessed over port 8500. Fabio Web UI is port 9998 and the Fabio Load Balancer is port 9999. Traefik Web UI is port 8181 and the Traefik Load Balancer is port 8180.

In another article, I will show how to set up Nginx running on the VM to proxy connections from the network to Nomad services.

More Information

  1. HashiCorp Nomad
  2. SSH Port Forwarding
  3. OpenSSH Server Configuration
  4. Microsoft Windows OpenSSH
  5. Nomad Commands CLI
  6. HashiCorp Consul
  7. Nomad Security
  8. Fabio Load Balancer
  9. Traefik Load Balancer

Credits

I write free articles about technology. Recently, I learned about Pexels.com which provides free images. The image in this article is courtesy of Pixabay at Pexels.

Related Posts