Tailscale Interactive SSH (Experimental)

Connect to your workloads via Tailscale SSH for interactive development and debugging

Experimental features are new and their interface and implementation may change at any time. Expect sharp edges .

Tailscale is a mesh network VPN Service for P2P private network communication with a generous free tier and affordable starter tiers. Konduktor supports direct ssh sessions into workloads over a tailnet which allows for interactive shells in your workload environments via a userspace networking tunnel. This let’s you connect tools like:

• SSH
• VSCode
• Jupyter Notebooks

Requirements

Tailscale ACLs

Tailscale ACLs need to be set so that the development workstation launching jobs has access to the Trainy workers which will join the Tailnet upon startup. Below is an example ACL where we create a tag tag:trainyfor our jobs and give ownership of that tag to a single user [e-mail] and group group:trainy_user.

Example ACL

{
	// Create a single group `group:trainy_user` with a single user `[e-mail]`
	"groups": {
		"group:trainy_user": [
			"[e-mail]",
		],
	},

	// Define access control lists for users, groups, autogroups, tags,
	// Tailscale IP addresses, and subnet ranges.
	"acls": [
		{
			"action": "accept",
			"src":    ["group:trainy_user"],
			"dst":    ["tag:trainy:*"],
		},
		{
			"action": "accept",
			"src":    ["tag:trainy"],
			"dst":    ["tag:trainy:*"],
		},
	],

	// Define users and devices that can use Tailscale SSH.
	"ssh": [
		// Allow all users to SSH into their own devices in check mode.
		// Comment this section out if you want to define specific restrictions.
		{
			"action": "accept",
			"src":    ["group:trainy_user"],
			"dst":    ["tag:trainy"],
			"users":  ["autogroup:nonroot", "root"],
		},
		{
			"action": "accept",
			"src":    ["tag:trainy"],
			"dst":    ["tag:trainy"],
			"users":  ["autogroup:nonroot", "root"],
		},
	],

	"tests": [
		{
			"src":    "tag:trainy",
			"accept": ["tag:trainy:22"],
		},
	],
	"tagOwners": {
		"tag:ci-test": ["group:trainy_user"],
	},
}

Create/Set Authkey

We recommend creating a ephemeral, reusable, tagged auth key. The tag for the authkey should match that of the ACL you set. In the example above, it was tag:trainy. This will allow multiple workers and jobs to be authenticated while allowing them to be pruned from your tailnet automatically upon completion. Upon creating an auth key, you can store it as a secret to be used in your clusters via

kubectl apply -f tailscale_authkey.yaml

tailscale_authkey.yaml

# tailscale_authkey.yaml
apiVersion: v1
kind: Secret
metadata:
  name: tailscale-auth
stringData:
  TS_AUTHKEY: tskey-abcdef1432341818

Afterwards, to toggle running tailscale on your jobs set the following in your ~/.konduktor/config.yaml.

# ~/.konduktor/config.yaml
tailscale:
  secret_name: tailscale-auth

Tailscale auth keys have a maximum lifetime of 90 days. Jobs already connected to the tailnet won’t disconnect after expiry, but new jobs won’t be able to connect to the Tailnet. We recommend generating a long-lived OAuth key and using the get-authkey utility to that new auth keys can be minted easily and updated by reapplying kubectl apply -f tailscale_authkey.yaml without having to go to the admin dashboard to remint a new auth key every 90 days. Note: we are working on authenticating via OAuth key directly so users don’t have to worry about updating their keys. If you are interested in this, please let us know!

SSH into a dev workload

For a development workload, we can define a job that idles as in the following sleep.yaml

sleep.yaml

name: sleep 

num_nodes: 2

resources:
  cpus: 1
  memory: 2
  image_id: ubuntu
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    maxRunDurationSeconds: "3200" # set this to a large number

run: |
	trap : TERM INT; sleep infinity & wait

after which we can launch with

$ konduktor launch -y --detach-run sleep.yaml

After our workload is ACTIVE, it should appear in our tailnet after a few seconds.

# check that our job has connected to tailnet
$ tailscale status
100.65.130.5    andrews-macbook-air  asai@        macOS   -
100.95.231.96   sleep-fc25-workers-0-0 tagged-devices linux   -
100.114.39.43   sleep-fc25-workers-0-1 tagged-devices linux   -

# ssh into worker
$ ssh root@sleep-fc25-workers-0-0
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-1015-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@sleep-fc25-workers-0-0:~#