Environment Variables

Complete guide to all environment variables available in Konduktor containers and how to use them.

Konduktor automatically provides a rich set of environment variables in your containers that give you access to job metadata, networking information, and system configuration. These variables are automatically set when your container starts and can be accessed using standard shell syntax.

Setting Environment Variables

Environment variables in Konduktor can come from multiple sources, depending on how your workload and configuration are defined. When the same variable name is defined in more than one place, Konduktor applies a priority order to decide which value takes precedence.

Priority Order

If the same variable appears in multiple places, the highest priority source wins. Priorities from highest to lowest:

Priority	Source	Description
1	CLI	Variables passed directly via the CLI with konduktor launch
2	Task (task.yaml)	Variables defined under the envs: block of a task.yaml
3	Environment Secrets (kind=env)	Variables created using konduktor secret create --kind=env
4	Config (~/.konduktor/config.yaml)	Variables defined globally in the Konduktor configuration file at ~/.konduktor/config.yaml
5	Other/System Defaults	Variables automatically generated by Konduktor (ex. KONDUKTOR_JOB_NAME, NUM_NODES, etc.).

While the global priority is CLI > task.yaml > env secret > config.yaml > system, note that any exports (ex. export FOO=bar) done inside your run: script happens at runtime and will override the effective value for all subsequent commands in that script. So basically runtime > everything.

Examples

CLI

$ konduktor launch --env FOO=bar task.yaml

$ export FOO="bar" && konduktor launch --env FOO task.yaml

task.yaml

envs:
  FOO: bar

Env Secret

$ konduktor secret create --kind=env --inline FOO=bar my-env-secret-name

config.yaml

kubernetes:
  pod_config:
    spec:
      containers:
        - name: konduktor-container
          env:
            - name: FOO
              value: bar

Accessing Environment Variables

Environment variables can be accessed in several ways:

Shell Scripts

Task.yaml:

run: |
    # List all env vars
    env

    # Using $VAR syntax (recommended)
    echo "Job name: $KONDUKTOR_JOB_NAME"
    echo "Number of nodes: $NUM_NODES"

    # Using ${VAR} syntax (useful for concatenation)
    echo "Master address: ${MASTER_ADDR}"
    echo "Local IP: ${LOCAL_ADDR}"

    # Using ${VAR:-default} for fallback values
    echo "GPU count: ${NUM_GPUS_PER_NODE:-0}"

Python

Task.yaml:

run: |
    python pythonfile.py

pythonfile.py:

import os

# Get environment variables
job_name = os.environ.get('KONDUKTOR_JOB_NAME', 'unknown')
num_nodes = int(os.environ.get('NUM_NODES', '1'))
gpu_count = int(os.environ.get('NUM_GPUS_PER_NODE', '0'))

print(f"Running job: {job_name}")
print(f"Nodes: {num_nodes}")
print(f"GPUs per node: {gpu_count}")

Node.js

Task.yaml:

run: |
    node jsfile.js

jsfile.js:

// Get environment variables
const jobName = process.env.KONDUKTOR_JOB_NAME || 'unknown';
const numNodes = parseInt(process.env.NUM_NODES) || 1;
const gpuCount = parseInt(process.env.NUM_GPUS_PER_NODE) || 0;

console.log(`Running job: ${jobName}`);
console.log(`Nodes: ${numNodes}`);
console.log(`GPUs per node: ${gpuCount}`);

Core Konduktor Environment Variables

These variables are always available in every Konduktor container:

Job Information

Variable	Description	Example Value
KONDUKTOR_JOB_NAME	Unique identifier for the current job	my-training-job-a1b2
NUM_NODES	Total number of nodes in the job	2
NUM_GPUS_PER_NODE	Number of GPUs allocated per node	8

Networking Information

Variable	Description	Example Value
NODE_HOST_IPS	Comma-separated list of all node hostnames	job-123-workers-0-0.job-123,job-123-workers-0-1.job-123
MASTER_ADDR	Hostname of the master/rank 0 node	job-123-workers-0-0.job-123
LOCAL_ADDR	Pod’s internal IP address	10.104.2.17
RANK	Current node’s rank (0 for master, 1+ for workers)	0

System Configuration

Variable	Description	Example Value
PYTHONUNBUFFERED	Python output buffering setting	0
JOB_COMPLETION_INDEX	Kubernetes job completion index	0
RESTART_ATTEMPT	Job restart attempt number (0 for first attempt, 1+ for retries)	0

Conditional Environment Variables

These variables are only set when specific features are enabled:

Tailscale (when tailscale.secret_name is configured)

Variable	Description	Example Value
TS_USERSPACE	Tailscale userspace networking flag	true
TS_AUTHKEY	Tailscale authentication key	tskey-auth-...
POD_NAME	Kubernetes pod name	job-123-workers-0-0-abc123
POD_UID	Kubernetes pod UID	a1b2c3d4-e5f6-7890-abcd-ef1234567890

SSH (when ssh.enable is true)

Variable	Description	Example Value
KONDUKTOR_SSHPUB	Public SSH key for the job	ssh-rsa AAAAB3NzaC1...
KONDUKTOR_SSHPRIV	Private SSH key for the job	-----BEGIN OPENSSH PRIVATE KEY-----
KONDUKTOR_SSH_PORT	SSH port number	2222

Git SSH (when git-ssh secret exists)

Variable	Description	Value
GIT_SSH_COMMAND	SSH command for Git operations	ssh -i /run/konduktor/git-ssh-secret/gitkey -o StrictHostKeyChecking=no

Default Secrets (when default secrets exist)

Variable	Description	Value
KONDUKTOR_DEFAULT_SECRETS	Path to mounted default secrets	/konduktor/default-secrets

Practical Examples

WANDB Integration

import os
import wandb

# Automatically name your WANDB run after the Konduktor job
job_name = os.environ['KONDUKTOR_JOB_NAME']
num_nodes = int(os.environ['NUM_NODES'])
gpu_count = int(os.environ['NUM_GPUS_PER_NODE'])

wandb.init(
    project="my-training-project",
    name=job_name,
    config={
        "nodes": num_nodes,
        "gpus_per_node": gpu_count,
        "total_gpus": num_nodes * gpu_count
    }
)

print(f"WANDB run started: {job_name}")

Troubleshooting

If an environment variable is not set as expected:

1. Check if the feature is enabled (e.g., SSH, Tailscale)
2. Verify the variable name spelling
3. Check the Konduktor logs for any errors
4. Ensure you’re running the latest version of Konduktor