File Sync

Synchronize files to run your workload.

Users have the following options for submitting their application code as a Trainy workload.

• (Re)building a docker image for every change and pushing it to a registry for launching a workload.
• Committing changes to a development branch in git and checking it out in the workload
• Synchronizing application code through file sync via file_mounts and workdir definitions.

The first option is often slow given the size of deep learning images so we focus on the the latter two here.

Setup

We first need to decide which storage provider we are going to use. First, we have to place our cloud service account credentials into the Trainy cluster.

konduktor check <CLOUD STORAGE ALIAS> # s3, gs, etc.

afterwards we configure the storage provider by setting ~/.konduktor/config.yaml

# ~/.konduktor/config.yaml
allowed_clouds:
  - gs

Usage

When we run konduktor launch two things happen atomically in this order. If any step fails, the workload will fast-fail.

1. workdir and file_mounts are synchronized to object storage
2. workload is submitted
3. workload, once active, will sync down workdir and file_mounts

In our workload definition, we can define the following:

name: single-file-upload

num_nodes: 1

workdir: .

file_mounts:
  # syntax is <remote_dir>:<local_dir>
  ~/test_dir: ./test_dir
  # syntax is <remote_file>:<local_file>
  ~/static_path.txt: ./static_path.txt

resources:
  cpus: 1
  memory: 1
  image_id: ubuntu
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    maxRunDurationSeconds: "600"

run: |
  ls -lah
  ls -lah ~/

Cloning private GitHub Repositories.

Cloning private repositories is supported via both file sync of ssh keys to your object store or through secrets. This section demonstrates how to file sync an ssh key from our workstation onto the workload and configure SSH for pulling from a private repository.

name: private-repo-ssh

num_nodes: 1

resources:
  cpus: 1
  memory: 2
  image_id: ubuntu
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    maxRunDurationSeconds: "3200"


file_mounts:
  ~/.ssh/test-ssh-key: ./tests/secrets/test-ssh-key

run: |
  set -eux
  apt-get update && apt-get install -y git openssh-client

  if [[ -f ~/.ssh/test-ssh-key && -s ~/.ssh/test-ssh-key ]]; then
    echo "SSH key mounted and non-empty"
  else
    echo "SSH key missing or empty"
    exit 1
  fi

  chmod 600 ~/.ssh/test-ssh-key
  echo -e "Host github.com\n\tIdentityFile ~/.ssh/test-ssh-key\n\tStrictHostKeyChecking no\n" > ~/.ssh/config

  git clone [e-mail]:mygithubaccount/My-App.git

Assistant

Responses are generated using AI and may contain mistakes.