Directory Upload

Example task yamls for cloud storage

This example also demonstrates inline env variable use with ${TEST_FILE}.

Prerequisites

Config.yaml (~/.konduktor/config.yaml)

allowed_clouds:
  - {s3, gs}

Setup

1. Set up cloud storage credentials
2. Check cloud storage setup with:

$ konduktor check {gs, s3}

For more details, check out the setup of cloud storage here

.

Current Working Directory

$ ls
task.yaml   static_dir/a.txt   test_dir/b.txt

Launching

$ konduktor launch --env TEST_DIR=test_dir task.yaml

Task.yaml

name: directory-upload

file_mounts:
  # `file_mount` paths are 
  # <remote_path>: <local_path>
  ~/static_dir: ./static_dir
  ~/${TEST_DIR}: ./${TEST_DIR}

resources:
  cpus: 1
  memory: 1
  image_id: ubuntu
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    maxRunDurationSeconds: "600"

run: |
  if [ -f ~/static_dir/a.txt ]; then
    echo "File exists"
  else
    echo "File does not exist"
    exit 1
  fi
  if [ -f ~/${TEST_DIR}/b.txt ]; then
    echo "File exists"
  else
    echo "File does not exist"
    exit 1
  fi