Restrict batch jobs to matching clusters

Use placement labels to select a subset of your organization’s worker clusters.

Add placement.trainy.ai/ labels to a task’s resource labels to restrict its eligible worker clusters. An organization administrator assigns the same labels to the enrolled WorkerCluster objects on the Hub.

name: east-training
resources:
  cpus: 4
  memory: 16
  image_id: ubuntu:24.04
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    placement.trainy.ai/pool: forecasting
    placement.trainy.ai/region: us-east
run: python train.py

Submit it with konduktor launch task.yaml. The Hub considers only clusters with both matching labels. Labels without this prefix do not affect cluster placement. Keys and values are case-sensitive; an empty value requires the key to exist on the cluster.

The job stays queued if no eligible cluster matches. It never falls back to an unmatched cluster. Labels narrow your organization’s existing cluster access; they do not grant access to another organization. Readiness, cordons, quota, resource compatibility and machine0’s single-Pod restriction still apply.

Multiple clusters can share a pool label. For one specific cluster, ask an administrator to assign a unique placement label and use that label on the job. These are cluster filters, so worker Nodes do not need these labels.

Placement labels are fixed when a job is created. Submit a new job to change them. Administrators can relabel workers; the Hub re-evaluates pending nominations every 15 seconds. A job already assigned to a cluster stays there.

Raw Kubernetes batch/v1 Job and jobset.x-k8s.io/v1alpha2 JobSet submissions use these labels on their top-level metadata.labels. Putting them only on a Pod template does not set a cluster restriction. Serving deployments are not supported.

​

Administrator setup

Label enrolled clusters on the Hub, for example:

kubectl -n forecasting label workercluster east-worker \
  placement.trainy.ai/pool=forecasting placement.trainy.ai/region=us-east

The Hub must run the placement-aware controller, Kueue 0.19 and the custom dispatcher from the machine0 dispatch feature:

multiKueue:
  dispatcherName: trainy.ai/machine0-dispatcher

The dispatcher supports ordinary workers with machine0 provisioning disabled. Block restricted submissions while configuring Kueue and restarting its manager; accept them only after the rollout completes. No integrations.labelKeysToCopy entries are needed: the Hub reads the exact owning Job or JobSet, including its UID.

The config/gke-hub overlay pins controller image sha-84a4e58720ed11f44d243bc8657360d40c403691 and leaves placement admission as an explicit rollout step. First promote a published image that serves /validate-cluster-placement. Use the target Hub’s Kubernetes context, then wait for that image to finish its rollout and install the webhook:

kubectl -n trainy-controller-system rollout status \
	deployment/trainy-controller-controller-manager --timeout=300s
kubectl apply -f trainy-controller/config/gke-hub/cluster-placement-webhook.yaml
kubectl wait --for=jsonpath='{.webhooks[0].clientConfig.caBundle}' \
	validatingwebhookconfiguration/trainy-controller-cluster-placement --timeout=120s

The manifest keeps failurePolicy: Fail and selects namespaces labeled trainy.ai/org. It rejects placement label changes and restricted submissions when the ConfigMap does not select the custom dispatcher. The ConfigMap reports the desired dispatcher. Verify that the running Kueue manager uses that dispatcher before reopening submissions. Confirm admission of a valid tenant Job or JobSet with a server-side dry run. Custom Hubs must adapt the manifest’s Service and certificate references and enable the webhook TLS listener.

Keep the webhook active while restricted jobs exist. Before disabling the custom dispatcher or rolling back to an older controller, stop restricted submissions and drain those jobs. Delete the placement configuration before restoring an image that lacks the handler:

kubectl delete validatingwebhookconfiguration trainy-controller-cluster-placement

For disposable candidate tests, install the manifest after the candidate rollout and delete it before restoring an older baseline image.

To inspect a queued job, find its owning Workload and check status.nominatedClusterNames, status.clusterName and its admission checks. An empty nomination list can mean that no matching worker is currently eligible.