> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trainy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Restrict batch jobs to matching clusters

> Use placement labels to select a subset of your organization's worker clusters.

Add `placement.trainy.ai/` labels to a task's resource labels to restrict its
eligible worker clusters. An organization administrator assigns the same labels
to the enrolled `WorkerCluster` objects on the Hub.

```yaml theme={null}
name: east-training
resources:
  cpus: 4
  memory: 16
  image_id: ubuntu:24.04
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    placement.trainy.ai/pool: forecasting
    placement.trainy.ai/region: us-east
run: python train.py
```

Submit it with `konduktor launch task.yaml`. The Hub considers only clusters
with **both** matching labels. Labels without this prefix do not affect cluster
placement. Keys and values are case-sensitive; an empty value requires the key
to exist on the cluster.

The job stays queued if no eligible cluster matches. It never falls back to an
unmatched cluster. Labels narrow your organization's existing cluster access;
they do not grant access to another organization. Readiness, cordons, quota,
resource compatibility and machine0's single-Pod restriction still apply.

Multiple clusters can share a pool label. For one specific cluster, ask an
administrator to assign a unique placement label and use that label on the job.
These are cluster filters, so worker Nodes do not need these labels.

Placement labels are fixed when a job is created. Submit a new job to change
them. Administrators can relabel workers; the Hub re-evaluates pending
nominations every 15 seconds. A job already assigned to a cluster stays there.

Raw Kubernetes `batch/v1 Job` and `jobset.x-k8s.io/v1alpha2 JobSet` submissions
use these labels on their top-level `metadata.labels`. Putting them only on a
Pod template does not set a cluster restriction. Serving deployments are not
supported.

## Administrator setup

Label enrolled clusters on the Hub, for example:

```bash theme={null}
kubectl -n forecasting label workercluster east-worker \
  placement.trainy.ai/pool=forecasting placement.trainy.ai/region=us-east
```

The Hub must run the placement-aware controller, Kueue 0.19 and the custom
dispatcher from the machine0 dispatch feature:

```yaml theme={null}
multiKueue:
  dispatcherName: trainy.ai/machine0-dispatcher
```

The dispatcher supports ordinary workers with machine0 provisioning disabled.
Block restricted submissions while configuring Kueue and restarting its manager;
accept them only after the rollout completes. No
`integrations.labelKeysToCopy` entries are needed: the Hub reads the exact
owning Job or JobSet, including its UID.

The `config/gke-hub` overlay pins controller image
`sha-84a4e58720ed11f44d243bc8657360d40c403691` and leaves placement admission as
an explicit rollout step. First promote a published image that serves
`/validate-cluster-placement`. Use the target Hub's Kubernetes context, then
wait for that image to finish its rollout and install the webhook:

```bash theme={null}
kubectl -n trainy-controller-system rollout status \
	deployment/trainy-controller-controller-manager --timeout=300s
kubectl apply -f trainy-controller/config/gke-hub/cluster-placement-webhook.yaml
kubectl wait --for=jsonpath='{.webhooks[0].clientConfig.caBundle}' \
	validatingwebhookconfiguration/trainy-controller-cluster-placement --timeout=120s
```

The manifest keeps `failurePolicy: Fail` and selects namespaces labeled
`trainy.ai/org`. It rejects placement label changes and restricted submissions
when the ConfigMap does not select the custom dispatcher. The ConfigMap reports
the desired dispatcher. Verify that the running Kueue manager uses that
dispatcher before reopening submissions. Confirm admission of a valid tenant Job
or JobSet with a server-side dry run. Custom Hubs must adapt the manifest's
Service and certificate references and enable the webhook TLS listener.

Keep the webhook active while restricted jobs exist. Before disabling the
custom dispatcher or rolling back to an older controller, stop restricted
submissions and drain those jobs. Delete the placement configuration **before**
restoring an image that lacks the handler:

```bash theme={null}
kubectl delete validatingwebhookconfiguration trainy-controller-cluster-placement
```

For disposable candidate tests, install the manifest after the candidate
rollout and delete it before restoring an older baseline image.

To inspect a queued job, find its owning Workload and check
`status.nominatedClusterNames`, `status.clusterName` and its admission checks.
An empty nomination list can mean that no matching worker is currently eligible.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.