Kubernetes execution platform — submits TAO container jobs as k8s Jobs with NVIDIA GPU scheduling; single-pod for one node, Indexed Jobs for multi-node distributed training. Use when running on EKS / GKE / AKS / on-prem clusters with the NVIDIA GPU Operator installed, or when integrating TAO into an existing k8s-native ML platform.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Kubernetes execution platform — submits TAO container jobs as k8s Jobs with NVIDIA GPU scheduling; single-pod for one node, Indexed Jobs for multi-node distributed training. Use when running on EKS / GKE / AKS / on-prem clusters with the NVIDIA GPU Operator installed, or when integrating TAO into an existing k8s-native ML platform.
license
Apache-2.0
compatibility
Requires GPU worker nodes with NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0; a `kubectl` client authenticated to the cluster; and the NVIDIA GPU Operator or device plugin. No nvidia-tao-sdk required — jobs are submitted with plain `kubectl`.
metadata
{"author":"NVIDIA Corporation","version":"0.1.0"}
allowed-tools
Read Bash
tags
["kubernetes","k8s","gpu","compute","container"]
Kubernetes
Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).
Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS / GKE / AKS / on-prem) or in-cluster service account (when running inside a pod).
Single-pod by default; opt into multi-node distributed training via num_nodes > 1 (uses Indexed Job + headless Service, see Multi-node training below).
Preflight
Three checks: GPU host runtime ready, cluster reachable via kubectl, GPU
Operator/device plugin present.
# 0. GPU node host runtime.# Run this on each self-managed GPU worker node or in the node image build.# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose# driver/toolkit lifecycle is owned by the cloud provider or GPU Operator policy.if [ "${TAO_K8S_SKIP_NODE_RUNTIME_CHECK:-0}" != "1" ]; then
TAO_SKILL_BANK_ROOT="${TAO_SKILL_BANK_ROOT:-$PWD}"
SETUP_SCRIPT="${TAO_SKILL_BANK_ROOT}/skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh"
bash "$SETUP_SCRIPT" --backend kubernetes --check-only || {
echo"MISSING: TAO Kubernetes GPU node runtime is not ready."echo"For self-managed GPU nodes, run after user approval:"echo" bash \"$SETUP_SCRIPT\" --backend kubernetes --install --yes"echo"For managed clusters, verify the node image/GPU Operator policy installs driver 580 and toolkit 1.19.0, then set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1."exit 1
}
fi
-v kubectl >/dev/null 2>&1 || {
1
}
kubectl cluster-info >/dev/null 2>&1 || {
1
}
gpu=$(kubectl get nodes -o jsonpath= 2>/dev/null | grep -v | -1)
[ -z ] || [ = ];
# 1. Cluster reachable (kubeconfig OR in-cluster service account)
command
echo
"MISSING: kubectl not found on PATH. Install kubectl to submit Jobs."
exit
echo
"MISSING: no reachable cluster (kubeconfig at ~/.kube/config, \$KUBECONFIG, or in-pod service account)."
echo
"Configure kubectl for your cluster, or set \$KUBECONFIG:"
The GPU node runtime check is mandatory for self-managed nodes. For managed
clusters where the client is not running on a GPU worker, verify the provider
node image or GPU Operator policy and set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1
instead of running the installer on the client. The GPU-capacity warning here is
a soft check; the submit verb re-checks allocatable nvidia.com/gpu and
hard-fails before applying the manifest (there is no gang scheduling, so a
too-big Job would sit Pending forever).
Credentials & configuration
Kubeconfig (one of):
~/.kube/config — default discovery path
$KUBECONFIG — alternate path
In-cluster service account — used when running inside a pod (no kubeconfig needed)
TAO_K8S_NAMESPACE (optional): default namespace for Job submission. Defaults to default.
TAO_K8S_CONTEXT (optional): kubeconfig context name to switch clusters.
NGC_KEY (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, reference its name in the rendered manifest's imagePullSecrets.
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / S3_BUCKET_NAME / S3_ENDPOINT_URL (optional): for S3 dataset I/O (storage tier C), injected into the pod via the per-job Secret (envFrom.secretRef), never inline. Legacy ACCESS_KEY/SECRET_KEY are mapped by tao-data-io.
Do not ask for Brev or SLURM credentials for Kubernetes runs. Ask for
S3 credentials only when the selected workflow uses s3:// inputs or outputs,
and ask for model-specific credentials such as HF_TOKEN only when the selected
model requires them. Before launch, verify the selected namespace can create
Jobs, dataset/result paths are visible from the pod, and PVC/mounted filesystem
paths are proven to be mounted into the job container; an agent-host local path
is not sufficient proof.
Execution — the four verbs
tao-run-on-kubernetes is a platform consumer: it runs a spec-bundle via
kubectl, mutating only the job-record. No nvidia-tao-sdk, no tao_sdk import —
jobs are submitted with plain kubectl apply.
$BANK = ${TAO_SKILL_BANK_PATH}.
submit
GPU-capacity gate — hard-fail first (no gang scheduling → a too-big Job
sits Pending forever):
Storage tier (via tao-data-io): A = mount a bound PVC/NFS holding the
data (author the mount paths, no fetch — the air-gap answer, and what the
packaged template does); C = ephemeral: an initContainer fetches from S3
into a shared emptyDir and a final step uploads results to S3 before TTL.
Tier C holds the GPU while it downloads. A pod reserves nvidia.com/gpu
for its whole lifetime, initContainers included, so a large tier-C fetch —
or a first-time multi-GB image pull — is billed and reaper-eligible idle GPU
time, exactly like pulling inside a SLURM allocation. Prefer tier A when the
data is already on a PVC; choose tier C knowingly, for small inputs.
Credentials → a per-job Secret (never inline in the manifest — it lands on
disk and is readable via kubectl get job -o yaml). Create it from an env-file
on stdin so no value hits a command line:
set -a; source /path/to/.env; set +a # omit if already exportedprintf'AWS_ACCESS_KEY_ID=%s\nAWS_SECRET_ACCESS_KEY=%s\nHF_TOKEN=%s\n' \
"$AWS_ACCESS_KEY_ID""$AWS_SECRET_ACCESS_KEY""$HF_TOKEN" \
| kubectl create secret generic "tao-creds-$JOB_ID" --from-env-file=/dev/stdin
The template references it via envFrom.secretRef — only the Secret name is
in the manifest.
Open the record — mints the id, binds results_dir, before launch:
results_dir must be a mounted (surviving) volume path or an S3 prefix —
ttlSecondsAfterFinished deletes the Job and its logs after it ends, so
nothing is recoverable from the Job object later.
Rendertemplates/k8s/single-pod-job.yaml.tmpl (CRED_SECRET=tao-creds-$JOB_ID),
then gate: redact_secrets.py lint <manifest> (fails on any inline
credential) + kubectl apply --dry-run=server -f <manifest> (schema validity).
Apply + record RUNNING:
kubectl apply -f "$MANIFEST""$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state RUNNING --backend-ref "$NAMESPACE/$JOB_ID"
A submit that skipped the gate or the open has no id — so it cannot launch.
status
kubectl get job "$JOB_ID" -o jsonpath='{.status.conditions[0].type} {.status.active} {.status.succeeded} {.status.failed}'
kubectl signal
vocab
no pods scheduled
PENDING (kubectl get pods -l job-name=$JOB_ID → ImagePullBackOff / Insufficient nvidia.com/gpu in message)
active ≥ 1
RUNNING
condition Complete
COMPLETE
condition Failed
ERROR (classify from the pod's terminated reason — OOMKilled → ERR_INFRA)
Job/pod not found
UNKNOWN (may be TTL-deleted — the job-record is the source of truth)
kubectl delete job "$JOB_ID" --cascade=foreground # also deletes the pods
kubectl delete secret "tao-creds-$JOB_ID" --ignore-not-found # tear down the per-job Secret"$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state CANCELED --source agent
Multi-node (nodes > 1)
Same four verbs, plus:
Version gate: require k8s ≥ 1.28 (kubectl version -o json) — the pod
hostname <job>-<index> (PodIndexLabel) that MASTER_ADDR=<job>-0.<svc>
resolves to needs it; on older clusters rank-0 hangs at rendezvous.
Capacity gate ×nodes: hard-fail unless allocatable GPUs ≥ gpus_per_node × nodes (no gang scheduling → a partial start leaves rank-0 waiting forever).
Render templates/k8s/indexed-job.yaml.tmpl — the headless Service +
Indexed Job + rendezvous env (WORLD_SIZE = node count, NODE_RANK from
JOB_COMPLETION_INDEX, MASTER_ADDR=<job>-0.<svc>, /dev/shm 16Gi so NCCL
doesn't silently hang). kubectl apply -f creates the Service and Job together;
cancel deletes the Job (Foreground) and the Service.
NCCL probe first (as SLURM) — a 2-node all-reduce with a timeout; on hang,
set the cluster NCCL env and re-probe; cache per cluster.
Local cluster (development, CI, and evals)
A throwaway minikube/kind cluster exercises admission, the four verbs,
job-record wiring, and log plumbing without cluster quota — and is what an
agent-driven eval should provision for itself. kubectl and minikube are
single static binaries needing no root, so a non-root CI container can install
them itself.
Two prerequisites keep a rendered Job Pending, and the first masks the second:
the PVC the template mounts must exist (persistentvolumeclaim "<name>" not found fires before any GPU complaint), then a Job requesting nvidia.com/gpu
on a GPU-less cluster reports Insufficient nvidia.com/gpu and waits forever.
Render NUM_GPUS=0 for a lifecycle-only run and say GPU scheduling was not
verified; on a Linux GPU host, minikube start --driver=docker --gpus all
passes real GPUs through, so one GPU box suffices for a GPU-real smoke.
Install commands, driver choice, the container/host-networking caveat, and the
fake-device-plugin middle option: references/local-cluster.md.
Container shell
The single-pod template invokes the container command via /bin/sh -c (POSIX
sh, present in busybox/distroless as well as TAO images). If your image relies
on bash-only syntax, override the interpreter in the rendered manifest.
GPU Operator dependency
The submit verb refuses to launch GPU jobs on a cluster with no nvidia.com/gpu allocatable. For self-managed clusters, first run the tao-setup-nvidia-gpu-host install action on every GPU worker node or bake the same package set into the node image:
Set num_nodes > 1 (see the Multi-node (nodes > 1) verb
steps above) to run distributed training across N pods. Rendering
templates/k8s/indexed-job.yaml.tmpl provisions:
A headless Service named after the Job (selector: job-name=<job-name>, clusterIP: None, publishNotReadyAddresses: true so pods can rendezvous before they're all Ready).
An Indexed Job with parallelism = completions = num_nodes, completionMode: Indexed. Each pod gets JOB_COMPLETION_INDEX injected by k8s automatically (= the node rank).
A command wrapper that exports the rendezvous env vars before invoking the user command. Two naming conventions are exported simultaneously:
Env var
Value
Read by
WORLD_SIZE
num_nodes
TAO PyTorch container's nvidia_tao_pytorch/core/entrypoint.py (uses this to mean node count, even though PyTorch's own convention is total processes)
NUM_GPU_PER_NODE
gpu_count
TAO PyTorch container's entrypoint
NNODES
num_nodes
torchrun and PyTorch-standard rendezvous
NPROC_PER_NODE
gpu_count
torchrun
NODE_RANK
$JOB_COMPLETION_INDEX
both
MASTER_ADDR
<job-name>-0.<job-name> (pod-0's DNS)
both
MASTER_PORT
29500
both (TAO's default)
Both naming conventions are set so TAO entrypoints (dino train, etc.) and raw torchrun commands work without modification.
For a TAO entrypoint, the container reads spec.train.num_nodes and the wired
env vars — e.g. dino train -e /tmp/spec.yaml with gpu_count=8, num_nodes=4
(4 × 8 = 32 GPUs total).
For raw torchrun-based commands (non-TAO containers), the wrapper invokes:
The capacity check sums across nodes: gpu_count × num_nodes ≤ cluster's allocatable nvidia.com/gpu.
Cluster requirements for multi-node
k8s 1.28+ is required for stable pod hostnames in Indexed Jobs (the PodIndexLabel feature). On older clusters the MASTER_ADDR=<job>-0.<svc> DNS lookup fails. Verify with kubectl version.
Pod-to-pod networking must be open on port 29500 (PyTorch default; configurable via MASTER_PORT env var). Most CNIs (Calico, Cilium, AWS VPC CNI) allow this by default; restrictive NetworkPolicies must be relaxed.
NCCL in the container talks GPU-to-GPU; if the cluster has multi-NIC nodes or RDMA, set NCCL_SOCKET_IFNAME / NCCL_IB_HCA in the container env of the rendered manifest.
For more sophisticated topologies (gang scheduling, PyTorch elastic / fault-tolerant training, MPI / Horovod, RDMA setup), reach for an operator instead of plain Indexed Job:
This skill's Indexed Job path is intentionally simple and dependency-free; if you need elastic restart or gang scheduling, layer one of these on top and submit jobs through the operator's CRD instead.
Common error patterns
No nvidia.com/gpu resources allocatable on the cluster — the GPU Operator (or NVIDIA Device Plugin) isn't installed. Install per the link above; verify with kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'.
ImagePullBackOff / ErrImagePull — the cluster can't pull the image. For nvcr.io: pre-create an image-pull secret in the namespace and reference it as the pod's imagePullSecrets in the rendered manifest:
Feed the key over stdin — --docker-password=$NGC_KEY would put the secret in
argv, where it is visible in the host's process table and shell history:
set -a; source /path/to/.env; set +a # omit if already exported
kubectl create secret generic ngc-pull-secret -n tao-jobs \
--type=kubernetes.io/dockerconfigjson \
--from-file=.dockerconfigjson=/dev/stdin <<EOF
{"auths": {"nvcr.io": {"username": "\$oauthtoken", "password": "${NGC_KEY}"}}}
EOF# Verify without reading the secret back:
kubectl get secret ngc-pull-secret -n tao-jobs >/dev/null && echo SECRET_OK
Pod stays Pending forever — kubectl describe pod -l job-name=$JOB_ID shows the scheduling reason in the Events. Common causes: insufficient GPU capacity (Insufficient nvidia.com/gpu), no node matches the pod's nodeSelector, missing image-pull secret, or PVC mount failure.
OOMKilled (exit 137) — container exceeded memory. Reduce batch size, lower max_length, or add a memory request/limit and target a larger node.
CredentialError: Could not authenticate to a Kubernetes cluster — neither kubeconfig nor in-cluster auth worked. Run kubectl get nodes to verify your config, or set $KUBECONFIG to the right path.
What this skill does NOT support (yet)
Elastic / fault-tolerant training. Indexed Job has backoff_limit=0 — failures fail the whole training run. For elastic restart (e.g., resume from checkpoint after a node death), use Kubeflow's PyTorchJob operator instead.
Gang scheduling. Indexed Job pods are scheduled independently — no all-or-nothing. Multi-node training will partially start if only some pods can be scheduled (rank-0 will hang waiting for peers). For all-or-nothing scheduling on shared clusters, use Volcano or Kueue.
MPI / Horovod. Use the MPI Operator. The Indexed Job path here is PyTorch-distributed-shaped (env-var rendezvous on MASTER_ADDR:MASTER_PORT).
Auto-creating image-pull secrets from $NGC_KEY. You pre-create the secret in the target namespace and pass the name. K8s namespace conventions vary widely, so we keep secret creation explicit.
