nvidia-physical-ai-infrastructure-setup-and-resilient-scaling

name	nvidia-physical-ai-infrastructure-setup-and-resilient-scaling
description	Use when the user wants to set up, scale, validate, or harden NVIDIA physical AI infrastructure for synthetic data generation workflows across local MicroK8s or Azure AKS, including Kubernetes clusters, inference endpoint deployment, OSMO deployment, workload submission readiness, and infrastructure failure recovery. Trigger keywords: physical ai infrastructure, resilient scaling, SDG infrastructure, microk8s, azure aks, NVCF deployment, NIM Operator, OSMO deploy, workflow scaling. Don't trigger for: OSMO log summarization or workload-only operations unless infrastructure setup, scaling, validation, or recovery is requested.
license	Apache-2.0
version	1.0.0
tools	["Read","Shell"]
compatibility	Requires the selected component prerequisites, usually kubectl plus either MicroK8s or Azure CLI/Terraform, and OSMO or inference credentials for the chosen target.
metadata	{"author":"NVIDIA Physical AI","tags":["physical-ai","infrastructure","kubernetes","azure","microk8s","osmo","nim-operator","scaling"],"domain":"ai-ml","languages":["bash","hcl","yaml"]}

Physical AI Infrastructure Setup And Resilient Scaling

Canonical skill for the Physical AI infrastructure stack. Use it to compose cluster, inference, OSMO, and workload stages into a reproducible Physical AI SDG environment, then keep the environment observable and recoverable.

Operating Rules

• Read only the component references needed for the selected target. Do not load every component by default.
• Keep the repo as the durable artifact. Fix checked-in config or scripts, then rerun. Do not recover a failed install with untracked one-off changes.
• Run mutating cluster, OSMO, Helm, Terraform, or Azure operations through checked-in scripts when a script exists. Read-only diagnostics are allowed.
• Stop at the first red gate. Fix the lowest owning layer in this order: config, script, then skill guidance.
• Derive values from the environment when possible. Ask only for values that cannot be inferred, such as API keys, target choice, or quota tradeoffs.
• Store secrets in ${REPO_ROOT}/.env. Cluster-derived values such as storage, database, Redis, and endpoint names come from Terraform outputs or platform queries, not .env.
• Preflight means no deployed state: no cluster API, Terraform outputs, Helm releases, OSMO pools, or workflow state. Those belong to deploy/verify gates.
• Never print, echo, or paste raw keys into commands, YAML, logs, or transcripts. Prefer credential handles, Kubernetes secretKeyRef, and runtime-only secret injection. Scan raw transcript exports with scripts/scan_transcript_secrets.py before sharing.
• Use absolute paths. Derive repo root with git rev-parse --show-toplevel.

Component References

Each component lives inside this skill so the stack has one canonical trigger. Load the component reference only when the selected target needs that slice.

Concern	Load	Assets
Stage matrix and old driver notes	components/driver/reference.md	None
MicroK8s cluster	components/cluster-microk8s/reference.md	components/cluster-microk8s/scripts/, components/cluster-microk8s/runtimeclass-nvidia-runc.yaml
Azure AKS cluster	components/cluster-azure/reference.md	components/cluster-azure/scripts/,