| name | orchestrate-streaming-stack |
| description | Orchestrate an end-to-end NVIDIA self-managed streaming stack across AWS or Azure. Use when planning, generating, sequencing, deploying, or validating the full path from CSP base infrastructure through NVCF self-managed control plane, NVCF auxiliary services, caches, storage APIs, and post-deploy readiness gates. Trigger when the user asks for orchestration, full-stack setup, bring-up order, runbook, dependency sequencing, AWS/Azure streaming deployment, csp-infra plus NVCF, caches, or storage API coordination. Do not use for a single component task when the user explicitly asks only for one child skill. |
Orchestrate Streaming Stack
Overview
Coordinate the full self-managed streaming deployment without duplicating the component skills. Treat this skill as the top-level runbook: choose the CSP path, load the correct child skills, keep state in streaming-env.sh, and advance only when each phase's readiness gate passes.
Run from the streaming-staging repository root unless a child skill says otherwise.
Safety Rules
- Never ask for, print, echo, grep, decode, or paste secret values. Treat
NGC_API_KEY_FILE, cloud credentials, kubeconfigs, registry passwords, and generated secret manifests as sensitive.
- Prefer generate/plan mode unless the user explicitly asks to deploy, apply, run, provision, or execute.
- Before executing cloud-changing commands, confirm the target CSP, account/subscription, region, cluster name, and mode in one concise message.
- Do not skip a readiness gate to continue the sequence. If a gate fails, stop at the failing phase, summarize the failing checks, and route to the owning child skill.
- Keep AWS and Azure assumptions separate. Do not apply AKS load-balancer, ACR, Azure Blob, or Azure CNI details to EKS. Do not apply EKS, ECR, NLB, or IRSA details to AKS.
Mode Detection
Infer the operating mode from the user's prompt:
| Mode | Trigger | Behavior |
|---|
| Plan | "plan", "runbook", "sequence", "what order", ambiguous setup language | Produce ordered phases, required inputs, and gates. Do not write deploy scripts unless useful and requested. |
| Generate | "generate scripts", "draft manifests", "prepare Terraform", "write runbook artifacts" | Generate artifacts for review, then stop. Do not execute. |
| Deploy | "deploy", "apply", "create", "provision", "run", "bring up" | Generate or use existing artifacts, then execute one phase at a time with stop-on-failure semantics. |
| Recover | "resume", "continue", "fix failed phase", "rerun from phase N" | Inspect current state and logs first, then continue from the earliest failed or incomplete gate. |
When intent is ambiguous, choose Plan or Generate mode and tell the user the exact phrase to use for execution.
Required Context
Collect or discover this context before sequencing work:
| Context | Source |
|---|
| CSP | User prompt; if missing, ask AWS or Azure. |
| Cluster name | streaming-env.sh, generated env files, Terraform outputs, kube context, or user prompt. |
| Region/location | streaming-env.sh, AWS/Azure CLI config, or user prompt. |
| Account/subscription | aws sts get-caller-identity for AWS; az account show for Azure. |
| Artifact mode | Plan, Generate, Deploy, or Recover. |
| Storage target | Azure Blob, S3, existing object store, or filesystem adapter. |
| Cache target | DDCS, UCC, GXCache, NVCF container/proxy cache, or none. Prefer DDCS for shader cache when the workload needs Kit/write-render-worker shader caching. Treat NVCF container/proxy cache as required for ARB/validation unless the user explicitly declines registry/S3 proxy caching. |
| NVCF stack path | User-provided extracted nvcf-self-managed-stack path, or repo-local sample path when the task is only planning. |
Use streaming-env.sh at the repo root as the shared env file. If it is missing and work is not read-only planning, copy streaming-env.sh.example to streaming-env.sh, set mode 600, and populate only the values needed for the selected path.
Child Skill Routing
Read the child skill before performing that phase. Open only the provider or component references needed for the selected path.
| Phase | Azure child skill | AWS child skill | Notes |
|---|
| Base infra | csp-infra/azure/.agents/skills/aks-bootstrap/SKILL.md | csp-infra/aws/.agents/skills/aws-infra/SKILL.md | Azure has a seven-stage orchestrator. AWS owns Terraform foundation plus GPU Operator. |
| Registry mirror | csp-infra/azure/.agents/skills/acr-mirror/SKILL.md | csp-infra/aws/.agents/skills/aws-infra/ecr-mirror/SKILL.md | Mirror after registry exists and before installs that require private pulls. |
| NVCF prerequisites | nvcf/.agents/skills/nvcf-self-managed-prerequisite/SKILL.md | same | Installs cluster prerequisites such as KAI/SMB CSI when required by the selected NVCF stack path. |
| NVCF control plane | nvcf/.agents/skills/nvcf-self-managed-installation/SKILL.md | same | Requires extracted stack path, helmfile 1.1.x, helm-diff, kubectl, NGC auth, and gateway domain. |
| NVCF CLI | nvcf/.agents/skills/nvcf-self-managed-cli/SKILL.md | same | Use for post-install function and CLI interaction when requested. |
| LLS | nvcf/.agents/skills/nvcf-lls-install/SKILL.md with providers/kubernetes.md | same | Required for WebRTC/UDP media ingress. |
| GXCache | caches/.agents/skills/nvcf-gxcache/SKILL.md with providers/kubernetes.md | same | Use when specifically requested or required by the target stack. |
| DDCS | caches/.agents/skills/caches-ddcs-deploy/SKILL.md | same, adapt provider assumptions | Use for Kit/write-render-worker shader cache. GKE examples in the skill are not AWS/Azure defaults. |
| OV Content Cache | caches/.agents/skills/caches-ovcontentcache-deploy/SKILL.md | same | Use when OV Content Cache is part of the requested cache layer. |
| NVCF container cache | nvcf/container-cache/README.md | same | Required for ARB/validation unless user explicitly declines. Install into the namespace. Configures registry and proxy caching backed by the CSP-native registry (ECR on AWS, ACR on Azure), DaemonSet node runtime config, and PVCs. |
End-to-End Sequence
Phase 0 - Preflight
- Confirm mode, CSP, account/subscription, region, cluster name, and whether commands will execute.
- Verify local tools for the selected path:
- Common:
kubectl, helm (Helm 3.x, >=3.12 and <4 — Helm 4 is
unsupported), jq, curl, python3. Confirm with helm version; if the
distro default is Helm 4 (e.g. snap helm on current Linux), install Helm
3.18.x and put it ahead of v4 in PATH before continuing.
- NVCF:
helmfile 1.1.x and helm-diff.
- Azure:
az, and terraform only if generating Terraform artifacts.
- AWS:
aws, terraform.
- Registry mirroring:
skopeo or the child skill's selected mirror backend.
- Check
NGC_API_KEY_FILE exists and is non-empty without printing its contents.
- Inspect
image-manifest/manifest.yaml for chart/image versions before pinning versions manually.
Gate: toolchain exists (including helm reporting a 3.x version >=3.12 and
<4), identity matches target, streaming-env.sh has required non-secret
values, and no secrets were exposed.
Phase 1 - Base CSP Infrastructure
Azure:
- Load
aks-bootstrap.
- Use its mode detection and required
streaming-env.sh variables.
- Generate or execute stages 01-07 in order.
AWS:
- Load
aws-infra.
- Generate or apply Terraform for VPC, subnets, EKS, node groups, IAM, and ECR.
- Install GPU Operator after kubeconfig works.
Gate: kubectl cluster-info succeeds, GPU nodes can become schedulable, GPU Operator is healthy, registry exists, and provider storage primitives exist or are explicitly deferred.
Phase 2 - Registry Mirror
Mirror images and charts only after base infra creates the target registry.
- Azure: load
acr-mirror and mirror to ACR.
- AWS: load
aws-infra/ecr-mirror and mirror to ECR.
OCI vs non-OCI charts. The manifest has a mix of OCI and non-OCI artifacts. Container images (nvcr.io/...) are already OCI and mirror as-is. Some Helm charts are published to an https:// Helm repository (for example helm.ngc.nvidia.com). Those are not OCI and cannot be resolved by global.helm.sources (OCI) or mirrored directly. Before pointing the stack at your mirror, convert each https:// chart to OCI: helm repo add / helm pull the .tgz, then helm push <chart>.tgz oci://<your-registry>/<mirror-sub-path>. Only OCI charts can live under global.helm.sources.
Gate: target registry contains required images/charts for NVCF, LLS, caches, and storage components selected for the deployment. Verify helm-nvcf-container-cache, nvcf-container-cache, and nvcf-proxy-tls-certs are mirrored when container/proxy cache is in scope.
Phase 3 - NVCF Prerequisites and Control Plane
- Load
nvcf-self-managed-prerequisite for cluster prerequisites required by the selected stack.
- Load
nvcf-self-managed-installation for the extracted nvcf-self-managed-stack.
- In
<nvcf-self-managed-stack>/environments/base.yaml, set:
addon:
lls:
enabled: true
If TURN enablement is not automatically derived from the addon flag, set this in <nvcf-self-managed-stack>/global.yaml.gotmpl:
notary:
turn:
enabled: true
- Configure gateway/domain values from the provider ingress/LB output.
- Run
helmfile template before helmfile sync.
Gate: dependency namespaces are healthy, NVCF API/backend services are ready, ingress routes exist, and the selected gateway host resolves or is otherwise reachable for validation.
Phase 4 - NVCF Auxiliary Services
Install only the services required by the target workload:
- LLS for WebRTC/UDP streaming. Load the Kubernetes provider guide and apply the notes for the target cluster.
- GXCache only when the stack or workload requires it.
- DDCS for shader cache when Kit/write-render-worker startup shader compilation needs acceleration. Install with
--set monitoring.enabled=false when Prometheus CRDs (e.g. PodMonitoring) are not present on the cluster.
- UCC when object/content cache is in scope.
- NVCF container/proxy cache — required for ARB/validation unless the user explicitly declines. Read
nvcf/container-cache/README.md, install nvcf-container-cache into the container-caching namespace, configure images from the CSP-native registry (ECR on AWS, ACR on Azure), and validate the nvcf-container-cache StatefulSet, nvcf-proxy-cache, DaemonSet node runtime config, and PVCs.
Gate: service rollouts complete, provider LoadBalancers or Services have addresses, UDP 5004 is reachable for LLS, cache health/proof-of-life checks pass, and when container cache is installed: helm list -n container-caching shows deployed releases, nvcf-container-cache StatefulSet and nvcf-container-cache-cc DaemonSet are Ready, all PVCs are Bound, and node runtime config is applied.
Phase 5 - Storage API
- Load
storage/.agents/skills/ovstorage/SKILL.md.
- Read
GUIDE.md in that skill directory.
- Choose adapter:
- S3 production adapter for AWS.
- Azure production adapter for Azure.
- Example filesystem adapter for development or custom adapter work.
- Deploy Discovery plus the selected adapter, then optional Event Aggregation, Event Consumer, RabbitMQ, Envoy Auth Extension, Storage Navigator, and ingress only when requested.
Gate: Discovery answers, adapter CRUD works, notifications/permissions checks pass when enabled, and storage endpoint details are recorded for downstream functions.
Phase 6 - Final Readiness
Run a concise acceptance pass:
kubectl get pods -A has no unexpected CrashLoopBackOff, ImagePullBackOff, or Pending pods.- GPU allocatable resources are visible.
- NVCF API health and auth path are reachable.
- LLS/media path is validated when streaming is in scope.
- Cache proof-of-life passes for selected cache services.
- When container/proxy cache is in scope:
helm list -n container-caching shows deployed, nvcf-container-cache StatefulSet and nvcf-container-cache-cc DaemonSet are Ready, all PVCs are Bound, and node runtime config is applied on GPU nodes.
- Storage API CRUD and Discovery checks pass.
- Generated artifacts and env files are listed with paths.
Recovery Workflow
For resume or repair requests:
- Identify the last completed gate from generated scripts, Terraform outputs, Helm releases, namespaces, and pod state.
- Do not rerun earlier destructive phases unless the child skill declares them idempotent.
- Route to the owning child skill for the failed phase.
- After fixing, rerun the failed phase's gate, then continue the top-level sequence.
Output Format
For Plan or Generate mode, return a phase table:
| Phase | Owner skill | Action | Gate | Status |
|---|
For Deploy or Recover mode, report after every phase:
- Completed action.
- Artifact paths created or used.
- Gate result.
- Next phase and what it needs.
Keep the final answer short, with the current state, failed gate if any, and exact next command or prompt needed.
