Generates a read-only validation script that confirms an AKS cluster bootstrapped via iam-vpc → aks-cluster → aks-nodes → acr → blob-storage → gpu-operator is healthy end-to-end. Checks RG, VNet, AKS provisioning state, node pool composition, ACR + AcrPull role, Storage + Blob Data Contributor role, GPU operator pods, GPU resource advertisement, a CUDA smoke pod, and a Blob auth probe. Emits PASS/WARN/FAIL plus a JSON report. Emitted as Stage 07 by `aks-bootstrap`; can be run standalone after the stack is up. Trigger when the user mentions "validate the cluster", "is the cluster ready", "smoke test the bootstrap", "kick the tires", or after any stack change.
Instrucciones de origen · Vista previa de solo lectura
name
aks-validate
description
Generates a read-only validation script that confirms an AKS cluster bootstrapped via iam-vpc → aks-cluster → aks-nodes → acr → blob-storage → gpu-operator is healthy end-to-end. Checks RG, VNet, AKS provisioning state, node pool composition, ACR + AcrPull role, Storage + Blob Data Contributor role, GPU operator pods, GPU resource advertisement, a CUDA smoke pod, and a Blob auth probe. Emits PASS/WARN/FAIL plus a JSON report. Emitted as Stage 07 by `aks-bootstrap`; can be run standalone after the stack is up. Trigger when the user mentions "validate the cluster", "is the cluster ready", "smoke test the bootstrap", "kick the tires", or after any stack change.
AKS bootstrap validation
What this skill produces
A read-only bash script that runs ~10 checks against the live stack and emits:
Expected pools present; node-vm-size matches aks-nodes inputs; GPU pool zone matches
FAIL
acr-pull
AcrPull role assignment exists on the ACR scope to the kubelet identity
FAIL
blob-rbac
Storage Blob Data Contributor on the storage account scope to the grantee
WARN if grantee/account unset
gpu-op
GPU operator pods all Ready; chart version matches GPU_OPERATOR_VERSION if set
FAIL
gpu-resource
At least one node advertises nvidia.com/gpu >= 1
FAIL
cuda-smoke
Pod with nvcr.io/nvidia/cuda:... requesting 1 GPU runs nvidia-smi, exits 0; pod is cleaned up
FAIL unless SKIP_CUDA=true
blob-probe
(Optional) From a pod using BLOB_PROBE_SERVICE_ACCOUNT, az storage blob list --auth-mode login succeeds
WARN if SKIP_BLOB_PROBE=true, storage account unset, or service account unset
Each check has a per-check timeout (default 60s; cuda-smoke up to 5min).
Bash script (complete)
#!/bin/bashset -uo pipefail # NOT -e — we want to collect all check results, not bail on the first fail# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"source"$REPO_ROOT/streaming-env.sh"for envfile in .env.iam-vpc .env.aks-cluster .env.acr .env.blob-storage; do
[[ -f "$REPO_ROOT/generated/$envfile" ]] && source"$REPO_ROOT/generated/$envfile"done
: "${CLUSTER_NAME:?}""${RESOURCE_GROUP:?}""${AZURE_SUBSCRIPTION_ID:?}"export KUBECONFIG="${KUBECONFIG:-/tmp/kubeconfig-${CLUSTER_NAME}}"
SKIP_CUDA="${SKIP_CUDA:-false}"
SKIP_BLOB_PROBE="${SKIP_BLOB_PROBE:-false}"
TIMEOUT_MIN="${TIMEOUT_MIN:-5}"mkdir -p "$REPO_ROOT/generated"
REPORT="$REPO_ROOT/generated/stage07-${CLUSTER_NAME}-validate-report.json"
-a CHECKS
PASS=0; WARN=0; FAIL=0
() {
= status= detail=
CHECKS+=()
PASS) ((PASS++));; WARN) ((WARN++));; FAIL) ((FAIL++));;
}
tool az kubectl jq;
-v >/dev/null || { ; 2; }
az account show >/dev/null 2>&1 || { ; 2; }
kubectl cluster-info >/dev/null 2>&1 || { ; 2; }
LOC=$(az group show -n --query location -o tsv 2>/dev/null || )
[[ -n ]]; record rg PASS ; record rg FAIL ;
VNET_NAME=
NODE_SUBNET_NAME=
API_SERVER_SUBNET_NAME=
[[ -z ]];
record vnet WARN
VNET_EXISTS=$(az network vnet show -n -g --query addressSpace.addressPrefixes[0] -o tsv 2>/dev/null || )
[[ -z ]];
record vnet FAIL
SUBNETS=$(az network vnet subnet list --vnet-name -g --query -o tsv 2>/dev/null || )
MISSING=
[[ != ** ]] && MISSING=
[[ != ** ]] && MISSING=
[[ -n ]];
record vnet FAIL
record vnet PASS
STATE=$(az aks show -n -g --query provisioningState -o tsv 2>/dev/null || )
OIDC=$(az aks show -n -g --query oidcIssuerProfile.issuerUrl -o tsv 2>/dev/null || )
WI=$(az aks show -n -g --query securityProfile.workloadIdentity.enabled -o tsv 2>/dev/null || )
K8S_VER=$(az aks show -n -g --query currentKubernetesVersion -o tsv 2>/dev/null || )
[[ == && -n && == ]];
record aks PASS
record aks FAIL
GPU_POOL_NAME=
GPU_VM_SIZE=
GPU_POOL_ZONE=
POOL_JSON=$(az aks nodepool list -g --cluster-name -o json 2>/dev/null || )
SYSTEM_COUNT=$( | jq )
GPU_COUNT=$( | jq --arg n )
[[ -eq 0 ]];
record nodes FAIL
[[ -eq 0 ]];
record nodes FAIL
DETAIL=
NODE_STATUS=
[[ -n && -n ]];
ACTUAL_SIZE=$( | jq -r --arg n )
[[ -n && != ]];
DETAIL=
NODE_STATUS=
DETAIL=
[[ -n && -n ]];
ACTUAL_ZONES=$( | jq -r --arg n )
DETAIL=
record nodes
[[ -z || -z ]];
record acr-pull WARN
ACR_ID=$(az acr show -n -g --query -o tsv 2>/dev/null || )
[[ -z ]];
record acr-pull FAIL
ROLE_COUNT=$(az role assignment list --scope \
--query \
-o tsv 2>/dev/null || )
[[ -gt 0 ]];
record acr-pull PASS
record acr-pull FAIL
[[ -z ]];
record blob-rbac WARN
SA_ID=$(az storage account show -n -g --query -o tsv 2>/dev/null || )
[[ -z ]];
record blob-rbac FAIL
BLOB_ROLE_COUNT=$(az role assignment list --scope \
--query \
-o tsv 2>/dev/null || )
[[ -gt 0 ]];
record blob-rbac PASS
record blob-rbac FAIL
GPU_OP_NS=
GPU_OP_TIMEOUT=
PODS_JSON=$(kubectl get pods -n -o json 2>/dev/null || )
TOTAL=$( | jq )
[[ -eq 0 ]];
record gpu-op FAIL
NOT_READY=$( | jq )
CHART_VER=$(kubectl get helmrelease -n gpu-operator -o jsonpath= 2>/dev/null || )
DETAIL=
[[ -eq 0 ]];
record gpu-op PASS
record gpu-op FAIL
GPU_NODE_COUNT=$(kubectl get nodes -o json 2>/dev/null \
| jq )
[[ -gt 0 ]];
record gpu-resource PASS
record gpu-resource FAIL
[[ == ]];
record cuda-smoke WARN
SMOKE_POD=
CUDA_IMAGE=
GPU_TAINT_KEY=
() { kubectl delete pod --ignore-not-found >/dev/null 2>&1 || ; }
cleanup_smoke EXIT
<<
kubectl pod --=jsonpath==Succeeded --= >/dev/null 2>&1;
GPU_MODEL=$(kubectl logs 2>/dev/null | grep -m1 | sed || )
record cuda-smoke PASS
POD_PHASE=$(kubectl get pod -o jsonpath= 2>/dev/null || )
record cuda-smoke FAIL
cleanup_smoke
- EXIT
[[ == || -z || -z ]];
record blob-probe WARN
BLOB_PROBE_NAMESPACE=
BLOB_PROBE_IMAGE=
BLOB_CONTAINER=
BLOB_POD=
() { kubectl delete pod -n --ignore-not-found >/dev/null 2>&1 || ; }
cleanup_blob_probe EXIT
kubectl run -n \
--image= \
--restart=Never \
--overrides= \
-- /bin/sh -lc >/dev/null 2>&1
kubectl pod -n --=jsonpath==Succeeded --= >/dev/null 2>&1;
record blob-probe PASS
POD_PHASE=$(kubectl get pod -n -o jsonpath= 2>/dev/null || )
record blob-probe FAIL
cleanup_blob_probe
- EXIT
\
\
| jq . >
[[ -gt 0 ]] && 1 || 0
Tolerant execution
When run from aks-bootstrap deploy mode, the orchestrator reports the exit code but does NOT treat exit 1 as a bootstrap failure — stages 01–06 have already succeeded. The validate stage is diagnostic, not constructive. See aks-bootstrap for the contract.
Validation checklist (for the generator)
Script uses set -uo pipefail (NOT -e — needs to keep collecting results past failures)
Preflight exits with code 2 (not 1) if a tool is missing or cluster is unreachable, so the orchestrator can distinguish "ran but failed checks" from "couldn't run"
Every check is wrapped to capture its own success/failure without aborting the script
GPU operator check inspects pod Ready conditions, not just pod phase
CUDA smoke pod has the GPU pool toleration
CUDA smoke waits for phase=Succeeded directly so short-lived nvidia-smi pods do not false-fail before becoming Ready
CUDA smoke pod is deleted on every exit path (use trap or explicit cleanup at end of check)
Blob probe runs inside the cluster using BLOB_PROBE_SERVICE_ACCOUNT, not the operator's local Azure login
Report is written unconditionally on exit 0 and exit 1 (only exit 2 skips it)
No persistent write/modify operations against cluster or Azure resources; temporary smoke/probe pods are cleaned up
Per-check timeout — a hung Helm pod can't stall the whole validate
SKIP_CUDA and SKIP_BLOB_PROBE flags work
Summary line on stdout includes counts (consumed by aks-bootstrap in deploy mode)
