aks-validate

name

aks-validate

description

Generates a read-only validation script that confirms an AKS cluster bootstrapped via iam-vpc → aks-cluster → aks-nodes → acr → blob-storage → gpu-operator is healthy end-to-end. Checks RG, VNet, AKS provisioning state, node pool composition, ACR + AcrPull role, Storage + Blob Data Contributor role, GPU operator pods, GPU resource advertisement, a CUDA smoke pod, and a Blob auth probe. Emits PASS/WARN/FAIL plus a JSON report. Emitted as Stage 07 by `aks-bootstrap`; can be run standalone after the stack is up. Trigger when the user mentions "validate the cluster", "is the cluster ready", "smoke test the bootstrap", "kick the tires", or after any stack change.

AKS bootstrap validation

What this skill produces

A read-only bash script that runs ~10 checks against the live stack and emits:

• Stdout: per-check PASS, WARN, FAIL lines + summary.
• generated/stage07-<cluster-name>-validate-report.json — machine-readable report.
• Exit code: 0 if all checks PASS (WARN is non-fatal), 1 if any FAIL, 2 if preflight failed (missing tool, cluster unreachable).

This is the "cluster is ready" gate. Tolerant in orchestrator mode (see aks-bootstrap).

Inputs

Source from streaming-env.sh and any .env.* files written by upstream stages.

Required:

Variable	Description
CLUSTER_NAME	Name
RESOURCE_GROUP	RG
AZURE_SUBSCRIPTION_ID	Subscription

Optional (passed through from upstream stages for observed-vs-expected checks):

Variable	Description	Default
GPU_POOL_NAME, GPU_VM_SIZE, GPU_MIN_COUNT, GPU_MAX_COUNT	from aks-nodes	(unset)
ACR_NAME, KUBELET_IDENTITY_OBJECT_ID	from acr/aks-cluster	(unset)
STORAGE_ACCOUNT_NAME	from blob-storage	(unset)
BLOB_PROBE_NAMESPACE	Namespace containing the workload identity service account	default
BLOB_PROBE_SERVICE_ACCOUNT	K8s ServiceAccount bound to a workload identity with Blob access	(unset; skips probe with WARN)
BLOB_PROBE_IMAGE	Azure CLI image used for the in-cluster Blob auth probe	mcr.microsoft.com/azure-cli:2.61.0
SKIP_CUDA	Skip the CUDA smoke pod	false
SKIP_BLOB_PROBE	Skip the Blob auth probe	false
TIMEOUT_MIN	Per-check timeout cap	5

Output

generated/stage07-<cluster-name>-validate.sh and at runtime, generated/stage07-<cluster-name>-validate-report.json.

JSON shape:

{
  "cluster": "aks-nvcf-validation-westus3",
  "timestamp": "2026-05-27T12:34:56Z",
  "summary": { "pass": 8, "warn": 1, "fail": 0 },
  "checks": [
    { "id": "rg",          "status": "PASS", "detail": "RG exists in westus3" },
    { "id": "vnet",        "status": "PASS", "detail": "VNet 10.10.0.0/16; subnets: snet-aks-nodes, snet-aks-api-server" },
    { "id"              
                 
              
             
                
          
            
            
  

Checks

ID	What it verifies	Severity if fail
rg	RG exists in the expected region	FAIL
vnet	VNet exists; node + API-server subnets present	FAIL
aks	provisioningState=Succeeded; OIDC issuer URL non-empty; workload identity enabled	FAIL
nodes	Expected pools present; node-vm-size matches aks-nodes inputs; GPU pool zone matches	FAIL
acr-pull	AcrPull role assignment exists on the ACR scope to the kubelet identity	FAIL
blob-rbac	Storage Blob Data Contributor on the storage account scope to the grantee	WARN if grantee/account unset
gpu-op	GPU operator pods all Ready; chart version matches GPU_OPERATOR_VERSION if set	FAIL
gpu-resource	At least one node advertises nvidia.com/gpu >= 1	FAIL
cuda-smoke	Pod with nvcr.io/nvidia/cuda:... requesting 1 GPU runs nvidia-smi, exits 0; pod is cleaned up	FAIL unless SKIP_CUDA=true
blob-probe	(Optional) From a pod using BLOB_PROBE_SERVICE_ACCOUNT, az storage blob list --auth-mode login succeeds	WARN if SKIP_BLOB_PROBE=true, storage account unset, or service account unset

Each check has a per-check timeout (default 60s; cuda-smoke up to 5min).

Bash script (complete)

#!/bin/bash
set -uo pipefail   # NOT -e — we want to collect all check results, not bail on the first fail

# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$REPO_ROOT/streaming-env.sh"
for envfile in .env.iam-vpc .env.aks-cluster .env.acr .env.blob-storage; do
  [[ -f "$REPO_ROOT/generated/$envfile" ]] && source "$REPO_ROOT/generated/$envfile"
done

: "${CLUSTER_NAME:?}" "${RESOURCE_GROUP:?}" "${AZURE_SUBSCRIPTION_ID:?}"
export KUBECONFIG="${KUBECONFIG:-/tmp/kubeconfig-${CLUSTER_NAME}}"
SKIP_CUDA="${SKIP_CUDA:-false}"
SKIP_BLOB_PROBE="${SKIP_BLOB_PROBE:-false}"
TIMEOUT_MIN="${TIMEOUT_MIN:-5}"

mkdir -p "$REPO_ROOT/generated"
REPORT="$REPO_ROOT/generated/stage07-${CLUSTER_NAME}-validate-report.json"
 -a CHECKS
PASS=0; WARN=0; FAIL=0

() {
   = status= detail=
  CHECKS+=()
     PASS) ((PASS++));; WARN) ((WARN++));; FAIL) ((FAIL++));; 
      
}


 tool  az kubectl jq; 
   -v  >/dev/null || {  ;  2; }

az account show >/dev/null 2>&1 || {  ;  2; }
kubectl cluster-info >/dev/null 2>&1 || {  ;  2; }


LOC=$(az group show -n  --query location -o tsv 2>/dev/null ||  )
 [[ -n  ]];  record rg PASS ;  record rg FAIL ; 


VNET_NAME=
NODE_SUBNET_NAME=
API_SERVER_SUBNET_NAME=
 [[ -z  ]]; 
  record vnet WARN 

  VNET_EXISTS=$(az network vnet show -n  -g  --query addressSpace.addressPrefixes[0] -o tsv 2>/dev/null ||  )
   [[ -z  ]]; 
    record vnet FAIL 
  
    SUBNETS=$(az network vnet subnet list --vnet-name  -g  --query  -o tsv 2>/dev/null ||  )
    MISSING=
    [[  != ** ]] && MISSING=
    [[  != ** ]] && MISSING=
     [[ -n  ]]; 
      record vnet FAIL 
    
      record vnet PASS 
    
  



STATE=$(az aks show -n  -g  --query provisioningState -o tsv 2>/dev/null ||  )
OIDC=$(az aks show -n  -g  --query oidcIssuerProfile.issuerUrl -o tsv 2>/dev/null ||  )
WI=$(az aks show -n  -g  --query securityProfile.workloadIdentity.enabled -o tsv 2>/dev/null ||  )
K8S_VER=$(az aks show -n  -g  --query currentKubernetesVersion -o tsv 2>/dev/null ||  )
 [[  ==  && -n  &&  ==  ]]; 
  record aks PASS 

  record aks FAIL 



GPU_POOL_NAME=
GPU_VM_SIZE=
GPU_POOL_ZONE=
POOL_JSON=$(az aks nodepool list -g  --cluster-name  -o json 2>/dev/null ||  )
SYSTEM_COUNT=$(  | jq )
GPU_COUNT=$(  | jq --arg n  )
 [[  -eq 0 ]]; 
  record nodes FAIL 
 [[  -eq 0 ]]; 
  record nodes FAIL 

  DETAIL=
  NODE_STATUS=
   [[ -n  && -n  ]]; 
    ACTUAL_SIZE=$(  | jq -r --arg n  )
     [[ -n  &&  !=  ]]; 
      DETAIL=
      NODE_STATUS=
    
      DETAIL=
    
  
   [[ -n  && -n  ]]; 
    ACTUAL_ZONES=$(  | jq -r --arg n  )
    DETAIL=
  
  record nodes  



 [[ -z  || -z  ]]; 
  record acr-pull WARN 

  ACR_ID=$(az acr show -n  -g  --query  -o tsv 2>/dev/null ||  )
   [[ -z  ]]; 
    record acr-pull FAIL 
  
    ROLE_COUNT=$(az role assignment list --scope  \
      --query  \
      -o tsv 2>/dev/null ||  )
     [[  -gt 0 ]]; 
      record acr-pull PASS 
    
      record acr-pull FAIL 
    
  



 [[ -z  ]]; 
  record blob-rbac WARN 

  SA_ID=$(az storage account show -n  -g  --query  -o tsv 2>/dev/null ||  )
   [[ -z  ]]; 
    record blob-rbac FAIL 
  
    BLOB_ROLE_COUNT=$(az role assignment list --scope  \
      --query  \
      -o tsv 2>/dev/null ||  )
     [[  -gt 0 ]]; 
      record blob-rbac PASS 
    
      record blob-rbac FAIL 
    
  



GPU_OP_NS=
GPU_OP_TIMEOUT=
PODS_JSON=$(kubectl get pods -n  -o json 2>/dev/null ||  )
TOTAL=$(  | jq )
 [[  -eq 0 ]]; 
  record gpu-op FAIL 

  NOT_READY=$(  | jq )
  CHART_VER=$(kubectl get helmrelease -n  gpu-operator -o jsonpath= 2>/dev/null ||  )
  DETAIL=
   [[  -eq 0 ]]; 
    record gpu-op PASS 
  
    record gpu-op FAIL 
  



GPU_NODE_COUNT=$(kubectl get nodes -o json 2>/dev/null \
  | jq )
 [[  -gt 0 ]]; 
  record gpu-resource PASS 

  record gpu-resource FAIL 



 [[  ==  ]]; 
  record cuda-smoke WARN 

  SMOKE_POD=
  CUDA_IMAGE=
  GPU_TAINT_KEY=
  
  () { kubectl delete pod  --ignore-not-found >/dev/null 2>&1 || ; }
   cleanup_smoke EXIT
  
  
  
   <<
   kubectl  pod  --=jsonpath==Succeeded --= >/dev/null 2>&1; 
    GPU_MODEL=$(kubectl logs  2>/dev/null | grep -m1  | sed  ||  )
    record cuda-smoke PASS 
  
    POD_PHASE=$(kubectl get pod  -o jsonpath= 2>/dev/null ||  )
    record cuda-smoke FAIL 
  
  cleanup_smoke
   - EXIT



 [[  ==  || -z  || -z  ]]; 
  record blob-probe WARN 

  BLOB_PROBE_NAMESPACE=
  BLOB_PROBE_IMAGE=
  BLOB_CONTAINER=
  BLOB_POD=
  () { kubectl delete pod  -n  --ignore-not-found >/dev/null 2>&1 || ; }
   cleanup_blob_probe EXIT
  kubectl run  -n  \
    --image= \
    --restart=Never \
    --overrides= \
    -- /bin/sh -lc  >/dev/null 2>&1
   kubectl  pod  -n  --=jsonpath==Succeeded --= >/dev/null 2>&1; 
    record blob-probe PASS 
  
    POD_PHASE=$(kubectl get pod  -n  -o jsonpath= 2>/dev/null ||  )
    record blob-probe FAIL 
  
  cleanup_blob_probe
   - EXIT



  \
       \
   | jq . > 


 
 
[[  -gt 0 ]] &&  1 ||  0

Tolerant execution

When run from aks-bootstrap deploy mode, the orchestrator reports the exit code but does NOT treat exit 1 as a bootstrap failure — stages 01–06 have already succeeded. The validate stage is diagnostic, not constructive. See aks-bootstrap for the contract.

Validation checklist (for the generator)

• Script uses set -uo pipefail (NOT -e — needs to keep collecting results past failures)
• Preflight exits with code 2 (not 1) if a tool is missing or cluster is unreachable, so the orchestrator can distinguish "ran but failed checks" from "couldn't run"
• Every check is wrapped to capture its own success/failure without aborting the script
• GPU operator check inspects pod Ready conditions, not just pod phase
• CUDA smoke pod has the GPU pool toleration
• CUDA smoke waits for phase=Succeeded directly so short-lived nvidia-smi pods do not false-fail before becoming Ready
• CUDA smoke pod is deleted on every exit path (use trap or explicit cleanup at end of check)
• Blob probe runs inside the cluster using BLOB_PROBE_SERVICE_ACCOUNT, not the operator's local Azure login
• Report is written unconditionally on exit 0 and exit 1 (only exit 2 skips it)
• No persistent write/modify operations against cluster or Azure resources; temporary smoke/probe pods are cleaned up
• Per-check timeout — a hung Helm pod can't stall the whole validate
• SKIP_CUDA and SKIP_BLOB_PROBE flags work
• Summary line on stdout includes counts (consumed by aks-bootstrap in deploy mode)

References

Topic	Source
az aks show JSON shape	https://learn.microsoft.com/cli/azure/aks#az-aks-show
kubectl get nodes -o json	https://kubernetes.io/docs/reference/kubectl/jsonpath/
GPU Operator readiness	https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html