aks-nodes

name

aks-nodes

description

Generates a bash or Terraform snippet that adds additional node pools to an existing AKS cluster — GPU pool (single-zone, NV/NC SKU) and optional compute pool (for LLS streaming-proxy or other CPU-heavy workloads). Use as Stage 03 of `aks-bootstrap`, after `aks-cluster`. Trigger when the user mentions "GPU node pool", "add node pool", "compute pool", "aks-nodes", or asks "how do I add a GPU pool to AKS".

AKS additional node pools (GPU + compute)

What this skill produces

Adds 0–2 additional node pools to an existing AKS cluster:

Pool	SKU (default)	Zone(s)	Labels	Purpose
GPU pool	Standard_NV36ads_A10_v5	["1"]	nvidia.com/gpu=true	NVCF function workloads (GPU)
Compute pool (optional)	Standard_D8ds_v5	["1"]	node-type=compute	LLS streaming-proxy (5 CPU / 24 GiB)

GPU pool MUST be single-zone if LLS will be deployed — Azure Standard LB zonality must match the GPU pool's zone (see nvcf-azure-deploy architecture rule #9 — or locally tf-conventions).

GPU SKU recommendations (validate quota per region; quota requests take 24–72h):

Use case	SKU	GPU
Validation (easy quota)	Standard_NV36ads_A10_v5	1× A10
L40S parity	Standard_NV18ads_L40S_v3	1× L40S
H100 prod-intent	Standard_NC40ads_H100_v5	1× H100

Inputs

Source from streaming-env.sh + generated/.env.iam-vpc + generated/.env.aks-cluster.

Required:

Variable	Description
AZURE_SUBSCRIPTION_ID	Subscription
RESOURCE_GROUP	RG
CLUSTER_NAME	Existing AKS cluster
NODE_SUBNET_ID	from .env.iam-vpc

Optional (firm defaults):

Variable	Description	Default
GPU_POOL_NAME	GPU pool name (1–12 chars, lowercase, alphanumeric only — NO hyphens; this is an AKS constraint, not project naming)	gpu
GPU_VM_SIZE	GPU SKU	Standard_NV36ads_A10_v5
GPU_NODE_COUNT	Initial GPU nodes (autoscaler floor)	0
GPU_MIN_COUNT	Autoscale floor	0
GPU_MAX_COUNT	Autoscale ceiling	2
GPU_ZONES	Single zone for the GPU pool	1
GPU_NODE_TAINTS	Comma-separated taints. Empty by default — the GPU pool is reserved via the nvidia.com/gpu=true label alone, matching the AWS foundation (aws-infra uses labels, no taint). A sku=gpu:NoSchedule taint blocks NVCF function pods (no matching toleration) → NVCA reports capacity: 0 → function deploy 400s. Only set this if you intend to wire matching tolerations onto every GPU workload.	(empty)
GPU_DRIVER_INSTALL	enabled (AKS installs driver) or disabled (GPU operator owns the driver lifecycle) — see "GPU driver source" below	enabled
ENABLE_COMPUTE_POOL	Create the compute pool?	false
COMPUTE_POOL_NAME	Compute pool name (same naming constraint as GPU)	compute
COMPUTE_VM_SIZE	Compute SKU	Standard_D8ds_v5
COMPUTE_MIN_COUNT	Autoscale floor	1
COMPUTE_MAX_COUNT

Outputs

• Bash: generated/stage03-<cluster-name>-aks-nodes.sh — idempotent (az aks nodepool add errors on exists; the script checks first).
• Terraform: generated/<cluster-name>/aks-nodes/main.tf containing azurerm_kubernetes_cluster_node_pool resources.

GPU driver source — pick one

AKS NV/NC SKUs pre-install the NVIDIA driver. The GPU Operator (from the gpu-operator skill) also wants to install one. Pick exactly one source:

GPU_DRIVER_INSTALL	What happens	When to use
enabled (default)	AKS installs the driver. GPU Operator runs with driver.enabled=false.	Validation; smallest blast radius.
disabled	Pass --skip-gpu-driver-install (Azure CLI flag; not in AVM resource module today — needs azapi patch or post-create az aks nodepool update). GPU Operator owns driver lifecycle with driver.enabled=true.	Production when NVCF runtime requires operator-managed drivers.

For default validation, leave at enabled.

Bash snippet (reference)

#!/bin/bash
set -euo pipefail

# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$REPO_ROOT/streaming-env.sh"
for envfile in .env.iam-vpc .env.aks-cluster; do
  if [[ ! -f "$REPO_ROOT/generated/$envfile" ]]; then
    echo "ERROR: $REPO_ROOT/generated/$envfile not found. Run earlier stages first." >&2
    exit 1
  fi
  source "$REPO_ROOT/generated/$envfile"
done

: "${CLUSTER_NAME:?}" "${RESOURCE_GROUP:?}" "${NODE_SUBNET_ID:?}"
GPU_POOL_NAME="${GPU_POOL_NAME:-gpu}"
GPU_VM_SIZE="${GPU_VM_SIZE:-Standard_NV36ads_A10_v5}"
GPU_NODE_COUNT="${GPU_NODE_COUNT:-0}"
GPU_MIN_COUNT="${GPU_MIN_COUNT:-0}"
GPU_MAX_COUNT="${GPU_MAX_COUNT:-2}"
GPU_ZONES=
GPU_NODE_TAINTS=   
GPU_DRIVER_INSTALL=
ENABLE_COMPUTE_POOL=

az account  --subscription 




 
SKU_PREFIX=$(  | awk -F_ )
az vm list-usage --location  \
  --query  \
  -o table ||  


 az aks nodepool show --cluster-name  --resource-group  --name  &>/dev/null; 
   

  DRIVER_FLAG=
   [[  ==  ]]; 
    DRIVER_FLAG=
  
  
  
  TAINT_FLAG=
   [[ -n  ]]; 
    TAINT_FLAG=
  
   
  az aks nodepool add \
    --cluster-name  --resource-group  \
    --name  --node-vm-size  \
    --node-count  \
    --enable-cluster-autoscaler --min-count  --max-count  \
    --zones  \
     \
    --labels nvidia.com/gpu= \
    --vnet-subnet-id  \
     \
    --only-show-errors



 [[  ==  ]]; 
  COMPUTE_POOL_NAME=
   az aks nodepool show --cluster-name  --resource-group  --name  &>/dev/null; 
     
  
     
    az aks nodepool add \
      --cluster-name  --resource-group  \
      --name  \
      --node-vm-size  \
      --node-count  \
      --enable-cluster-autoscaler --min-count  --max-count  \
      --zones  \
      --labels  \
      --vnet-subnet-id  \
      --only-show-errors
  


kubectl get nodes -o wide

Terraform snippet (reference)

resource "azurerm_kubernetes_cluster_node_pool" "gpu" {
  name                  = var.gpu_pool_name
  kubernetes_cluster_id = var.cluster_id
  vm_size               = var.gpu_vm_size
  node_count            = var.gpu_node_count
  auto_scaling_enabled  = true
  min_count             = var.gpu_min_count
  max_count             = var.gpu_max_count
  zones                 = var.gpu_zones
  vnet_subnet_id        = var.node_subnet_id
  node_labels           = { "nvidia.com/gpu" = "true" }
  # No taint by default — label-only reservation, matching the AWS foundation.
  # A sku=gpu:NoSchedule taint blocks NVCF function pods (no matching toleration) → capacity:0.
  # Set node_taints only if you also wire matching tolerations onto every GPU workload.
  node_taints           = var.gpu_node_taints   # declare with default [] in your variables.tf

  # GPU_DRIVER_INSTALL = "disabled" needs an azapi resource (not exposed by azurerm today):
  #   resource "azapi_update_resource" "skip_driver" {
  #     type        = "Microsoft.ContainerService/managedClusters/agentPools@2024-02-01"
  #     resource_id = azurerm_kubernetes_cluster_node_pool.gpu.id
  #     body = jsonencode({ properties = { gpuProfile = { installGPUDriver = false } } })
  #   }
}

resource "azurerm_kubernetes_cluster_node_pool" "compute" {
  count = var.enable_compute_pool ? 1 : 0

  name                  = var.compute_pool_name
  kubernetes_cluster_id = var.cluster_id
  vm_size               = var.compute_vm_size
  auto_scaling_enabled  = true
  min_count             = var.compute_min_count
  max_count             = var.compute_max_count
  zones                 = var.compute_zones
  vnet_subnet_id        = var.node_subnet_id
  node_labels           = { "node-type" = "compute" }
}

Teardown

az aks nodepool delete --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$GPU_POOL_NAME" --no-wait
az aks nodepool delete --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$COMPUTE_POOL_NAME" --no-wait

Validation checklist

• GPU pool name is 1–12 chars, lowercase alphanumeric (no hyphens) — AKS hard constraint
• Quota preflight runs before nodepool add (skippable but should print a warning, not silently bypass)
• Quota query uses an uppercase SKU prefix (e.g. NV, NC) — az vm list-usage returns uppercase quota names; lowercase prefix never matches
• GPU pool is single-zone if LLS will be deployed
• nvidia.com/gpu=true label is set on the GPU pool
• GPU pool has no taint by default (label-only reservation, matching the AWS foundation). A taint is opt-in via GPU_NODE_TAINTS and must be paired with matching workload tolerations — otherwise NVCF function pods can't schedule (NVCA capacity: 0)
• GPU_DRIVER_INSTALL=disabled path uses --skip-gpu-driver-install (and a TF azapi snippet is provided as a comment)
• Compute pool is gated by ENABLE_COMPUTE_POOL=true
• Both pools attach to NODE_SUBNET_ID from iam-vpc
• Idempotent (check before add)
• --teardown removes both pools

References

Topic	Source
AKS GPU node pools	https://learn.microsoft.com/azure/aks/gpu-cluster
--skip-gpu-driver-install	https://learn.microsoft.com/azure/aks/gpu-cluster#skip-gpu-driver-installation
GPU SKU families	https://learn.microsoft.com/azure/virtual-machines/sizes-gpu
Quota requests	https://learn.microsoft.com/azure/quotas/regional-quota-requests