aks-nodes

name

aks-nodes

description

Generates a bash or Terraform snippet that adds additional node pools to an existing AKS cluster — GPU pool (single-zone, NV/NC SKU) and optional compute pool (for LLS streaming-proxy or other CPU-heavy workloads). Use as Stage 03 of `aks-bootstrap`, after `aks-cluster`. Trigger when the user mentions "GPU node pool", "add node pool", "compute pool", "aks-nodes", or asks "how do I add a GPU pool to AKS".

AKS additional node pools (GPU + compute)

What this skill produces

Adds 0–2 additional node pools to an existing AKS cluster:

Pool	SKU (default)	Zone(s)	Labels	Purpose
GPU pool	Standard_NV36ads_A10_v5	["1"]	nvidia.com/gpu=true	NVCF function workloads (GPU)
Compute pool (optional)	Standard_D8ds_v5	["1"]	node-type=compute	LLS streaming-proxy (5 CPU / 24 GiB)

GPU pool MUST be single-zone if LLS will be deployed — Azure Standard LB zonality must match the GPU pool's zone (see nvcf-azure-deploy architecture rule #9 — or locally tf-conventions).

GPU SKU recommendations (validate quota per region; quota requests take 24–72h):

Use case	SKU	GPU
Validation (easy quota)	Standard_NV36ads_A10_v5	1× A10
L40S parity	Standard_NV18ads_L40S_v3	1× L40S
H100 prod-intent	Standard_NC40ads_H100_v5	1× H100

Inputs

Source from streaming-env.sh + generated/.env.iam-vpc + generated/.env.aks-cluster.

Required:

Variable	Description
AZURE_SUBSCRIPTION_ID	Subscription
RESOURCE_GROUP	RG
CLUSTER_NAME	Existing AKS cluster
NODE_SUBNET_ID	from .env.iam-vpc

Optional (firm defaults):

Variable	Description	Default
GPU_POOL_NAME	GPU pool name (1–12 chars, lowercase, alphanumeric only — NO hyphens; this is an AKS constraint, not project naming)	gpu
GPU_VM_SIZE	GPU SKU	Standard_NV36ads_A10_v5
GPU_NODE_COUNT	Initial GPU nodes (autoscaler floor)	0
GPU_MIN_COUNT	Autoscale floor	0
GPU_MAX_COUNT	Autoscale ceiling	2
GPU_ZONES	Single zone for the GPU pool	1
GPU_NODE_TAINTS	Comma-separated taints. Empty by default — the GPU pool is reserved via the nvidia.com/gpu=true label alone, matching the AWS foundation (aws-infra uses labels, no taint). A sku=gpu:NoSchedule taint blocks NVCF function pods (no matching toleration) → NVCA reports capacity: 0 → function deploy 400s. Only set this if you intend to wire matching tolerations onto every GPU workload.	(empty)
GPU_DRIVER_INSTALL	enabled (AKS installs driver) or disabled (GPU operator owns the driver lifecycle) — see "GPU driver source" below	enabled
ENABLE_COMPUTE_POOL	Create the compute pool?	false
COMPUTE_POOL_NAME	Compute pool name (same naming constraint as GPU)	compute
COMPUTE_VM_SIZE	Compute SKU	Standard_D8ds_v5
COMPUTE_MIN_COUNT	Autoscale floor	1
COMPUTE_MAX_COUNT	Autoscale ceiling	1
COMPUTE_ZONES	Compute zone(s) (single-zone to match LLS LB)	1
COMPUTE_NODE_LABELS	Pool labels	node-type=compute
OUTPUT_FORMAT	bash or terraform	bash

Outputs

• Bash: generated/stage03-<cluster-name>-aks-nodes.sh — idempotent (az aks nodepool add errors on exists; the script checks first).
• Terraform: generated/<cluster-name>/aks-nodes/main.tf containing azurerm_kubernetes_cluster_node_pool resources.

GPU driver source — pick one

AKS NV/NC SKUs pre-install the NVIDIA driver. The GPU Operator (from the gpu-operator skill) also wants to install one. Pick exactly one source:

GPU_DRIVER_INSTALL	What happens	When to use
enabled (default)	AKS installs the driver. GPU Operator runs with driver.enabled=false.	Validation; smallest blast radius.
disabled	Pass --skip-gpu-driver-install (Azure CLI flag; not in AVM resource module today — needs azapi patch or post-create az aks nodepool update). GPU Operator owns driver lifecycle with driver.enabled=true.	Production when NVCF runtime requires operator-managed drivers.

For default validation, leave at enabled.

Bash snippet (reference)

#!/bin/bash
set -euo pipefail

# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$REPO_ROOT/streaming-env.sh"
for envfile in .env.iam-vpc .env.aks-cluster; do
  if [[ ! -f "$REPO_ROOT/generated/$envfile" ]]; then
    echo "ERROR: $REPO_ROOT/generated/$envfile not found. Run earlier stages first." >&2
    exit 1
  fi
  source "$REPO_ROOT/generated/$envfile"
done

: "${CLUSTER_NAME:?}" "${RESOURCE_GROUP:?}" "${NODE_SUBNET_ID:?}"
GPU_POOL_NAME="${GPU_POOL_NAME:-gpu}"
GPU_VM_SIZE="${GPU_VM_SIZE:-Standard_NV36ads_A10_v5}"
GPU_NODE_COUNT="${GPU_NODE_COUNT:-0}"
GPU_MIN_COUNT="${GPU_MIN_COUNT:-0}"
GPU_MAX_COUNT="${GPU_MAX_COUNT:-2}"
GPU_ZONES="${GPU_ZONES:-1}"
GPU_NODE_TAINTS="${GPU_NODE_TAINTS:-}"   # empty = no taint (AWS-parity); label-only GPU-pool reservation
GPU_DRIVER_INSTALL="${GPU_DRIVER_INSTALL:-enabled}"
ENABLE_COMPUTE_POOL="${ENABLE_COMPUTE_POOL:-false}"

az account set --subscription "$AZURE_SUBSCRIPTION_ID"

# Quota preflight — surface failures BEFORE the slow nodepool add.
# Derive a 2-char UPPERCASE prefix from the SKU family (e.g. Standard_NV36ads_A10_v5 → "NV")
# because `az vm list-usage` returns uppercase quota names.
echo "--- Quota check for $GPU_VM_SIZE in $LOCATION ---"
SKU_PREFIX=$(echo "$GPU_VM_SIZE" | awk -F_ '{print toupper(substr($2,1,2))}')
az vm list-usage --location "$LOCATION" \
  --query "[?contains(name.value, '$SKU_PREFIX')].{name:name.localizedValue, limit:limit, current:currentValue}" \
  -o table || echo "WARN: quota lookup failed; proceeding anyway"

# GPU pool — idempotent add
if az aks nodepool show --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$GPU_POOL_NAME" &>/dev/null; then
  echo "GPU pool $GPU_POOL_NAME already exists; skipping add"
else
  DRIVER_FLAG=""
  if [[ "$GPU_DRIVER_INSTALL" == "disabled" ]]; then
    DRIVER_FLAG="--skip-gpu-driver-install"
  fi
  # Taint is opt-in and empty by default (AWS-parity). Only pass --node-taints when set,
  # otherwise the pool is reserved by the nvidia.com/gpu=true label alone.
  TAINT_FLAG=""
  if [[ -n "$GPU_NODE_TAINTS" ]]; then
    TAINT_FLAG="--node-taints $GPU_NODE_TAINTS"
  fi
  echo "--- Adding GPU node pool $GPU_POOL_NAME ($GPU_VM_SIZE, zone $GPU_ZONES) ---"
  az aks nodepool add \
    --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" \
    --name "$GPU_POOL_NAME" --node-vm-size "$GPU_VM_SIZE" \
    --node-count "$GPU_NODE_COUNT" \
    --enable-cluster-autoscaler --min-count "$GPU_MIN_COUNT" --max-count "$GPU_MAX_COUNT" \
    --zones $GPU_ZONES \
    $TAINT_FLAG \
    --labels nvidia.com/gpu=true \
    --vnet-subnet-id "$NODE_SUBNET_ID" \
    $DRIVER_FLAG \
    --only-show-errors
fi

# Optional compute pool
if [[ "$ENABLE_COMPUTE_POOL" == "true" ]]; then
  COMPUTE_POOL_NAME="${COMPUTE_POOL_NAME:-compute}"
  if az aks nodepool show --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$COMPUTE_POOL_NAME" &>/dev/null; then
    echo "Compute pool $COMPUTE_POOL_NAME already exists; skipping add"
  else
    echo "--- Adding compute node pool $COMPUTE_POOL_NAME ---"
    az aks nodepool add \
      --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" \
      --name "$COMPUTE_POOL_NAME" \
      --node-vm-size "${COMPUTE_VM_SIZE:-Standard_D8ds_v5}" \
      --node-count "${COMPUTE_MIN_COUNT:-1}" \
      --enable-cluster-autoscaler --min-count "${COMPUTE_MIN_COUNT:-1}" --max-count "${COMPUTE_MAX_COUNT:-1}" \
      --zones "${COMPUTE_ZONES:-1}" \
      --labels "${COMPUTE_NODE_LABELS:-node-type=compute}" \
      --vnet-subnet-id "$NODE_SUBNET_ID" \
      --only-show-errors
  fi
fi

kubectl get nodes -o wide

Terraform snippet (reference)

resource "azurerm_kubernetes_cluster_node_pool" "gpu" {
  name                  = var.gpu_pool_name
  kubernetes_cluster_id = var.cluster_id
  vm_size               = var.gpu_vm_size
  node_count            = var.gpu_node_count
  auto_scaling_enabled  = true
  min_count             = var.gpu_min_count
  max_count             = var.gpu_max_count
  zones                 = var.gpu_zones
  vnet_subnet_id        = var.node_subnet_id
  node_labels           = { "nvidia.com/gpu" = "true" }
  # No taint by default — label-only reservation, matching the AWS foundation.
  # A sku=gpu:NoSchedule taint blocks NVCF function pods (no matching toleration) → capacity:0.
  # Set node_taints only if you also wire matching tolerations onto every GPU workload.
  node_taints           = var.gpu_node_taints   # declare with default [] in your variables.tf

  # GPU_DRIVER_INSTALL = "disabled" needs an azapi resource (not exposed by azurerm today):
  #   resource "azapi_update_resource" "skip_driver" {
  #     type        = "Microsoft.ContainerService/managedClusters/agentPools@2024-02-01"
  #     resource_id = azurerm_kubernetes_cluster_node_pool.gpu.id
  #     body = jsonencode({ properties = { gpuProfile = { installGPUDriver = false } } })
  #   }
}

resource "azurerm_kubernetes_cluster_node_pool" "compute" {
  count = var.enable_compute_pool ? 1 : 0

  name                  = var.compute_pool_name
  kubernetes_cluster_id = var.cluster_id
  vm_size               = var.compute_vm_size
  auto_scaling_enabled  = true
  min_count             = var.compute_min_count
  max_count             = var.compute_max_count
  zones                 = var.compute_zones
  vnet_subnet_id        = var.node_subnet_id
  node_labels           = { "node-type" = "compute" }
}

Teardown

az aks nodepool delete --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$GPU_POOL_NAME" --no-wait
az aks nodepool delete --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$COMPUTE_POOL_NAME" --no-wait

Validation checklist

• GPU pool name is 1–12 chars, lowercase alphanumeric (no hyphens) — AKS hard constraint
• Quota preflight runs before nodepool add (skippable but should print a warning, not silently bypass)
• Quota query uses an uppercase SKU prefix (e.g. NV, NC) — az vm list-usage returns uppercase quota names; lowercase prefix never matches
• GPU pool is single-zone if LLS will be deployed
• nvidia.com/gpu=true label is set on the GPU pool
• GPU pool has no taint by default (label-only reservation, matching the AWS foundation). A taint is opt-in via GPU_NODE_TAINTS and must be paired with matching workload tolerations — otherwise NVCF function pods can't schedule (NVCA capacity: 0)
• GPU_DRIVER_INSTALL=disabled path uses --skip-gpu-driver-install (and a TF azapi snippet is provided as a comment)
• Compute pool is gated by ENABLE_COMPUTE_POOL=true
• Both pools attach to NODE_SUBNET_ID from iam-vpc
• Idempotent (check before add)
• --teardown removes both pools

References

Topic	Source
AKS GPU node pools	https://learn.microsoft.com/azure/aks/gpu-cluster
--skip-gpu-driver-install	https://learn.microsoft.com/azure/aks/gpu-cluster#skip-gpu-driver-installation
GPU SKU families	https://learn.microsoft.com/azure/virtual-machines/sizes-gpu
Quota requests	https://learn.microsoft.com/azure/quotas/regional-quota-requests