Generates a Terraform module (AVM resource module) that creates an AKS Standard cluster with a system node pool, AAD + Azure RBAC, OIDC issuer + workload identity, Azure CNI Overlay (Cilium NetworkPolicy), API server VNet integration, and an optional corp-egress allowlist. Does NOT create the GPU node pool — that's `aks-nodes`. Use as Stage 02 of `aks-bootstrap`, after `iam-vpc`. Trigger when the user mentions "create AKS cluster", "aks-cluster", "AKS control plane", or asks to "spin up an AKS cluster".
التثبيت
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
Generates a Terraform module (AVM resource module) that creates an AKS Standard cluster with a system node pool, AAD + Azure RBAC, OIDC issuer + workload identity, Azure CNI Overlay (Cilium NetworkPolicy), API server VNet integration, and an optional corp-egress allowlist. Does NOT create the GPU node pool — that's `aks-nodes`. Use as Stage 02 of `aks-bootstrap`, after `iam-vpc`. Trigger when the user mentions "create AKS cluster", "aks-cluster", "AKS control plane", or asks to "spin up an AKS cluster".
AKS cluster (system pool + control plane)
What this skill produces
A Terraform module that creates one AKS Standard cluster with the system pool only. GPU pool and compute pool come from aks-nodes after this lands.
VNet integration + public FQDN, no IP allowlist by default; access gated by AAD + Azure RBAC (disable_local_accounts=true). IP allowlist is optional hardening.
Why Standard_D8ds_v5 (not the AVM default D4d_v5): the NVCF self-managed stack + caches don't fit on 4 vCPU / 16 GiB. See tf-conventions R2.
Why 3 nodes (not 1): the NVCF self-managed stack is a 3-replica HA design — OpenBao server runs a 3-node Raft cluster, Cassandra replicaCount: 3, NATS 3, and the OpenBao injector uses hard anti-affinity. A 1-node floor leaves these Pending and the OpenBao Raft join cannot form. Default the system pool to a floor of 3 so a fresh deploy runs the control plane HA without a manual scale-up. Drop to 1 only for a throwaway single-replica test (set SYSTEM_POOL_MIN_COUNT=1 and disable HA in the stack env).
Inputs
Source from streaming-env.sh and generated/.env.iam-vpc (written by the iam-vpc skill).
Required:
Variable
Description
Example
AZURE_SUBSCRIPTION_ID
Subscription
60762596-...
RESOURCE_GROUP
RG (from iam-vpc)
rg-nvcf-westus3
LOCATION
Region
westus3
CLUSTER_NAME
AKS cluster name (hyphens, lowercase)
aks-nvcf-validation-westus3
NODE_SUBNET_ID
from .env.iam-vpc
(resource ID)
API_SERVER_SUBNET_ID
from .env.iam-vpc
(resource ID)
CLUSTER_IDENTITY_ID
from .env.iam-vpc
(resource ID)
AUTHORIZED_IP_RANGES
Optional corp-egress CIDR(s) for the optional IP allowlist. Empty (default) = no allowlist; access is gated by AAD + Azure RBAC. NOT a laptop /32 (see tf-conventions R3)
(empty)
ADMIN_AAD_OBJECT_IDS
Comma-separated AAD object IDs granted kubectl cluster-admin (each gets "Cluster User Role" + "RBAC Cluster Admin" on the cluster scope). User or group object IDs both work. For your own: az ad signed-in-user show --query id -o tsv
<obj-id-1>,<obj-id-2>
The AVM module input parent_id (TF var resource_group_id) is the RG's full resource ID, not its name — construct it as /subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}.
kubelet_identity_object_id (for AcrPull role assignment in acr)
oidc_issuer_url (for federated credentials)
After terraform apply, write the downstream env handoff that later script stages source:
AKS_DIR="generated/${CLUSTER_NAME}/aks-cluster"mkdir -p generated
export KUBECONFIG="${KUBECONFIG:-/tmp/kubeconfig-${CLUSTER_NAME}}"
az account set --subscription "$AZURE_SUBSCRIPTION_ID"
az aks get-credentials --name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --overwrite-existing --file "$KUBECONFIG"command -v kubelogin >/dev/null || {
echo"ERROR: kubelogin not found. Install with: az aks install-cli" >&2
exit 1
}
kubelogin convert-kubeconfig -l azurecli
cat > generated/.env.aks-cluster <<EOF
export AKS_CLUSTER_ID="$(terraform -chdir="$AKS_DIR" output -raw cluster_id)"
export KUBELET_IDENTITY_OBJECT_ID="$(terraform -chdir="$AKS_DIR" output -raw kubelet_identity_object_id)"
export OIDC_ISSUER_URL="$(terraform -chdir="$AKS_DIR" output -raw oidc_issuer_url)"
export KUBECONFIG="$KUBECONFIG"
EOF
This handoff maps Terraform outputs into the env-file contract consumed by aks-nodes, acr, blob-storage, gpu-operator, and aks-validate. It does not create the AKS cluster.
Architectural rules
Defer to tf-conventions for the full set. The rules with cluster-side consequences:
Resource module, not pattern module (avm-res-containerservice-managedcluster, not avm-ptn-aks-production). Pattern module forces private_cluster_enabled = true and doesn't expose authorized_ip_ranges. Resource module exposes everything.
VNet integration + identity-gated public endpoint by default; optional corp-egress allowlist — if authorized_ip_ranges is set, use stable corp-egress CIDRs, not laptop /32. See tf-conventions R3.
Azure CNI Overlay + Cilium — pods get IPs from the synthetic POD_CIDR; only nodes get VNet IPs. App-port restrictions belong in K8s NetworkPolicies (not the Azure NSG; see R5).
AAD + Azure RBAC — aad_profile.enable_azure_rbac = true, disable_local_accounts = true. Because local accounts are disabled, kubectl access is only via Azure RBAC: the skill emits azurerm_role_assignments ("Azure Kubernetes Service Cluster User Role" for kubeconfig pull + "Azure Kubernetes Service RBAC Cluster Admin" for in-cluster authz) on the cluster scope for each id in ADMIN_AAD_OBJECT_IDS (works for users and groups). Without these the operator gets Forbidden on every kubectl call.
OIDC + workload identity — oidc_issuer_profile.enabled = true and security_profile.workload_identity.enabled = true. Federated credentials live with the consuming workload, not in this skill.
Single-AZ system pool when LLS will be deployed — Azure LB zonality must match GPU pool zonality, and the system pool LB carries cluster-internal Services. If LLS won't be deployed, set SYSTEM_POOL_ZONES=1,2,3.
API server access
The default is VNet integration + a public API FQDN with no IP allowlist, gated by identity: AAD authentication + Azure RBAC + disable_local_accounts=true (all enabled by this skill). The endpoint is reachable from anywhere but does nothing without a valid AAD token and an Azure-RBAC role assignment — "public endpoint, still authenticated," not open access. This matches Azure's default posture and removes the IP-allowlist toil that the laptop-/32 approach causes.
The ids in ADMIN_AAD_OBJECT_IDS are granted cluster-admin automatically by the skill (see Design principle 4). To grant additional people later — by identity, not by IP — assign them an Azure RBAC role on the cluster (prefer an AAD group over individual emails):
# 1. let them pull a kubeconfig (az aks get-credentials)
az role assignment create --assignee <user@org.com> \
--role "Azure Kubernetes Service Cluster User Role" --scope <cluster-resource-id>
# 2. in-cluster authorization — least-privilege: RBAC Reader / Writer / Admin / Cluster Admin
az role assignment create --assignee <user@org.com> \
--role "Azure Kubernetes Service RBAC Admin" --scope <cluster-resource-id>
Do not allowlist laptop /32s (R3): on corporate VPN the egress IP rotates, and every rotation silently knocks kubectl/helm off the allowlist → Unable to connect … EOF or, worse, a mid-operation http2: client connection lost that can corrupt a helm release. Re-running az aks update --api-server-authorized-ip-ranges per rotation is toil and racy. The identity gate above is the primary control; only add a network restriction if your security posture requires defense-in-depth.
Optional network-layer hardening (on top of the identity gate; pick at most one):
Allowlist the corporate VPN egress CIDR(s) — one stable range, not per-laptop /32s, so it survives per-session rotation. Requires knowing the VPN NAT egress block. Set via AUTHORIZED_IP_RANGES; toggle live with az aks update --api-server-authorized-ip-ranges <cidr> (restore) / "" (clear).
Private cluster (API server private endpoint) reached over VPN/ExpressRoute — removes the public FQDN entirely. Heaviest change (the AVM resource module supports private_cluster_enabled; note R1 — the pattern module forces it on); best end state for a long-lived cluster.
Bastion / jump host with a static IP in (or peered to) the cluster VNet — allowlist that one IP; operators ssh/SSM in. Stable, but another box to run.
For one-off ops from a rotating IP without touching any allowlist, az aks command invoke runs kubectl/helmserver-side in the cluster (no allowlist entry needed; slower, no kubectl logs -f/port-forward).
If you do keep an allowlist, make the per-IP fallback idempotent and safe: detect the lockout, add the current egress IP to the existing list (don't replace it), and never start a destructive helm op while connectivity is unverified (a rotation mid-helm upgrade corrupts release state).
Terraform snippet (reference)
terraform {
required_version = ">= 1.11" # AVM 0.5.4 submodules declare ~> 1.11
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = ">= 4.46.0, < 5.0.0" # required by AVM 0.5.4 (NOT azurerm 3.x)
}
azapi = {
source = "Azure/azapi"
version = "~> 2.4" # AVM 0.5.4 drives the cluster through azapi
}
}
}
provider "azurerm" {
features {}
}
module "aks" {
source = "Azure/avm-res-containerservice-managedcluster/azurerm"
version = "0.5.4" # exact pin per tf-conventions Override 1
# This is an AVM RESOURCE MODULE: inputs follow the AVM interface, NOT the
# azurerm_kubernetes_cluster resource schema (e.g. parent_id not
# resource_group_name; managed_identities not identity; default_agent_pool
# not default_node_pool).
name = var.cluster_name
parent_id = var.resource_group_id # the RG's FULL resource ID (from iam-vpc), not its name
location = var.location
enable_telemetry = false
kubernetes_version = var.kubernetes_version
enable_rbac = true # Kubernetes RBAC
disable_local_accounts = true
addon_profile_azure_policy = { enabled = true }
oidc_issuer_profile = { enabled = true }
security_profile = {
workload_identity = { enabled = true } # workload identity lives under security_profile in AVM
}
managed_identities = {
user_assigned_resource_ids = [var.cluster_identity_id] # user-assigned identity from iam-vpc
}
aad_profile = {
managed = true
enable_azure_rbac = true
admin_group_object_ids = var.admin_aad_object_ids # only GROUP ids take effect here (managed-AAD admin binding); individual USERS get access via the role assignments below
}
network_profile = {
network_plugin = "azure"
network_plugin_mode = "overlay"
network_dataplane = "cilium" # Azure CNI Powered by Cilium — REQUIRED when network_policy = "cilium", else AKS create fails with NetworkPolicyNotSupported / NetworkPolicyCiliumRequiresCiliumDataplane
network_policy = "cilium" # enforce NetworkPolicy via the Cilium dataplane above
pod_cidr = var.pod_cidr
service_cidr = var.service_cidr
dns_service_ip = var.dns_service_ip
outbound_type = "loadBalancer"
}
api_server_access_profile = {
enable_vnet_integration = true
subnet_id = var.api_server_subnet_id
enable_private_cluster = false
authorized_ip_ranges = var.authorized_ip_ranges # default [] = no allowlist (identity-gated); set CIDRs only for optional hardening
}
default_agent_pool = {
name = "system"
vm_size = var.system_pool_vm_size
vnet_subnet_id = var.node_subnet_id
enable_auto_scaling = true
min_count = var.system_pool_min_count
max_count = var.system_pool_max_count
availability_zones = var.system_pool_zones
# No CriticalAddonsOnly taint: the system pool is the only general-purpose pool by default
# (GPU pool floors at 0, compute pool off) and is sized for the HA control plane, so cluster
# services — gpu-operator control plane, KAI scheduler, the NVCF self-managed stack — schedule
# here directly without per-consumer tolerations. Add a dedicated untainted compute pool in
# aks-nodes before reserving the system pool for critical addons.
}
tags = var.tags
}
# Cluster-admin access. With disable_local_accounts = true + Azure RBAC, kubectl access is
# ONLY via Azure RBAC role assignments — admin_group_object_ids alone grants nothing to a
# USER, and an AAD admin group is optional. Grant each id in ADMIN_AAD_OBJECT_IDS (works for
# users AND groups; see tf-conventions R7):
# - "Cluster User Role" → lets them pull a kubeconfig (az aks get-credentials)
# - "RBAC Cluster Admin" → in-cluster cluster-admin authorization
# Without these, every kubectl call is "Forbidden ... User does not have access to the
# resource in Azure" after a clean create.
resource "azurerm_role_assignment" "cluster_user" {
for_each = toset(var.admin_aad_object_ids)
scope = module.aks.resource_id
role_definition_name = "Azure Kubernetes Service Cluster User Role"
principal_id = each.value
}
resource "azurerm_role_assignment" "rbac_cluster_admin" {
for_each = toset(var.admin_aad_object_ids)
scope = module.aks.resource_id
role_definition_name = "Azure Kubernetes Service RBAC Cluster Admin"
principal_id = each.value
}
output "cluster_id" { value = module.aks.resource_id }
output "kubelet_identity_object_id" { value = module.aks.kubelet_identity.objectId } # azapi raw output is camelCase
output "oidc_issuer_url" { value = module.aks.oidc_issuer_profile_issuer_url }
Teardown
Use terraform destroy on the aks-cluster module. This does NOT delete the RG, VNet, or subnets; use iam-vpc for those resources.
Validation checklist
authorized_ip_ranges defaults to [] (no allowlist; identity-gated) and is set only for optional network-layer hardening
API server VNet integration is enabled with a delegated /28 subnet
aad_profile.managed = true, aad_profile.enable_azure_rbac = true, and disable_local_accounts = true are all present
oidc_issuer_profile.enabled = true and security_profile.workload_identity.enabled = true are both present
An azurerm_role_assignment for "Azure Kubernetes Service RBAC Cluster Admin" (and "Cluster User Role") on the cluster scope is emitted for each ADMIN_AAD_OBJECT_IDS id — without it, disable_local_accounts=true leaves kubectl Forbidden
Inputs use the AVM interface, not azurerm_kubernetes_cluster resource arguments (parent_id/managed_identities/default_agent_pool, not resource_group_name/identity/default_node_pool)
Network plugin is Azure CNI Overlay + Cilium policy — network_profile sets bothnetwork_dataplane = "cilium" and network_policy = "cilium" (the policy requires the dataplane; omitting it fails AKS create with NetworkPolicyNotSupported)
System pool single-zone if LLS will be deployed; multi-zone otherwise
User-assigned identity from iam-vpc is attached to the cluster
After apply, generated/.env.aks-cluster maps Terraform outputs to AKS_CLUSTER_ID, KUBELET_IDENTITY_OBJECT_ID, OIDC_ISSUER_URL, and KUBECONFIG for downstream stages
No secrets, no image versions, no AAD object IDs hardcoded
