gke-kueue-debugging

# GKE Kueue Workload & Admission Debugging Playbook > [!WARNING] > This skill is **experimental** and under active validation for Cluster Toolkit. Diagnostic queries are read-only, but remediation plans must be reviewed carefully by cluster operators before execution. Use this playbook when distributed AI/ML training jobs (PyTorchJob, RayCluster, JobSet, Job, LeaderWorkerSet) submitted to GKE remain indefinitely in `Pending`, `Inadmissible`, `Admitted: False`, or experience unexpected evictions. --- ## 1. Diagnostic Triage (Read-Only) ### Step 0: Fast-Path Workload Inspection (SRE First Response) If investigating a parent Job, JobSet, or RayCluster, locate its generated Workload name first: ```bash # Method A: Direct lookup via owner UID (safely queries installed controllers without failing on missing CRDs) JOB_UID=$(kubectl get job <PARENT_JOB_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.uid}' 2>/dev/null) if [ -z "$JOB_UID" ]; then for kind in jobset raycluster rayjob pytorchjob mpijob leaderworkerset; do JOB_UID=$(kubectl get "$kind" <PARENT_JOB_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.uid}' 2>/dev/null) [ -n "$JOB_UID" ] && break done fi [ -n "$JOB_UID" ] && kubectl get workloads -n <NAMESPACE> -l "kueue.x-k8s.io/job-uid=$JOB_UID" # Method B: Universal filter by parent job name prefix kubectl get workloads -n <NAMESPACE> | grep <PARENT_JOB_NAME> ``` Always inspect high-level workload events and conditions: ```bash kubectl describe workload <WORKLOAD_NAME> -n <NAMESPACE> ``` *Look directly at the bottom `Events:` section for `QuotaReserved`, `AdmissionCheckFailed`, `Inadmissible`, `Preempted`, or `WorkloadEvicted`.* --- ### Step 1: Query Workload Conditions, Evictions & LocalQueue Assignment Inspect workloads in the target namespace. Do NOT query Pods directly -- Kueue suspends Jobs before Pod creation (`.spec.suspend: true`). ```bash kubectl get workloads.kueue.x-k8s.io -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,ADMITTED:.status.conditions[?(@.type=="Admitted")].status,EVICTED:.status.conditions[?(@.type=="Evicted")].status,RESERVED_REASON:.status.conditions[?(@.type=="QuotaReserved")].reason,EVICTED_REASON:.status.conditions[?(@.type=="Evicted")].reason,MSG:.status.conditions[?(@.type=="QuotaReserved")].message' ``` *Condition Interpretation:* * `QuotaReserved: False`, `Reason: Pending` or `Inadmissible`: Workload is either waiting for quota (Step 3) OR failed flavor matching (Step 4). **Always inspect `MSG`**: - If `MSG` indicates `insufficient quota ... > maximum capacity`, proceed to Step 3. - If `MSG` indicates `doesn't match node affinity` or `untolerated taint`, proceed to Step 4 and Step 6. * `QuotaReserved: True`, `Admitted: False`: Quota is reserved, but the workload is gated by AdmissionChecks (e.g., GKE DWS Flex-start, TPU Dynamic Slicing). Inspect checks: ```bash kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .status.admissionChecks[*]}{"Check: "}{.name}{" | State: "}{.state}{" | RequeueAfter: "}{.requeueAfterSeconds}{"s | Message: "}{.message}{"\n"}{end}' ``` * `Evicted: True`: Workload was interrupted after admission. Check `EVICTED_REASON`: - `Preempted`: Evicted to reclaim quota or satisfy higher-priority job. Proceed to Step 5. - `PodsReadyTimeout`: Worker pods failed to become ready within `waitForPodsReady.timeout` (check pod image pulls / driver daemonsets). - `AdmissionCheck`: An admission check transitioned to Retry/Rejected while running. - `Deactivation`: Workload deactivated (`spec.active: false`) due to a rejected admission check. - `ClusterQueueStopped` / `LocalQueueStopped`: Queue was paused with `stopPolicy: HoldAndDrain`. --- ### Step 2: Verify LocalQueue & Upstream ClusterQueue Binding Verify that the LocalQueue exists, routes to an active ClusterQueue, and is not administratively stopped: ```bash kubectl get localqueue <QUEUE_NAME> -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,CLUSTER_QUEUE:.spec.clusterQueue,STOP_POLICY:.spec.stopPolicy,ACTIVE:.status.conditions[?(@.type=="Active")].status,REASON:.status.conditions[?(@.type=="Active")].reason,PENDING:.status.pendingWorkloads,RESERVING:.status.reservingWorkloads,ADMITTED:.status.admittedWorkloads' ``` *Root Cause Check:* * If `STOP_POLICY` is `Hold` or `HoldAndDrain`, the queue is inactive and will not admit new workloads. * If `ACTIVE` is False or the resource does not exist, the workload will remain pending indefinitely. * `RESERVING` indicates workloads that have passed quota checks but are awaiting AdmissionChecks. --- ### Step 3: Audit ClusterQueue Quotas, Cohorts, and Reservation vs. Usage Query the upstream `ClusterQueue` without dumping full YAML. Extract queue policy, nominal quotas, borrowing limits, reservations, and active usage: ```bash # 1. Inspect ClusterQueue policy, cohort, and strategy kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"ClusterQueue: "}{.metadata.name}{"\nCohort: "}{.spec.cohortName}{.spec.cohort}{"\nQueueingStrategy: "}{.spec.queueingStrategy}{"\nStopPolicy: "}{.spec.stopPolicy}{"\nPendingWorkloads: "}{.status.pendingWorkloads}{" | ReservingWorkloads: "}{.status.reservingWorkloads}{" | AdmittedWorkloads: "}{.status.admittedWorkloads}{"\n---\n"}{range .spec.resourceGroups[*].flavors[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{" Resource: "}{.name}{" | Nominal: "}{.nominalQuota}{" | BorrowingLimit: "}{.borrowingLimit}{"\n"}{end}{end}' # 2. Check live reservations vs. actual running pod usage kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"=== FLAVORS RESERVATION (Admitted + Reserving) ===\n"}{range .status.flavorsReservation[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{" Resource: "}{.name}{" | Reserved: "}{.total}{" | Borrowed: "}{.borrowed}{"\n"}{end}{end}{"=== FLAVORS USAGE (Actively Admitted Pods) ===\n"}{range .status.flavorsUsage[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{" Resource: "}{.name}{" | Used: "}{.total}{" | Borrowed: "}{.borrowed}{"\n"}{end}{end}' ``` *Root Cause Check:* * **Head-of-Line (HOL) Blocking**: If `QueueingStrategy` is `StrictFIFO`, an inadmissible older job at the head of the queue blocks all newer jobs across the cluster even if quota is free! Inspect oldest pending workloads for this queue: ```bash # In target namespace (sorted by creation timestamp to spot head-of-line blockers) kubectl get workloads -n <NAMESPACE> --sort-by=.metadata.creationTimestamp -o custom-columns='NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,CREATED:.metadata.creationTimestamp' | head -n 10 # Or across all namespaces sharing this ClusterQueue kubectl get workloads -A --sort-by=.metadata.creationTimestamp -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,CREATED:.metadata.creationTimestamp' | head -n 10 ``` * **Cohort Saturation**: If `borrowingLimit` is empty/null, borrowing is unlimited up to the cohort's total available quota. If `total >= nominalQuota` and cohort borrowing is exhausted, the queue must wait for active jobs to finish. For multi-tenant quota sharing architectures, refer to the [GKE Kueue Cohort Tutorial](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/kueue-cohort). * **Reservation vs. Usage Delta**: If `Reserved` is high but `Used` is 0, quota is tied up by workloads waiting on `AdmissionChecks` (e.g. DWS capacity acquisition). > [!NOTE] > **Tenant RBAC Fallback**: If querying `ClusterQueue` returns `Error from server (Forbidden)`, inspect the namespaced admission message directly: > ```bash > kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{.status.conditions[*].message}' > ``` --- ### Step 4: Verify ResourceFlavor Node Selectors, Taints, and Nodepools When `QuotaReserved: False` with `Reason: Pending` or `Inadmissible` (and `MSG` indicates "couldn't assign flavors" or "doesn't match node affinity"): ```bash # 1. Inspect ResourceFlavors in cluster (nodeLabels, nodeTaints, tolerations, topology) kubectl get resourceflavor -o jsonpath='{range .items[*]}{"Flavor: "}{.metadata.name}{"\nNodeLabels: "}{.spec.nodeLabels}{"\nNodeTaints: "}{.spec.nodeTaints}{"\nTolerations: "}{.spec.tolerations}{"\nTopologyName: "}{.spec.topologyName}{"\n\n"}{end}' # Or inspect a single flavor directly: # kubectl get resourceflavor <FLAVOR_NAME> -o jsonpath='{"Flavor: "}{.metadata.name}{"\nNodeLabels: "}{.spec.nodeLabels}{"\nNodeTaints: "}{.spec.nodeTaints}{"\nTolerations: "}{.spec.tolerations}{"\nTopologyName: "}{.spec.topologyName}{"\n"}' # 2. GKE accelerator node inspection (custom columns for GPU/TPU labels, taints, and nodepools) kubectl get nodes -o custom-columns='NAME:.metadata.name,NODEPOOL:.metadata.labels.cloud\.google\.com/gke-nodepool,GPU:.metadata.labels.cloud\.google\.com/gke-accelerator,TPU:.metadata.labels.cloud\.google\.com/gke-tpu-accelerator,TPU_TOPOLOGY:.metadata.labels.cloud\.google\.com/gke-tpu-topology,TAINTS:.spec.taints' | head -n 25 # For large clusters, target accelerator pools directly with label selectors: # kubectl get nodes -l 'cloud.google.com/gke-accelerator' -o custom-columns=... | head -n 25 # kubectl get nodes -l 'cloud.google.com/gke-tpu-accelerator' -o custom-columns=... | head -n 25 # 3. Inspect requested accelerator resource types in the workload's PodSets kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.podSets[*]}{"PodSet: "}{.name}{" | Count: "}{.count}{" | Requests: "}{.template.spec.containers[*].resources.requests}{"\n"}{end}' ``` *Verify that the workload's Pod template (or individual `spec.podSets[*]`):* * **Heterogeneous / Gang Scheduling**: For multi-podSet workloads (e.g., RayCluster with CPU head and GPU workers, or LeaderWorkerSet), check the `pod set <name>` prefix in `MSG` to identify which specific podSet failed flavor assignment. 1. **NVIDIA GPU Jobs (`nvidia.com/gpu`)**: Has toleration for GPU node taints using `operator: Exists` (e.g., `key: nvidia.com/gpu, operator: Exists, effect: NoSchedule`) and nodeSelector matching `cloud.google.com/gke-accelerator` (e.g., `nvidia-h100-80gb`, `nvidia-h100-mega-80gb`, `nvidia-l4`). If running on GKE Dynamic Workload Scheduler (DWS Flex) node pools, also verify toleration for `cloud.google.com/gke-queued=true:NoSchedule`. 2. **Google TPU Jobs (`google.com/tpu`)**: Has toleration for TPU node taints using `operator: Exists` (e.g., `key: google.com/tpu, operator: Exists, effect: NoSchedule`). Verify nodeSelector matches `cloud.google.com/gke-tpu-accelerator` (e.g. `tpu-v5p-slice`, `tpu-v6e-slice`) and topology matches `cloud.google.com/gke-tpu-topology` (e.g. `2x2x1` for 3D torus, `2x4` for 2D mesh). If running on GKE DWS Flex node pools, also verify toleration for `cloud.google.com/gke-queued=true:NoSchedule`. 3. Has `nodeSelector` / `nodeAffinity` compatible with the flavor's `nodeLabels`. --- ### Step 5: Audit Workload Priority, Flavor Fungibility & Preemption If a high-priority workload is blocked behind lower-priority workloads, inspect priority classes, fungibility, and preemption policies: ```bash # 1. Check Workload Priority and WorkloadPriorityClass kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{"PriorityClass: "}{.spec.priorityClassName}{" | Priority: "}{.spec.priority}{"\n"}' # 2. Check ClusterQueue Preemption and Flavor Fungibility kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"Preemption: "}{.spec.preemption}{"\nFlavorFungibility: "}{.spec.flavorFungibility}{"\n"}' ``` *Preemption Check:* * If `withinClusterQueue` is `Never`, high-priority workloads cannot preempt lower-priority workloads in the same queue. * If `reclaimWithinCohort` is `Never`, workloads cannot reclaim borrowed quota from peer ClusterQueues. * If `FlavorFungibility.whenCanPreempt` is `TryNextFlavor`, Kueue attempts to find available quota in fallback flavors before preempting running workloads. --- ### Step 6: Topology-Aware Scheduling (TAS) & AdmissionCheck Deep-Dive