recover-stuck-k8s-pod

name	recover-stuck-k8s-pod
description	Diagnose and safely recover stuck terminating Kubernetes pods on Marin CoreWeave clusters, especially node-bound GPU pods. Use for deletion hangs, suspected uninterruptible GPU/NCCL waits, node cordoning or reboot decisions, and force-deletion requests.

Recover a stuck Kubernetes pod

Treat a pod that remains beyond its deletion deadline as a stuck Kubernetes object. Determine whether the node still owns live work before changing it. Force deletion removes the API object; it does not prove that the process or GPU allocation is gone.

Guardrails

• Start with read-only inspection. Get explicit operator approval before cordoning, deleting workloads, rebooting a node, or changing Iris state.
• Never force-delete a node-bound GPU pod while its process might still be running. Cordon the node, quiesce Iris retries, and obtain provider-level proof of reboot completion first.
• Extract the canonical Iris attempt from the task container's IRIS_TASK_ID. Labels are sanitized or truncated display identifiers; never use them as targets for iris job kick.
• Do not use a broad kubectl drain --force. Inventory sibling pods, owners, disruption budgets, volumes, and local storage before approving collateral.
• Record whether the node was already cordoned. Uncordon only a node this procedure cordoned, and only after health checks pass.
• Do not restart an Iris cluster as part of pod recovery.

Read platform.coreweave.kubeconfig_path and kube_context from the canonical lib/iris/config/<cluster>.yaml before operating. Do not copy those values from an alert, an old incident, or this skill:

rg -n 'kubeconfig_path|kube_context' lib/iris/config/<cluster>.yaml

Set <kubeconfig>, <context>, <namespace>, <pod>, and <node> explicitly in every command. Do not rely on the current kubectl context.

Classify the object

Inspect the pod and its events:

kubectl --kubeconfig <kubeconfig> --context <context> \
  -n <namespace> get pod <pod> -o yaml
kubectl --kubeconfig <kubeconfig> --context <context> \
  -n <namespace> get events --field-selector involvedObject.name=<pod> \
  --sort-by=.lastTimestamp

The Grafana classification column chooses the first applicable case:

• invalid_timestamp: repair or escalate the malformed object metadata.
• finalizer: investigate and repair the owning controller. Do not reboot a node solely because an object has a finalizer.
• terminal: the workload finished; investigate API or runtime cleanup.
• unbound: no node owns the pod; investigate API or scheduler cleanup.
• node_cleanup: the nonterminal object is bound to a node with no finalizer. Continue with node-level recovery only for this case.

FailedKillPod or ExceededGracePeriod events strengthen the diagnosis, but events can expire. Missing logs do not identify a stuck termination, and Kubernetes/DCGM metrics do not reliably identify an NCCL collective. A process in D state or a kernel stack involving GPU/NCCL code is supporting evidence, not the alert condition.

Identify the Iris attempt and blast radius

Extract the exact attempt and inspect all workloads on the node:

kubectl --kubeconfig <kubeconfig> --context <context> \
  -n <namespace> get pod <pod> \
  -o jsonpath='{.spec.containers[?(@.name=="task")].env[?(@.name=="IRIS_TASK_ID")].value}{"\n"}'
kubectl --kubeconfig <kubeconfig> --context <context> \
  get pods --all-namespaces --field-selector spec.nodeName=<node> -o wide
kubectl --kubeconfig <kubeconfig> --context <context> \
  get node <node> -o jsonpath='{.spec.unschedulable}{"\n"}'

Also inspect owner references, pod disruption budgets, PVCs, emptyDir, and unmanaged pods. Split <attempt> (for example /user/job/0:3) at the task suffix and check current Iris state:

uv run iris --cluster=<cluster> job summary <job>

Before node recovery, choose an explicit retry policy with the operator:

• Stop the parent job with iris job stop <job> when all work must remain quiescent.
• Mark only the canonical attempt preempted with iris job kick <attempt> --state preempted when the operator accepts Iris scheduling its retry within the job's preemption budget.

Do not kick by pod label, and do not permit an immediate retry onto a node that has not yet been cordoned.

Isolate and try graceful cleanup

With approval, cordon first and then request an ordinary deletion:

kubectl --kubeconfig <kubeconfig> --context <context> cordon <node>
kubectl --kubeconfig <kubeconfig> --context <context> \
  -n <namespace> delete pod <pod> --wait=false

Wait for kubelet cleanup and inspect fresh events. If the object disappears, verify the process and GPU allocation are gone before considering the node healthy. If it remains and node cleanup is implicated, continue to reboot.

For additional evidence, an approved privileged node debug session can inspect the host under /host:

kubectl --kubeconfig <kubeconfig> --context <context> \
  debug node/<node> -it --image=ubuntu --profile=sysadmin

Do not kill arbitrary PIDs from the debug shell.

Reboot the node before force deletion

CoreWeave cwic is a prerequisite. If it is unavailable or authentication fails, stop and use the CoreWeave console or support; do not substitute force deletion.

command -v cwic
cwic auth login
cwic node get <node>
cwic node describe <node>
cwic node reboot --force --message "stuck terminating GPU pod <namespace>/<pod>" <node>

Use --force only when the graceful path has failed and the collateral impact is approved. A Kubernetes NotReady/Ready transition alone is not proof of a physical reboot. Wait for cwic node describe to show the provider operation completed and the node returned through self-test to production. Record the provider operation, boot identity when available, and affected sibling pods.

Use the Kubernetes node.kubernetes.io/out-of-service taint only when the node is confirmed powered off and the storage implications are understood. It can force detach volumes and carries data-corruption risk; it is not the normal path for a wedged GPU pod.

Remove stale state and restore service

Only after provider-confirmed reboot completion may the remaining API object be force-deleted:

kubectl --kubeconfig <kubeconfig> --context <context> \
  -n <namespace> delete pod <pod> --grace-period=0 --force

Verify that the original pod UID is gone, no duplicate Iris attempt is active, GPU allocatable capacity is restored, required daemonsets are healthy, and the node has passed its workload health checks. If this procedure cordoned a node that was initially schedulable, uncordon it only after those checks:

kubectl --kubeconfig <kubeconfig> --context <context> uncordon <node>

Record the timeline, commands, canonical attempt, node, provider operation, collateral workloads, and verification evidence in the incident or task log.

References

• Kubernetes pod termination
• kubectl delete force-deletion warning
• CoreWeave node reboot
• CoreWeave node status and self-test
• Kubernetes non-graceful node shutdown