kubernetes-node-drain

name	kubernetes-node-drain
enabled	true
description	Use when performing kubernetes node drain — runbook for safely draining Kubernetes nodes for maintenance, upgrades, or decommissioning. Covers pod disruption budget validation, workload rescheduling, persistent volume handling, cordon and drain execution, and post-drain verification to ensure zero workload disruption.
required_connections	[{"prefix":"kubernetes","label":"Kubernetes Cluster"}]
config_fields	[{"key":"cluster_name","label":"Cluster Name","required":true,"placeholder":"e.g., prod-us-east-1"},{"key":"node_name","label":"Node Name","required":true,"placeholder":"e.g., ip-10-0-1-42.ec2.internal"},{"key":"drain_reason","label":"Drain Reason","required":true,"placeholder":"e.g., OS patching, instance type upgrade, decommission"}]
features	["DEVOPS","KUBERNETES"]

Kubernetes Node Drain Skill

Safely drain node {{ node_name }} in cluster {{ cluster_name }} for: {{ drain_reason }}.

Workflow

Phase 1 — Node Assessment

NODE INVENTORY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Node status: Ready / NotReady / SchedulingDisabled
[ ] Node labels and taints documented
[ ] Pods running on node:
    - Total pods: ___
    - DaemonSet pods (will not be evicted): ___
    - Pods with local storage: ___
    - Pods without PDB: ___
    - StatefulSet pods: ___
[ ] Resource utilization:
    - CPU: ___% allocated
    - Memory: ___% allocated
[ ] Persistent volumes attached to node: ___

Phase 2 — Pre-Drain Validation

CAPACITY AND SAFETY CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Remaining cluster capacity sufficient:
    - Available CPU after drain: ___ cores
    - Available memory after drain: ___ GB
    - Node count after drain: ___ / ___
[ ] Pod Disruption Budgets (PDBs) reviewed:
    - PDB-protected workloads: ___
    - All PDBs allow at least 1 disruption: [ ] YES  [ ] NO
[ ] No single-replica deployments without PDB: [ ] CONFIRMED
[ ] Anti-affinity rules will not block rescheduling: [ ] CONFIRMED
[ ] No critical CronJobs currently running on node: [ ] CONFIRMED

Phase 3 — Cordon Node

CORDON
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Execute: kubectl cordon {{ node_name }}
[ ] Verify node shows SchedulingDisabled
[ ] Confirm no new pods scheduled to node
[ ] Timestamp: ___

Phase 4 — Drain Execution

DRAIN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Execute drain command:
    kubectl drain {{ node_name }} \
      --ignore-daemonsets \
      --delete-emptydir-data \
      --grace-period=60 \
      --timeout=300s

[ ] Monitor pod evictions:
    - Pods evicted successfully: ___ / ___
    - Pods stuck in Terminating: ___
    - PDB violations encountered: ___

[ ] Handle stuck pods (if any):
    [ ] Investigate pod stuck reasons
    [ ] Force delete if safe: kubectl delete pod <name> --force
    [ ] Escalate if critical workload affected

[ ] Drain complete — timestamp: ___

Phase 5 — Post-Drain Verification

POST-DRAIN CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] All evicted pods rescheduled and Running on other nodes
[ ] No pods in Pending state due to capacity
[ ] Service endpoints updated (removed drained node)
[ ] Health checks passing for affected services
[ ] Persistent volumes reattached to new nodes (if applicable)
[ ] No increase in error rates across cluster
[ ] Monitoring shows stable request handling

Phase 6 — Node Maintenance and Restoration

POST-MAINTENANCE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
After maintenance is complete:
[ ] Node maintenance performed: {{ drain_reason }}
[ ] Uncordon node: kubectl uncordon {{ node_name }}
[ ] Verify node status: Ready
[ ] Verify pods can be scheduled to node
[ ] Node labels and taints reapplied (if needed)
[ ] OR — node decommissioned and removed from cluster

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams

Output Format

Produce a node drain report with:

1. Drain summary (node, cluster, reason, timestamps)
2. Pod eviction details (counts, rescheduling status)
3. Capacity impact (cluster utilization before and after)
4. Issues encountered (stuck pods, PDB violations, rescheduling failures)
5. Final status (DRAINED / RESTORED / DECOMMISSIONED)