SRE debugging methodology for Kubernetes incident investigation, root cause analysis,
and failure diagnosis.
Use when: (1) Pods not starting, stuck, or failing (CrashLoopBackOff, ImagePullBackOff, OOMKilled, Pending),
(2) Debugging Kubernetes errors or investigating "why is my pod...", (3) Service degradation or unavailability,
(4) Root cause analysis for any Kubernetes incident, (5) Network policy blocking traffic,
(6) Stalled HelmReleases or Flux failures that need troubleshooting.
Triggers: "pod not starting", "pod stuck", "CrashLoopBackOff", "ImagePullBackOff", "OOMKilled",
"Pending pod", "why is my pod", "kubernetes error", "k8s error", "service not available",
"can't reach service", "debug kubernetes", "troubleshoot k8s", "what's wrong with my pod",
"deployment not working", "helm install failed", "flux not reconciling", "root cause",
"5 whys", "incident", "network policy blocking", "hubble dropped", "stalled helmrelease",
"live not updating", "promotion pipeline stuck", "artifact not promoted"
Instrucciones de origen · Vista previa de solo lectura
name
sre
description
SRE debugging methodology for Kubernetes incident investigation, root cause analysis,
and failure diagnosis.
Use when: (1) Pods not starting, stuck, or failing (CrashLoopBackOff, ImagePullBackOff, OOMKilled, Pending),
(2) Debugging Kubernetes errors or investigating "why is my pod...", (3) Service degradation or unavailability,
(4) Root cause analysis for any Kubernetes incident, (5) Network policy blocking traffic,
(6) Stalled HelmReleases or Flux failures that need troubleshooting.
Triggers: "pod not starting", "pod stuck", "CrashLoopBackOff", "ImagePullBackOff", "OOMKilled",
"Pending pod", "why is my pod", "kubernetes error", "k8s error", "service not available",
"can't reach service", "debug kubernetes", "troubleshoot k8s", "what's wrong with my pod",
"deployment not working", "helm install failed", "flux not reconciling", "root cause",
"5 whys", "incident", "network policy blocking", "hubble dropped", "stalled helmrelease",
"live not updating", "promotion pipeline stuck", "artifact not promoted"
user-invocable
false
Cluster access (--context patterns) and internal service URLs are in the k8s skill.
Debugging Kubernetes Incidents
Core Principles
5 Whys Analysis — NEVER stop at symptoms. Ask "why" until you reach the root cause.
Multi-Source Correlation — Combine logs, events, metrics for a complete picture.
Zero Alert Tolerance — Every firing alert must be addressed: fix the root cause, or as a last resort, create a declarative Silence CR with justification. Never ignore or defer.
The 5 Whys Analysis (CRITICAL)
Apply 5 Whys before concluding any investigation. Stopping at symptoms leads to ineffective fixes.
Example:
Symptom: Helm install failed with "context deadline exceeded"
Why #1: Pods never became Ready
Why #2: Pods stuck in Pending state
Why #3: PVCs couldn't bind (StorageClass "fast" not found)
Why #4: longhorn-storage Kustomization failed to apply
Why #5: numberOfReplicas was integer instead of string
ROOT CAUSE: YAML type coercion issue
FIX: Use properly typed variable for StorageClass parameters
Common self-healed causes: missing Secret/ConfigMap (ExternalSecret eventually created it),
missing CRD, transient image pull failure, temporary resource quota exceeded. Ensure proper
dependsOn ordering to prevent recurrence.
Promotion Pipeline Debugging
Symptom: "Live cluster not updating after merge"
Walk through each stage in order — see investigation-guide.md for
the failure mode table. Quick diagnostic flow:
1. PR merged → did build-platform-artifact.yaml trigger?
└─ If not: was kubernetes/ modified? (paths filter)
2. OCI artifact in GHCR with stable X.Y.Z tag?
└─ flux list artifact oci://ghcr.io/<repo>/platform | grep <short-sha>
└─ gh release list — a GitHub Release is created per build
3. Live OCIRepository seeing the new version?
└─ kubectl --context live get ocirepository -n flux-system
└─ Semver constraint must be ">= 0.0.0" and the tag strictly higher than current
4. Live Kustomization healthy?
└─ flux --context live get kustomizations -n flux-system
5. HelmReleases healthy / canary firing?
└─ flux --context live get helmreleases -A
└─ kubectl --context live get canaries -n monitoring
The pipeline is direct build-to-live — artifacts are tagged stable at build time; there is no
integration validation gate or promotion workflow. See docs/architecture/promotion-pipeline.md
for full architecture and rollback procedures.
Keywords
kubernetes, debugging, crashloopbackoff, oomkilled, pending, root cause analysis, 5 whys, incident investigation, pod logs, events, troubleshooting, network policy, hubble, stalled helmrelease, promotion pipeline, live not updating
