k8s-pod-diagnostic

name	k8s-pod-diagnostic
description	Diagnose Kubernetes Pod problems such as restarts, CrashLoopBackOff, OOMKilled, Pending, ImagePullBackOff, eviction, health-check failure, and abnormal container exits using Prometheus metrics, VictoriaLogs, and Loki Kubernetes events.

Kubernetes Pod Diagnostic

Use a four-step diagnosis: current state, history, incident window, conclusion. Prefer the bundled scripts before hand-written queries.

Primary Command

scripts/diagnose_pod_restart.sh -c prod-cluster -n <namespace> -p <pod>
scripts/diagnose_pod_restart.sh -c prod-cluster -n <namespace> -p <pod> --window 30

--window is minutes before/after the detected incident time. Default is 15.

Workflow

1. Check current state: phase, readiness, container states, restart count, exit code, resource usage.
2. Review history: restarts over 24h, resource trends, last terminated reason, repeated patterns.
3. Lock the incident window: recent restart or event time plus/minus 15 minutes by default.
4. Query evidence from metrics, events, and logs; then state root cause, confidence, remediation, and prevention.

Tools

# Current Pod state
scripts/check_pod.sh -c <cluster> -n <namespace> -p <pod>
scripts/check_pod.sh -c <cluster> -n <namespace> -p <pod> -f text

# Kubernetes events from Loki/event-exporter
scripts/query_events.sh -c <cluster> -n <namespace> -p <pod>
scripts/query_events.sh -c <cluster> -n <namespace> -p <pod> \
  --start 2026-01-21T10:00:00Z --end 2026-01-21T10:30:00Z

check_pod.sh uses Prometheus. query_events.sh uses https://loki.example.com by default.

Parameters

• diagnose_pod_restart.sh: -c/--cluster, -n/--namespace, -p/--pod, --window.
• check_pod.sh: -c/--cluster, -n/--namespace, -p/--pod, -f/--format, --prometheus.
• query_events.sh: -c/--cluster, optional -n/--namespace, optional -p/--pod, --start, --end, -l/--limit, -f/--format, --loki-url.

Evidence Guide

• OOMKilled or exit code 137: compare memory working set with limit, inspect OOM events and node pressure.
• CrashLoopBackOff: inspect last exit code, application logs, config/env/secret changes, and startup probes.
• ImagePullBackOff or ErrImagePull: check image name, tag, registry, pull secret, and network.
• Pending: inspect FailedScheduling, node resources, taints/tolerations, selectors, affinity, PVC binding.
• Evicted: inspect node MemoryPressure, DiskPressure, ephemeral storage, and kubelet eviction messages.
• Unhealthy or Killing: inspect liveness/readiness probe configuration and response latency.

Useful event reasons: BackOff, OOMKilling, Failed, FailedScheduling, Killing, TaintManagerEviction, Unhealthy.

Cluster Labels

• prod-cluster: production
• compute-cluster: compute
• tencent-cluster: Tencent TKE
• aliyun-cluster: Alibaba ACK

Skill Collaboration

• Use victorialogs-analyzer for application/system logs during the locked incident window.
• Use k8s-diagnostic when multiple Pods, nodes, or namespaces fail together.

References

Load only when needed:

• references/troubleshooting_flows.md: detailed flowcharts.
• references/common_problems.md: common causes and fixes.
• references/exit-codes.md: container exit codes.
• references/prometheus-metrics.md: metric names and query details.
• references/usage_examples.md: example investigations.
• references/skill_collaboration.md: cross-skill workflow.