observability-k8s-investigation

# Kubernetes Investigation Diagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the kube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify root cause across the workload, node, and control-plane layers. ## Scope **In scope:** OTel-receiver-namespaced indices (`metrics-kubeletstatsreceiver.otel-*`, `metrics-k8sclusterreceiver.otel-*`, `logs-k8seventsreceiver.otel-*`, `logs-k8sobjectsreceiver.otel-*`) and OTel semantic conventions (`k8s.pod.name`, `k8s.namespace.name`, `k8s.container.restarts`). **Out of scope:** - The legacy Elastic Agent Kubernetes integration (`metrics-kubernetes.*`, `logs-kubernetes.*`, `kubernetes.*` fields). Being deprecated — do not author queries against these paths. - APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain — once a K8s root cause is ruled in or out, APM investigation continues outside this skill. - Cluster provisioning, capacity planning, cost optimization. Different domain. ## Guidelines These apply to every investigation. When in doubt, re-read them before writing the synthesis. **Absence of evidence is not evidence. Do not confabulate from empty results.** If log queries return 0 rows, logs are likely not collected or the pod has no recent lines — this does _not_ mean "dependency unavailable" or any other specific failure mode. Report `no_logs_available` and weight remaining signals accordingly. **Empty dependency data ≠ upstream healthy.** Services without APM instrumentation (load generators, workers) emit no destination metrics. Report `insufficient_dependency_data`, not "upstreams OK." **Co-symptoms are not causes.** Two services degrading simultaneously usually share an upstream, not a causal link. Only attribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5× error rate, >3× latency). **OOMKilled ≠ memory leak by default.** The limit may simply be undersized for the workload's working set. Compare against a 7-day baseline at the same hour-of-day before claiming a leak. **Error-termination ≠ application bug by default.** Check `k8s.pod.cpu_limit_utilization` first. CFS throttling driving liveness probe timeouts is the most common misdiagnosis in this space. **Average CPU hides throttling.** A pod can look healthy at 40–60% average `cpu_limit_utilization` while being throttled severely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads hit quota mid-period and stall. Look at max and p95, not just average. **Restart count is boolean, not a counter.** `k8s.container.restarts` is pulled directly from the K8s API and may be pruned by the kubelet at any time, so the absolute value is unreliable. Treat it as `== 0` (no recent restarts) vs `> 0` (recently restarting); do not derive backoff timing or "linear vs exponential" patterns from it. Confirm the restart pattern via K8s `Killing` / `BackOff` events instead. **Prefer to report uncertainty over manufacturing confidence.** If the evidence is ambiguous, the synthesis should say so. Competing hypotheses are a valid output. ## Indices and fields ### Where to look | Signal | Index pattern | Use | | --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | | Pod/container runtime | `metrics-kubeletstatsreceiver.otel-*` | CPU, memory, network, filesystem. Utilization ratios. | | Cluster state | `metrics-k8sclusterreceiver.otel-*` | Restarts, phase, last-terminated reason, HPA, quota, node condition | | K8s events | `logs-k8seventsreceiver.otel-*` | Killing, BackOff, FailedScheduling, Evicted, image pull events | | K8s object snapshots | `logs-k8sobjectsreceiver.otel-*` | Deployment/service/configmap state over time | | Application logs | `logs-*.otel-*` | `body.text`, `severity_text`, filtered by `k8s.pod.name` | | APM | `traces-*.otel-*`, `metrics-service_*.otel-default` | Correlate via `service.name` + K8s resource attrs | | ML anomalies | `.ml-anomalies-*` | Memory-growth, restart-rate, throttle jobs (if configured) | ### Key fields Flat OTel paths work in ES|QL. Prefer the flat form for readability; the nested `resource.attributes.*` form is for raw log documents only. | Field | Index | What it is | | ------------------------------------------------ | --------------------------- | ------------------------------------------------------- | | `k8s.pod.name` | all k8s | Pod name | | `k8s.namespace.name` | all k8s | Namespace | | `k8s.container.name` | all k8s | Container within pod | | `k8s.deployment.name` | k8sclusterreceiver + others | Parent deployment | | `k8s.pod.phase` | k8sclusterreceiver | Pending=1/Running=2/Succeeded=3/Failed=4/Unknown=5 | | `k8s.container.restarts` | k8sclusterreceiver | Total container restart count | | `k8s.container.status.last_terminated_reason` | k8sclusterreceiver | `OOMKilled`, `Error`, `Completed`, `ContainerCannotRun` | | `k8s.pod.status_reason` | k8sclusterreceiver | Pod-level reason (`Evicted`, `NodeLost`) | | `k8s.pod.memory_limit_utilization` | kubeletstatsreceiver | 0.0–1.0+ (can exceed 1 transiently before OOM) | | `k8s.pod.cpu_limit_utilization` | kubeletstatsreceiver | 0.0–N (frequently >1 under CFS throttling) | | `k8s.pod.memory.usage` / `.working_set` | kubeletstatsreceiver | Bytes | | `k8s.node.condition_memory_pressure` | k8sclusterreceiver | 1 = pressure, 0 = ok | | `k8s.node.condition_ready` | k8sclusterreceiver | 0 = NotReady | | `k8s.hpa.current_replicas` / `.desired_replicas` | k8sclusterreceiver | HPA state | | `attributes.k8s.event.reason` | k8seventsreceiver | Event reason (filter on this) | | `body.text` | k8seventsreceiver / logs | Event message / log message | | `k8s.object.name` | k8seventsreceiver | involvedObject name (log attribute, use flat form) | ### Field availability Several fields above are off by default in stock kube-stack collectors and require explicit configuration. Verify presence before relying on them; if absent, fall back as noted and call out the substitution in the synthesis. | Field | Why it might be missing | Fall-back | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `k8s.container.status.last_terminated_reason` | Optional metric in k8sclusterreceiver; gated behind `metrics_collected.metadata` config. | Infer from K8s `Killing` / `OOMKilling` events in `logs-k8seventsreceiver.otel-*` and exit codes in app logs. | | `k8s.pod.status_reason` | Same — optional metric on k8sclusterreceiver. | Infer from events: `Evicted`, `NodeLost`, `Preempted`. | | `k8s.pod.cpu_limit_utilization` / `memory_limit_utilization` | Only emitted when the pod has the corresponding limit set, and the kubeletstatsreceiver metric is enabled. | Compute manually as `k8s.pod.cpu.usage / <limit>` from k8sclusterreceiver, or use absolute usage trending against a baseline. | | `k8s.node.condition_memory_pressure` | Gated behind k8sclusterreceiver `node_conditions_to_report` (default omits this). | Compare `k8s.node.memory.usage` against `k8s.node.allocatable_memory`, or look for `Evicted` events on the node. | If a fall-back is used, note it in the synthesis (e.g. `(via memory.usage; limit_utilization not collected)`) so the reader knows the signal is indirect. ## ES|QL gotchas Before writing queries, know these. Each of them silently produces wrong answers rather than failing loudly. **`VALUES()` returns scalar for single distinct value, array for multiple.** Templating that assumes array shape (e.g. `| first`) extracts the first character of the string when scalar. Use `MV_FIRST(VALUES(...))` or handle both. **`PERCENTILE` does not work on OTel `histogram` type** (as of 8.15). For APM duration percentiles, use `AVG` on the `aggregate_metric_double` summary field (`AVG(transaction.duration.summary)` divides sum by value_count). For true percentiles, fall back to Kibana Query DSL. **`COUNT(agg_metric_double)` returns `value_count` (events), not doc count.** `SUM(field)` gives the sum component; `AVG(field)` gives sum/value_count. Do not use `SUM(transaction.duration.summary)` as an event-count proxy — it returns total duration. **K8s metrics use flat OTel field paths in ES|QL.** `k8s.pod.name`, not `resource.attributes.k8s.pod.name`. The nested form is for raw log documents. ## Failure-mode taxonomy Vocabulary for classification, not a decision tree. Use the pivotal-signal column to recognize which mode you're looking at; use "Investigate" to know what else should corroborate. ### Workload layer | Mode | Pivotal signal | Investigate | | ----------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **OOMKilled** | `last_terminated_reason == "OOMKilled"` + `memory_limit_utilization → 1.0` | Monotonic rise (leak) vs. load-driven spike? Compare current trend to 7-day baseline. Check heap metrics (JVM, Go, Node) for GC pressure. |