observe

# Observe Transient, session-scoped monitoring and ad-hoc querying. Unlike `manage-alerts` (which creates a durable saved object in Kibana), `observe` polls in-process and returns once fired, once its window closes, or — in `now` / `table` mode — immediately. ## Modes ### Decision tree — pick based on tense FIRST, then on shape ``` Does the user phrase it as past or future? PAST tense / windowed lookback FUTURE tense / live monitoring ("what WAS / over the past N / ("watch / monitor / wait until / in the last N / how did X look") live-sample / for the NEXT N") │ │ ▼ ▼ Is it a single number or a series? Is there a threshold to fire on? │ │ ┌───┴────┐ ┌──┴──┐ single series yes no │ │ │ │ ▼ ▼ ▼ ▼ now table metric metric (with (live- condition) sample) Other: • "tell me when anything unusual fires" → anomaly (default) • "list X / which X are Y / top N" (any tense) → table • "page me whenever..." → use manage-alerts instead ``` | Mode | When to pick it | Blocks? | |------|-----------------|---------| | `now` | **Past-tense windowed scalar.** "What was X right now / over the past 60 seconds / in the last 15 minutes / average X for the last hour". Put the window inside the ES\|QL via `WHERE @timestamp > NOW() - <window>`. Single number out. | Returns immediately | | `table` | **Past-tense time series OR group-by.** "How did X look over the past hour" (with `BUCKET()` for the chart), "list X", "which X are Y", any group-by / top-N. The view auto-charts 2-column time-series tables. | Returns immediately | | `metric` | **Forward-looking only.** "Watch X", "wait until X drops", "live-sample X for the next 60s", "wake me when X exceeds Y". Polls live for `max_wait` seconds. **Never use this for past-tense / "what was X" queries** — it will block for `max_wait` seconds before returning a value, which is the opposite of what the user asked. | Polls for `max_wait` seconds (default 60s) | | `anomaly` (default) | "Tell me when anything unusual fires", open-ended monitoring | Until an anomaly fires or `max_wait` elapses | > ⚠️ **Most common mistake: picking `metric` for a past-tense query.** A prompt like "what was the frontend memory over the past 60 seconds?" is asking about data that already exists — it's a windowed lookup, not a request to wait 60 more seconds and watch. Always inspect the verb before choosing `metric`: > > - **was / were / averaged / hit / spiked / over the past / in the last / for the last** → `now` or `table`. NEVER `metric`. > - **watch / monitor / poll / wait until / wake me / for the next / until X happens / live-sample** → `metric`. > > If the user wants durable alerting ("page me whenever..."), use `manage-alerts` instead. > > ⚠️ **Don't use observe for health rollups.** "Show me the health of X", "status of the X > environment", "how is X doing" — even with a time qualifier like "over the past hour" — should > route to `apm-health-summary`, not observe. observe is for raw-metric / single-query investigations; > apm-health-summary returns the full rollup (degraded services, anomalies, pod resources). Pick > apm-health-summary whenever the user is asking a HEALTH question rather than a specific metric > question. ## Prerequisites | Mode | Requires | |------|----------| | `anomaly` | Elastic ML anomaly detection jobs | | `metric` | Any ES\|QL-queryable numeric field | | `now` | Any ES\|QL-queryable numeric field | | `table` | Any ES\|QL-queryable data | ## How to call observe ### Anomaly mode (default) ```json { "mode": "anomaly", "min_score": 75, "max_wait": 600, "namespace": "otel-demo" } ``` - `min_score`: 75 default (major+), 50 for minor inclusion, 90 for critical-only. - `max_wait`: generous (600s default). Returns immediately on trigger — long waits are free. - `namespace`: only if the user scopes to a K8s namespace. ### Metric mode — threshold condition ```json { "mode": "metric", "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.pod.name == \"frontend-7d4b8f9c5-x2k9m\" | STATS v = AVG(metrics.k8s.pod.memory.working_set)", "condition": "< 80000000", "description": "frontend pod memory working set", "max_wait": 300 } ``` Condition format: `<comparator> <threshold>` — valid comparators: `<`, `<=`, `>`, `>=`, `==`. ### Metric mode — live sample (no threshold) Omit `condition` and the tool live-samples for the full `max_wait` window — use for "show me a live chart of X" prompts. The view renders an accumulating sparkline. ```json { "mode": "metric", "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.namespace.name == \"oteldemo-esyox-default\" | STATS v = AVG(metrics.k8s.pod.memory.working_set)", "description": "oteldemo-esyox-default avg pod memory", "max_wait": 60 } ``` ### Now mode — single read ```json { "mode": "now", "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.namespace.name == \"oteldemo-esyox-default\" | STATS v = AVG(metrics.k8s.pod.memory.working_set)", "description": "current avg pod memory in oteldemo-esyox-default" } ``` **Past-tense windowed read** — when the user asks "what was X over the past N seconds/minutes/hours", put the window in the ES\|QL `WHERE` clause, not in `max_wait`. The tool returns immediately with the aggregate; nothing is polled. ```json { "mode": "now", "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.pod.name == \"frontend-7d4b8f9c5-x2k9m\" AND @timestamp > NOW() - 60 seconds | STATS v = AVG(metrics.k8s.pod.memory.working_set)", "description": "frontend memory, last 60 seconds (avg)" } ``` If the user wants the time series (not just one number), use `table` mode with `BUCKET()` instead — that returns rows the view can chart. ### Table mode — full ES|QL rows and columns Use when the query groups, lists, or returns mixed-type rows (strings + numbers + dates). `now` mode discards everything except the first numeric cell — `table` mode preserves the whole result. ```json { "mode": "table", "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE metrics.k8s.pod.memory.working_set IS NOT NULL | STATS avg_mem = AVG(metrics.k8s.pod.memory.working_set) BY resource.attributes.k8s.pod.name, resource.attributes.k8s.namespace.name | SORT avg_mem DESC | LIMIT 10", "description": "top 10 pods by memory" } ``` **Past-tense time series** — use `table` with `BUCKET()` to return one row per time slice the view can chart: ```json { "mode": "table", "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.pod.name == \"frontend-7d4b8f9c5-x2k9m\" AND @timestamp > NOW() - 60 seconds | STATS v = AVG(metrics.k8s.pod.memory.working_set) BY bucket = BUCKET(@timestamp, 5 second) | SORT bucket ASC", "description": "frontend memory · 60s · 5s buckets" } ``` Rows are capped at 50 by default. Prefer tightening the ES|QL with `LIMIT` / `SORT` over raising `row_cap` — very wide tables clog the context window. ## Picking the right index pattern Fields live where the data is emitted — ES|QL rejects queries that reference a field the target index doesn't map (`verification_exception`). Before writing the query, match the user's question to the right layer: | User asks about… | Index | Carries | |---|---|---| | Node / pod / namespace topology, resource usage | `metrics-kubeletstatsreceiver.otel*` or `metrics-*` | `k8s.node.name`, `k8s.pod.name`, `k8s.namespace.name`, `service.name` (via resource attrs), CPU/memory/fs gauges | | Service behavior — latency, errors, throughput, spans | `traces-*.otel-*` or `traces-apm*` | `service.name`, `transaction.duration.us`, `event.outcome`, `span.*` | | Log rate / log content | `logs-*` | `message`, `log.level`, `service.name` | | ML anomalies | `.ml-anomalies-*` | `record_score`, `by_field_value`, `partition_field_value` | Cross-layer questions ("which **node** runs the most **services**") need the index that carries **both** fields — that's almost always `metrics-*`, because OTel resource attributes propagate through the Collector, so metrics docs carry `k8s.node.name` *and* `service.name`. Trace indices (`traces-apm*`, `traces-*.otel-*`) don't carry infra attributes like `k8s.node.name` — don't reach for them when the question is about nodes. Example: "which node is running the most services" ``` FROM metrics-* | WHERE @timestamp > NOW() - 5m AND k8s.node.name IS NOT NULL AND service.name IS NOT NULL | STATS service_count = COUNT_DISTINCT(service.name) BY k8s.node.name | SORT service_count DESC | LIMIT 20 ``` ## Common query patterns These are the field paths this deployment's data actually uses — prefer them over guessing. > ⛔ **BEFORE WRITING ANY QUERY — check the field type first:** > > Look up the field you intend to aggregate in the tables below. If its type is `counter_long`, **stop — do not write `FROM`**. Use `TS` + `RATE()` as described in the "Counter fields" section. This is a hard syntax requirement: `FROM` is rejected for `counter_long` fields regardless of which aggregation function you use, and the error cannot be fixed by changing `AVG` to `MAX` or `SUM`. The source command must be `TS`. ### OTel Kubernetes (kubeletstats receiver) Index: `metrics-kubeletstatsreceiver.otel*` Each kubeletstats scrape emits **separate documents per metric** — a CPU doc, a memory doc, a network doc, etc. Always filter `WHERE <field> IS NOT NULL` for the field you're aggregating, otherwise most rows carry nulls for it. **Gauge fields — use `AVG` / `MAX` / `MIN`, never `SUM`:** | Signal | Field | Type | |--------|-------|------| | Pod memory working set | `k8s.pod.memory.working_set` | `long` (bytes) | | Pod memory RSS | `k8s.pod.memory.rss` | `long` (bytes) | | Pod memory available | `k8s.pod.memory.available` | `long` (bytes) | | Pod CPU usage | `k8s.pod.cpu.usage` | `double` (cores — 1.0 = one full core) | | Pod filesystem usage | `k8s.pod.filesystem.usage` | `long` (bytes) | | Node memory working set | `k8s.node.memory.working_set` | `long` (bytes) | | Node memory available | `k8s.node.memory.available` | `long` (bytes) | | Node CPU usage | `k8s.node.cpu.usage` | `double` (cores) | | Node filesystem usage | `k8s.node.filesystem.usage` | `long` (bytes) | > For **counter** fields (network I/O, network errors, uptime), see the "Counter fields" section below — these require `TS` + `RATE()`. **Fields that may NOT exist in every cluster's kubeletstats schema** — depends on whether the OTel collector / kubelet is configured to scrape pod-spec data: | Field | When it's missing | What to do | |---|---|---| | `metrics.k8s.pod.cpu.limit` | Pods don't declare CPU limits, OR the collector isn't scraping pod-spec | Don't reference it — use raw `metrics.k8s.pod.cpu.usage` and report cores instead of % utilization | | `metrics.k8s.pod.memory.limit` | Same as above for memory | Same — report bytes instead of % | | `metrics.k8s.container.restart_count` | Collector isn't scraping pod-status (this is COMMON) | Don't reference it. Use APM error rates or alert reasons as a proxy for "things crashing." There is no canonical alternative field name (`k8s.pod.restarts` is wrong — don't try it) | **Schema-aware error recovery.** When ESQL returns `Unknown column [X], did you mean [Y, Z]?`, read the suggestion and retry with one of `Y` or `Z` — don't blindly retry the same query or guess a different name from memory. **Dimension fields — use for filtering and `BY` grouping:** | Dimension | Unprefixed | Prefixed (equivalent) | |---|---|---| | Pod name | `k8s.pod.name` | `resource.attributes.k8s.pod.name` | | Namespace | `k8s.namespace.name` | `resource.attributes.k8s.namespace.name` | | Node name | `k8s.node.name` | `resource.attributes.k8s.node.name` | | Cluster name | `k8s.cluster.name` | (same) | Both forms work on `metrics-kubeletstatsreceiver.otel*`. Prefer the unprefixed form — it's shorter and also works on counter-field queries via `TS`. **Common recipes:** Top pods by memory (last 5m, across all namespaces): ``` FROM metrics-kubeletstatsreceiver.otel* | WHERE @timestamp > NOW() - 5 minutes AND k8s.pod.memory.working_set IS NOT NULL | STATS avg_mem = AVG(k8s.pod.memory.working_set), max_mem = MAX(k8s.pod.memory.working_set) BY k8s.pod.name, k8s.namespace.name | SORT max_mem DESC | LIMIT 20 ``` Which pods are on a specific node: ``` FROM metrics-kubeletstatsreceiver.otel* | WHERE @timestamp > NOW() - 5 minutes AND k8s.node.name == "<node>" AND k8s.pod.name IS NOT NULL | STATS last_seen = MAX(@timestamp) BY k8s.pod.name, k8s.namespace.name | SORT last_seen DESC ```