observability-service-reliability

# Service Reliability Design reliability targets that people will actually act on, then operate them. This skill covers the judgment before the API call — which service-level indicator fits the data you have, what target is achievable rather than aspirational, whether an SLO is even the right instrument — and then the mechanics of creating, alerting on, resetting, and retiring SLOs through the Kibana API. Reliability instruments are not interchangeable. An SLO measures a user-visible outcome against a spendable budget; a threshold rule fires on a raw condition; an anomaly job finds deviations where no fixed threshold exists; a synthetics monitor is the only one of the four that can see a service that has stopped emitting telemetry entirely. Choosing wrong produces alerts that are technically correct and operationally useless. For diagnosing a service that is already degraded, and for the incident workflow itself, use the **observability-sre-triage** skill; for general rule lifecycle mechanics use the **kibana-alerting-rules** skill. <!-- begin-partial: preamble --> ## Environment Configuration This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds. This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly. <!-- end-partial: preamble --> ### Analysis without cluster access The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion. When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not. SLO and alerting operations run against Kibana and use the `kbn:` prefix (for example, `POST kbn:/api/observability/slos`); data validation runs against Elasticsearch with a bare path (for example, `POST /_query`). Kibana SLO commands are space-scoped — pass the space explicitly. For non-default spaces the HTTP path becomes `kbn:/s/<space_id>/api/observability/slos`. Full request-body schemas for every SLI type live in [references/slo-api-schemas.md](references/slo-api-schemas.md), the burn-rate rule schema in [references/burn-rate-rules.md](references/burn-rate-rules.md), and curated official documentation in [references/documentation.md](references/documentation.md). ## Jobs to be done - Translate a reliability concern into the right instrument: SLO, burn-rate rule, threshold rule, anomaly job, or synthetics monitor - Choose an SLI type for a given service and data shape, and validate the underlying fields before committing - Set a target, time window, and budgeting method that are defensible against measured history - Decide whether to group an SLO, and refuse high-cardinality grouping - Create, verify, update, reset, disable, and delete SLOs - Attach burn-rate alert rules with fast-burn and slow-burn windows routed to different severities - Keep the alerting surface signal-dense: remove duplicate coverage, snooze instead of disable, delete unactionable rules - Monitor availability of user-facing endpoints with synthetics and feed that into an availability SLI ## Output discipline Applies to every response produced under this skill. - **Commit to the best-supported conclusion.** Recommend one SLI design and defend it. Do not enumerate every SLI type with equal weight and leave the choice to the user — that is not a recommendation, it is a menu. - **State the target and window you chose, and why, once.** Do not restate the justification per bullet. - **Do not invent field names.** Verify every field against the mapping before writing it into an indicator. If a field cannot be confirmed, say so and stop; a plausible-looking field name is worse than an admitted gap. - **Do not speculate past the evidence.** If the measured history does not support a target, say what it does support. - **Report absence as absence.** Zero events means the data is missing or the service is not emitting; it never means the service is healthy. An SLI ratio computed over zero total events is undefined, not 100%. - **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered. - **End on the finding.** No trailing offers such as "want me to set this up?". Actionable follow-ups belong in a recommendations list, phrased as recommendations, not as questions. ## Process: route the reliability concern to an instrument Do this before designing anything. Most bad SLOs are threshold rules wearing a costume. 1. **Name the user-visible symptom.** Ask what a user or downstream consumer would notice and complain about. If the answer is a resource number rather than an experience — disk at 90%, heap climbing, replica lag — the concern is a capacity ceiling, not a reliability target. Route it to a threshold rule. 2. **Check whether coverage already exists.** List SLOs with `GET kbn:/api/observability/slos`, alerting rules with `GET kbn:/api/alerting/rules/_find`, and anomaly jobs with `GET /_ml/anomaly_detectors`. Adding a second instrument on a signal that is already covered is the most common source of duplicate pages. 3. **Pick the instrument.** | Concern | Instrument | Why, and what it costs | | ---------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A user-visible success or latency outcome you want to budget over weeks | SLO + burn-rate rule | Gives a spendable budget and history, and paces change velocity. Costs a transform, needs steady traffic, and is deliberately slow — even fast-burn uses a one-hour window. | | A hard bound with a known safe value and an immediate operator action | Threshold or custom rule | Fires within one schedule interval, no transform. But it has no budget and no memory, so it re-fires for as long as the condition holds. | | A signal with no fixed threshold, strong seasonality, or many entities | Anomaly detection job | Finds unknown-unknowns and adapts to seasonality. Needs weeks of history to be trustworthy and emits scores, not outcomes, so it is a poor pager. | | Reachability of a user-facing endpoint from outside your own telemetry | Synthetics monitor | The only instrument that detects a total outage. Costs a check budget (see the synthetics section) and cannot explain an internal partial failure. | | An internal job with no consumer contract, or a service still changing daily | None | An SLO with no owner and no stable baseline becomes permanently red and is then ignored, which is worse than no SLO. | 4. **Do not skip the outage case.** An SLI built from a service's own logs or traces cannot see the service disappear: with zero events there is no `total`, so the ratio is undefined and the SLO neither burns nor recovers. Pair every request-based SLO on a user-facing service with either a synthetics availability monitor or a no-data threshold rule. This is the one place where two instruments on one signal is correct rather than duplicative. ## Process: design the SLO 1. **Locate the data and confirm the fields exist.** Resolve the index pattern with `GET /_resolve/index/<pattern>`, then confirm every field the indicator will reference with `GET /<index>/_mapping` or `GET /_field_caps`. OTel-native data lives in `traces-*.otel-*`, `metrics-*.otel-*`, and `logs-*.otel-*`, with `service.name` populated on all three; APM aggregate metrics live in `metrics-service_summary.1m.otel-*`, `metrics-service_transaction.1m.otel-*`, `metrics-transaction.1m.otel-*`, and `metrics-service_destination.1m.otel-*`. Do not assume a status-code, duration, or outcome field exists because it is conventional — confirm it. A wrong field name produces an SLO that computes cleanly and means nothing; [How a missing field fails](#how-a-missing-field-fails) shows exactly how. Two field placements are worth knowing because they are commonly guessed wrong: `http.response.status_code` is on `traces-*.otel-*` and not on `logs-*.otel-*`, and `transaction.duration.us` is on `traces-*.otel-*` and not on the `metrics-*.1m.otel-*` rollups, which carry `transaction.duration.histogram` and `transaction.duration.summary` instead. Confirm both against the deployment in front of you rather than trusting this list. 2. **Validate the SLI with ES|QL before creating anything.** Run the good/total ratio over recent history with `POST /_query` so you know the measured baseline. This query is exploratory and is written in ES|QL, not KQL: ```esql FROM traces-*.otel-* | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 30 day | STATS total = COUNT(*), good = COUNT(*) WHERE http.response.status_code < 500 | EVAL achieved = good::DOUBLE / total ``` Against an OTel demo cluster this returns `total: 1393887`, `good: 1393887`, `achieved: 1.0`. That is the measured input to step 4, not a target to copy — a clean 30 days argues for a target below 100%, not at it. **HTTP status lives on traces, not on logs.** `http.response.status_code` is populated on `traces-*.otel-*`. OTel log records do not carry an HTTP status, so the field does not exist in `logs-*.otel-*` at all, and its coverage on traces is per-service: only spans emitted by an HTTP server carry it. Filtering `kind == "Server"` keeps the service's own inbound requests and drops the client spans that report the status of calls it made to others. Confirm coverage for the specific service before building an indicator on it: ```esql FROM traces-*.otel-* | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 24 hour | STATS spans = COUNT(*), with_status = COUNT(http.response.status_code) ``` On the same cluster `cart` returns `spans: 46420, with_status: 46420`, while `checkout` — a gRPC service — returns `spans: 4887, with_status: 0`. When `with_status` is 0 there is no HTTP status to budget. Use `sli.apm.transactionErrorRate`, or build the ratio on `event.outcome`, which is populated on every OTel span regardless of protocol: ```esql FROM traces-*.otel-* | WHERE service.name == "checkout" AND kind == "Server" AND @timestamp >= NOW() - 30 day | STATS total = COUNT(*), good = COUNT(*) WHERE event.outcome == "success" | EVAL achieved = good::DOUBLE / total ``` That returns `total: 147009`, `good: 146996`, `achieved: 0.9999`. Write `good` in the positive form (`== "success"`), not as a negation of `"failure"` — a null outcome must not count as good. This split is not specific to Serverless or to one demo application. The same measurement across every service on a Stack 9.4.4 cluster, over `traces-*.otel-*` with `kind == "Server"`: | Service | Server spans | `http.response.status_code` coverage | `event.outcome` coverage | | --------------- | ------------ | ------------------------------------ | ------------------------ | | catalog | 49,271 | 0% | 100% | | gateway | 37,773 | 100% | 100% | | orders | 37,754 | 100% | 100% | | payments | 34,710 | 100% | 100% | | recommendations | 34,692 | 0% | 100% | | shipping | 34,531 | 100% | 100% | Two of six services have no HTTP status at all while every one of the six has `event.outcome` on every span. An availability SLO built on `http.response.status_code < 500` for `catalog` would compute `good: 0` against `total: 49271` — an achieved SLI of 0%, a fully consumed error budget, and a burn-rate rule that pages continuously against a healthy service. Nothing in the API response or the SLO UI flags this; the numerator is simply always zero. Run the coverage check above for the specific service every time, and prefer `event.outcome` when you are writing one indicator to cover several services. Never substitute a plausible-looking status field for a missing one — see [How a missing field fails](#how-a-missing-field-fails) for what that costs. Also check that traffic is thick enough for a ratio to be meaningful. If the thinnest buckets carry only a handful of events, a single failure swings the SLI by whole percentage points and the SLO will be noise: ```esql FROM traces-*.otel-* | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 7 day | STATS events = COUNT(*) BY bucket = BUCKET(@timestamp, 1 hour) | SORT events ASC | LIMIT 10 ``` Bound this one explicitly. Without a `@timestamp` predicate it buckets the entire retention of the trace data streams, which is cheap on a demo cluster and expensive on a customer's. 3. **Choose the SLI type from the data shape, not from preference.** | SLI type | Use when | | ------------------------------ | ----------------------------------------------------------------------------------- | | `sli.kql.custom` | Raw logs or documents where good and total are expressible as filters over events | | `sli.metric.custom` | Pre-aggregated metric fields where good and total are equations over sums or counts | | `sli.metric.timeslice` | A metric compared against a threshold per slice, such as a p95 latency ceiling | | `sli.histogram.custom` | Histogram fields, using a range for good and a value count for total | | `sli.apm.transactionDuration` | APM transaction latency against a millisecond threshold | | `sli.apm.transactionErrorRate` | APM transaction success rate | | `sli.synthetics.availability` | Synthetics monitor uptime for a user-facing endpoint | Prefer an APM or synthetics type when it fits: they encode the service, environment, and transaction dimensions for you and stay correct when the underlying index layout changes. Reach for `sli.kql.custom` when the outcome is only visible in raw events, and for `sli.metric.custom` when the service already emits its own counters. 4. **Set a target you can meet.** Take the measured baseline from step 2 and set the target at or just below it, then ratchet upward once the service earns it. A target above the measured baseline burns the entire budget on day one, the SLO stays red permanently, and the team stops looking at it. `objective.target` is a decimal between 0 and 1 — `0.995`, not `99.5`. Sanity-check the target against the budget it implies over 30 days: | Target | Error budget over 30 days | | ------ | ------------------------- | | 99% | 7h 12m | | 99.5% | 3h 36m | | 99.9% | 43m 12s | | 99.95% | 21m 36s | | 99.99% | 4m 19s | If a single rolling deploy, a node restart, or one dependency blip costs more than the whole budget, the target is unachievable and should be rejected, not accepted with a caveat. 5. **Choose the time window.** `timeWindow.type` is `rolling` (`7d`, `30d`, `90d`) or `calendarAligned` (`1w`, `1M`). Rolling windows move continuously, so budget recovers gradually and burn-rate alerting stays meaningful — this is the default for anything operational. Calendar-aligned windows reset at the period boundary, which matches contractual or monthly-reporting language but produces a budget cliff on the first of the month. Use rolling for paging, and add a calendar-aligned SLO alongside it only when someone genuinely reports on calendar periods. 6. **Choose the budgeting method.** `occurrences` divides good events by total events across the whole window, so a high-traffic hour dominates and a quiet overnight outage barely registers. `timeslices` chops the window into slices, marks each slice good or bad against `objective.timesliceTarget`, and divides good slices by total slices, so every period counts equally. Choose `timeslices` when low-traffic periods matter or when the indicator is a threshold on an