observability

# Observability You are wiring the *inside* view of a service: when something breaks at 3am, an engineer must be able to answer "what happened, where, and why" from telemetry alone — without adding a `console.log` and redeploying into the fire. This skill emits a concrete artifact: SDK init code, instrumentation (spans/metrics/structured logs), a Collector config, and alert rules that the instrumentation makes possible. The outside-in half — is it up, who gets paged — is `../monitoring/SKILL.md`. ## The one rule **Every signal carries the same correlation identity: `trace_id`, `service.name`, `deployment.environment`.** A log line you cannot pivot to its trace, or a spiking metric you cannot pivot to an exemplar span, doubles your mean-time-to-resolution — you are back to grepping. Three signals that don't share keys are three disconnected tools; three that do are one queryable system. Set the resource once at SDK init, inject `trace_id`/`span_id` into every log, and never emit a metric you can't tie back to a service and environment. ## Start from the operator's questions Before choosing signals, write **two to four questions on-call must answer during the likely incident**. For example: “Are payment retries recovering?”, “Which dependency and failure class drives exhaustion?”, “Can one payment be charged twice?”, “Which customer-visible operations need intervention now?” Then assign the cheapest signal: metrics say **that/how much**, traces show **where/causal path**, logs explain **why for this event**. If a proposed event or label answers none of the questions, do not emit it. ## The three pillars — when each earns its place Don't emit all three of everything. Each signal answers a different question at a different cost. | Signal | Answers | Cost | Alert on it? | Main gotcha | |---|---|---|---|---| | **Logs** | "what exactly happened in this one event" | high per-event, cheap to skip | rarely (noisy) | high-cardinality fields belong in the *body*, not in stream labels | | **Metrics** | "what's the rate/aggregate over time" | cheap, pre-aggregated | yes — this is your alert source | cardinality explosion if a label is unbounded | | **Traces** | "what was the causal path across hops, and where did time go" | medium; sample it | indirectly (via derived RED metrics) | one giant span = no causality; sample or you pay for noise | The "fourth pillar," **continuous profiling** (CPU/heap flame graphs over time), is now a first-class OTel signal — reach for it only when traces say "the time is inside *this* function" and you need to know which line. ## Architecture Instrument once, route anywhere. The app talks OTLP to a Collector; the Collector fans the firehose out to backends. ```text ┌─────────────┐ OTLP/gRPC :4317 ┌───────────────────────────┐ │ app + SDK │ OTLP/HTTP :4318 ───▶ │ OTel Collector │ │ (resource: │ /v1/traces │ receivers → processors │ │ service.name│ /v1/metrics │ → exporters (per signal) │ │ +env+ver) │ /v1/logs │ wired in service.pipelines│ └─────────────┘ └───────────────────────────┘ │ │ │ logs │ traces │ metrics│ ▼ ▼ ▼ Loki Tempo Mimir/Prom (+ Grafana to view) └── or a single vendor: Datadog / Honeycomb ──┘ ``` OTLP is the wire format: **gRPC on 4317** (TLS + gzip by default), **HTTP on 4318** with per-signal paths `/v1/traces`, `/v1/metrics`, `/v1/logs`. Always export to a Collector, never straight to the vendor. Why: the Collector gives you one place to **batch** (fewer round-trips), **retry** (survive a backend blip), **redact** PII, and **swap or add a backend without redeploying the app**. App SDKs should be dumb pipes; policy lives in the Collector. ## Instrument in the right order: auto first, manual second **Never hand-roll a span for something auto-instrumentation already covers** (HTTP servers, DB clients, queues). You will miss edges and waste effort. Turn on zero-code instrumentation, confirm traces flow, *then* add manual spans only where your business logic lives. ```bash # Node — zero-code, no app changes. SDK is 2.0+; the register hook is compatible. npm i @opentelemetry/api @opentelemetry/auto-instrumentations-node OTEL_SERVICE_NAME=checkout-api \ OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.4.2 \ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \ node --require '@opentelemetry/auto-instrumentations-node/register' app.js ``` ```bash # Python — zero-code via the launcher; it patches known libraries on import. pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-bootstrap -a install OTEL_SERVICE_NAME=checkout-api \ OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.4.2 \ opentelemetry-instrument python app.py ``` Then add a manual span only around a meaningful business operation — and give it attributes and a status, or it tells you nothing: ```javascript // Bad — a span with no attributes and no status. You learn that "something ran." const span = tracer.startSpan('work'); await chargeCard(order); span.end(); ``` ```javascript // Good — named for the business op, carries the inputs you'd filter by, records outcome. const { trace, SpanStatusCode } = require('@opentelemetry/api'); const tracer = trace.getTracer('checkout'); await tracer.startActiveSpan('charge_card', async (span) => { span.setAttribute('order.id', order.id); // searchable dimension span.setAttribute('payment.provider', 'stripe'); try { await chargeCard(order); span.setStatus({ code: SpanStatusCode.OK }); } catch (err) { span.recordException(err); // attaches the stack as an event span.setStatus({ code: SpanStatusCode.ERROR, message: err.message }); throw err; } finally { span.end(); // a span you never end leaks forever } }); ``` Per-stack init (Node SDK 2.0 manual setup, Go SDK, context propagation across HTTP/queue hops, the GenAI span template) lives in `references/instrumentation-recipes.md`. ## Resource and semantic conventions **Use the standard attribute names; never invent bespoke keys.** The whole correlation story and every prebuilt backend dashboard assume `http.*`, `db.*`, `gen_ai.*`, `service.*`. A homegrown `mycompany.endpoint` attribute is invisible to every tool that expects `http.route`. - Set on the resource, once: `service.name` (required — unset means telemetry lands as `unknown_service`), `service.version`, `deployment.environment`. - Kubernetes attributes (`k8s.*`) reached release candidate (2026-03); DB conventions are on their 2nd RC. Prefer the standard names even while an area is still stabilizing. - For LLM calls, **GenAI conventions exited experimental for client spans (early 2026)** — use `gen_ai.*` so the same span feeds your spend view: ```javascript // LLM call as a span: model + token attrs. These attributes feed ../cost-tracking/SKILL.md. await tracer.startActiveSpan('chat gpt-4o', async (span) => { span.setAttribute('gen_ai.system', 'openai'); span.setAttribute('gen_ai.request.model', 'gpt-4o'); const res = await openai.chat.completions.create({ /* ... */ }); span.setAttribute('gen_ai.usage.input_tokens', res.usage.prompt_tokens); span.setAttribute('gen_ai.usage.output_tokens', res.usage.completion_tokens); span.end(); }); ``` This skill emits the *signal* (token attributes on a span). Turning those tokens into a dollar figure and a budget is `../cost-tracking/SKILL.md`. ## Structured, correlated logs **Logs are JSON, carry the trace context, and never carry PII.** Free-text `print` lines cost you twice: you can't query them, and you can't jump from the log to its trace. ```python # Bad — unstructured, unsearchable, unlinkable to a trace. print("charged user " + email + " amount " + str(amount)) ``` ```python # Good — structured, leveled, correlated, no PII (id not email). import logging, json from opentelemetry import trace def log_charge(order_id, amount): ctx = trace.get_current_span().get_span_context() logging.info(json.dumps({ "event": "charge.succeeded", "level": "info", "order_id": order_id, # an opaque id, not the customer's email "amount_cents": amount, "trace_id": format(ctx.trace_id, "032x"), # ← the pivot back to the trace "span_id": format(ctx.span_id, "016x"), })) ``` Level discipline: `error` = a human should look, `warn` = degraded but handled, `info` = business milestones, `debug` = off in prod. If everything is `error`, nothing is. ## Metrics that earn an alert **RED for request-driven services, USE for finite resources.** Alerts come from metrics — logs and traces are for *investigating* the alert, not firing it. - **RED** (per service/endpoint): **R**ate (requests/s), **E**rrors (failed/s), **D**uration (latency as a *histogram*, so you can read p50/p95/p99 — never a single average, which hides the tail). - **USE** (per resource — CPU, pool, disk): **U**tilization, **S**aturation (queue depth / waiting), **E**rrors. **The cardinality rule — this is the #1 way an observability stack falls over.** A metric's total series count is the product of its label cardinalities. Put an unbounded value on a label and you create a near-infinite series count; the time-series database (Prometheus/Mimir) is most often restarted because of exactly this. Loki indexes *labels only, not log contents* — so the same rule binds its stream labels. ```text # Bad — user_id is unbounded; 5M users = 5M series per metric. OOMs the TSDB. http_requests_total{route="/checkout", user_id="u_8f3a...", status="200"} ``` ```text # Good — only bounded, low-cardinality dimensions on the metric. http_requests_total{route="/checkout", method="POST", status="200"} # Need to slice by user? That's a trace attribute or a log field, never a metric label. ``` ## The Collector config A minimal valid pipeline. An exporter is **inert until it appears in a pipeline** — defining one under `exporters:` does nothing on its own. ```yaml # otel-collector.yaml — Collector v0.153.0 shape receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } http: { endpoint: 0.0.0.0:4318 } processors: memory_limiter: # first line of defense: shed load before OOM check_interval: 1s limit_percentage: 80 batch: {} # batch before export — fewer, bigger round-trips redaction: # strip PII before it leaves your network allow_all_keys: true blocked_values: ["[0-9]{13,16}", "\\b[\\w.]+@[\\w.]+\\b"] # PANs, emails exporters: otlphttp/traces: { endpoint: http://tempo:4318 } otlphttp/logs: { endpoint: http://loki:3100/otlp } otlphttp/metrics: { endpoint: http://mimir:9009/otlp } service: pipelines: traces: { receivers: [otlp], processors: [memory_limiter, redaction, batch], exporters: [otlphttp/traces] } logs: { receivers: [otlp], processors: [memory_limiter, redaction, batch], exporters: [otlphttp/logs] } metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/metrics] } ``` Tail sampling, gateway-vs-agent topology, multi-backend fan-out (LGTM **and** a vendor in parallel), and `resourcedetection` live in `references/collector-config.md`. Shipping the Collector as a container or in CI is `../docker/SKILL.md`. ## Alerts the telemetry now enables The instrumentation above makes these *possible* — defining them is your job; routing the page to a human is `../monitoring/SKILL.md`. - **SLO burn-rate (multi-window)** — alert when you're spending the error budget too fast, using a fast window (e.g. 5m) AND a slow window (e.g. 1h) so a brief blip doesn't page but a sustained burn does. This fires on *impact*, not on traffic. - **Latency SLO** — p99 of the duration histogram over budget for a sustained window. - **Saturation** — the USE "S": queue depth / pool waiters climbing, the leading indicator before errors appear. ## Anti-patterns | Anti-pattern | Why it bites | Do instead | |---|---|---| | Unbounded label (`user_id`, `request_id`, `email`) on a metric or Loki stream | Cardinality explosion → TSDB OOM/restart, cost blowup | Keep labels low-cardinality; put the high-cardinality field on a span attribute or log body | | Logging PII / secrets (email, card, token) | Compliance breach + the leak is now in every log backend | Log opaque ids; redact in the Collector before export | | 100% trace sampling in prod, no policy | Pay to store noise; backend throttles and drops the traces you needed | Head/tail sampling — keep all errors + slow traces, sample the rest | | App exports straight to the vendor, no Collector | Can't batch, retry, redact, or swap backends without a redeploy | Always route through a Collector | | Instrument everything before deciding the question | Noise with no signal; nobody opens the dashboard | Start from "what would I ask during an incident," instrument that path | | Alert on a raw error *count* | Fires on traffic spikes, silent during a low-traffic outage | Alert on error *rate* / SLO burn | | One giant span per request (or a thousand contentless ones) | No causality, or context with no detail — both useless | Span per meaningful operation, each with attributes + status | ## Verify Run `scripts/verify.sh` against the directory holding your Collector config + SDK init. It checks the config is valid, that every defined exporter is actually wired into a pipeline (the classic "defined but unused" footgun), that `service.name` is set, and warns on high-cardinality metric labels. It is read-only and exits 0 when there's nothing to check. Then prove the wire with one safe induced failure in a test/staging path. Confirm the expected error-rate metric changes, the trace records the failing operation and error status, and the structured log carries the same `trace_id` without PII. Query the backend/Collector output; “the instrumentation code ran” is not evidence that usable telemetry arrived. Live paging and escalation proof remains `../monitoring/SKILL.md`.