| name | observability-checklist |
| description | What good observability looks like — structured logs, trace spans, metrics, correlation ids, and the instrumentation rules that turn logs into answers. Use when adding a feature, reviewing a service, or debugging a prod issue. |
Observability checklist
The goal isn't logs — it's answering "what happened" in under five minutes at 2am. Design backward from the question.
Logs
Structured or nothing
- JSON (or logfmt), not free text.
- Every log has:
ts, level, service, request_id, msg, and relevant fields.
- Fields are snake_case or camelCase — pick one per service. No mixing.
- Durations in milliseconds as numbers, not "took 2s" strings.
Levels are a contract
error: something failed the user can't recover from. Alert-worthy.warn: something unexpected but handled. Review-worthy.info: the shape of normal traffic. Sampling OK.debug: developer-only. Off in prod.- Don't invent new levels.
What to log
- Request entry (method, path, user, request_id).
- Outcome (status, duration).
- Branch decisions that matter (authz check, cache hit/miss, retry).
- Errors with the full context you need to reproduce.
What NOT to log
- Secrets, tokens, passwords, PII — scrub before logging.
- The full request body by default. Specific fields, yes. Everything, no.
- Stack traces as separate lines — include as a
stack field so they stay with the event.
Traces
Span per unit of work
- HTTP request handler = 1 span.
- Outbound call = child span.
- DB query = child span (with
db.statement truncated / hashed, not raw if PII risk).
- Background job step = 1 span.
Mandatory attributes
service.name, service.version.- For HTTP:
http.method, http.route, http.status_code.
- For DB:
db.system, db.operation.
- For errors:
exception.type, exception.message.
Propagation
- Every outbound request carries trace context headers.
- Every background job enqueues with trace context.
- A trace that ends at the HTTP edge and restarts at the worker is useless.
Metrics
RED for services
- Rate — requests per second, per endpoint.
- Errors — error rate, per endpoint.
- Duration — p50/p95/p99, per endpoint.
USE for resources
- Utilization — % of capacity in use.
- Saturation — queue depth / wait time.
- Errors — failed operations.
Cardinality discipline
- No unbounded tags (user_id, tenant_id, url with params). They explode your TSDB bill.
- Pre-aggregate cardinal dimensions; keep labels coarse.
http.route (the template), not http.url (the instance).
Correlation
request_id in: logs, traces (as request.id attr), response headers, error payloads.trace_id in: logs (auto-injected by SDK), response headers for user bug reports.- A user reporting "I got a 500" should give you enough to find the request — the
request_id is visible and searchable.
Per-language
Node
- Pino or Winston with JSON transport.
- OpenTelemetry SDK (
@opentelemetry/sdk-node).
- Prom-client for metrics.
Python
structlog with ProcessorFormatter.opentelemetry-sdk + opentelemetry-instrumentation-*.prometheus_client.
Go
slog (stdlib, Go 1.21+) with JSON handler.go.opentelemetry.io/otel + auto-instrumentation.prometheus/client_golang.
Rust
tracing + tracing-subscriber with JSON formatter.opentelemetry-otlp.prometheus crate.
Anti-patterns
- "Log everything, we'll figure it out later" — the noise hides the signal you need.
- Traces without sampling — you'll drown; sample intelligently (head-based at edges, tail-based for errors).
- Metrics where logs would do — counting "user clicked button" 1M times/day is cheaper as a metric than a log.
- Logs where metrics would do — "latency: 247ms" as a log line is wasting money; it's a histogram.
- Alerts on everything — page only on symptoms that hurt users, not causes.
Smells
- A single log line has more than ~10 fields — probably two events glued together.
- You use
grep to find things — you should have a field + dashboard.
- Error messages are generic ("an error occurred") — they cost you the 2am minutes.
console.log in prod code — belongs in a logger, tagged and leveled.
