observability

name	observability
description	Use when: instrument a system with logs, metrics, and traces to understand it in production.

Goal: be able to answer new questions about production without redeploying.

Use for:

• instrumenting services for debugging and alerting
• diagnosing latency, errors, and saturation
• designing what to measure and emit

Workflow:

1. Cover the three pillars: logs, metrics, and traces.
2. Emit structured logs with correlation/trace IDs.
3. Track the golden signals: latency, traffic, errors, saturation.
4. Propagate trace context across service boundaries.
5. Alert on symptoms users feel, not noisy internals.
6. Verify you can trace one request end to end.

What to capture:

• request-level traces with timing per span
• RED/USE metrics for services and resources
• structured, queryable logs with context
• meaningful, low-noise alerts on SLOs

Rules:

• correlate logs, metrics, and traces with shared IDs
• alert on user-facing symptoms, not every blip
• instrument boundaries and dependencies first
• never log secrets or PII in telemetry