| name | observability-and-monitoring |
| description | Design logging, metrics, tracing, and alerting that make systems debuggable in production |
| difficulty | senior |
| domains | ["general"] |
Overview
A system you cannot observe is a system you cannot debug. Observability is not a dashboard — it is the ability to answer any question about system behavior using existing data, without deploying new code. This skill implements the three pillars (logs, metrics, traces) and the alerts that page you before users notice.
When to Use
- Before deploying any new service or significant feature
- When a production issue took more than 30 minutes to diagnose
- When adding a new data flow, user journey, or integration
- As part of the
/ship workflow
Process
Step 1: Define the SLIs and SLOs
SLI (Service Level Indicator): what you measure — latency, error rate, availability, throughput.
SLO (Service Level Objective): the target — "p99 latency < 500ms," "error rate < 0.1%."
Without SLOs, you don't know when to alert.
Step 2: Structured logging
- Log in JSON (not freeform text)
- Include: timestamp, log level, service name, trace ID, user ID (hashed), request ID
- Log at entry and exit of every significant operation
- Log errors with: exception type, message, stack trace, context
- Never log PII, passwords, tokens, or credit card numbers
- Log what happened, not just that it happened
Step 3: Key metrics
Implement the four golden signals for every service:
- Latency — p50, p95, p99 response time
- Traffic — requests per second, events per second
- Errors — error rate (4xx, 5xx), error count
- Saturation — CPU, memory, queue depth, connection pool
Add business metrics: active users, signups, payments, key conversions.
Step 4: Distributed tracing
- Assign a trace ID to every inbound request
- Propagate the trace ID to all downstream calls (outbound HTTP, database, queue)
- Use OpenTelemetry or compatible SDK
- Sample at 100% for errors; sample at 1–10% for success paths
Step 5: Health checks and readiness probes
/healthz — is the service alive?/readyz — is the service ready to accept traffic?- Health checks must not perform significant work (no database writes)
- Check downstream dependencies in readiness probe
Step 6: Alerting strategy
Alert on symptoms (user impact), not causes (CPU high):
- High error rate → alert immediately
- SLO breach approaching (error budget burn) → alert
- Latency SLO exceeded at p99 → alert
- Zero traffic (nothing coming in) → alert (might be upstream failure)
Do NOT alert on:
- Things that resolve themselves
- Things you can do nothing about at 3am
- Things that don't affect users
Every alert must have a runbook.
Step 7: Dashboards
Build one dashboard per service with:
- The four golden signals
- Key business metrics
- Recent deployments as annotations
- Links to runbooks
Step 8: Runbooks
For each alert: what does it mean, what is the likely cause, what are the first 3 debugging steps, what is the escalation path?
Anti-Rationalizations
"We'll add monitoring after we know what's important to monitor"
By the time you know what's important to monitor, you've already had the incident that taught you.
"Logs are enough — we don't need metrics"
Logs tell you what happened. Metrics tell you what is happening right now at scale.
Verification Requirements
