| name | observability |
| description | Use when adding logging, structured logs, metrics, traces, or alerts to a service, when designing dashboards, when defining SLOs, when investigating "how will I know if this breaks", or when production logs are too noisy to debug from. |
Observability
Overview
Define how the system tells you it's broken — before it's broken. Output is .forge/observability.md: structured logging conventions, correlation ID flow, the metrics taxonomy (golden signals per service, USE for resources), trace sampling policy, SLO + alert thresholds (page-worthy vs ticket-worthy), dashboard layouts, log retention, and PII redaction rules. Pairs with error-handling-and-resilience (errors classified there get observed here) and incident-response-and-postmortems (alerts there route to runbooks).
When to Use
- A service is going to production and there's no monitoring beyond
console.log
- Logs exist but are unstructured (free-text) and unsearchable
- Alerts fire constantly (fatigue) or don't fire when they should (gaps)
- Distributed system has no correlation IDs and debugging requires log archaeology
- A new component has been added without dashboards or alerts
- The team can't answer "what's the p95 latency right now?" in under 30 seconds
When NOT to Use
- Local-only scripts, prototypes, or one-off jobs
- Trivial CRUD additions to a service that already has observability
- Pure documentation or refactoring tasks with no runtime surface
Common Rationalizations
| Thought | Reality |
|---|
| "Log everything, we'll filter later" | Noise drowns signal. INFO-everything logs are unsearchable in production at scale. |
| "We'll add monitoring later" | You can't debug what you can't see. Add the dashboard before the first user hits the endpoint. |
| "Console.log is fine for now" | Unstructured logs can't be queried, aggregated, or correlated across services. |
| "Alerts can wait" | The first outage you miss without an alert costs more than every alert you'll set up this quarter. |
| "We don't need traces, we have logs" | Logs tell you what happened; traces tell you why it took 4 seconds. They're different. |
| "Sample 100% of traces" | Tracing cost grows linearly with throughput. Sample, with head-based + tail-based sampling for errors. |
Red Flags
- A service in production with no dashboard
- An alert without a linked runbook
- Logs that contain PII (emails, names, tokens) at INFO
- The same alert firing >10x/day without ack — fatigue
- An "ERROR" log line at INFO level (severity drift)
- A correlation ID that stops at a service boundary
- Traces sampled at 100% in a >10 RPS service
- A dashboard nobody has opened in 30 days
Core Process
Step 1: Define correlation ID flow
Every request entering the system gets a trace ID at the edge. Every downstream call (HTTP, queue, RPC) propagates it via the standard header (traceparent or x-request-id). Every log line carries it. Document the flow end-to-end in the architecture doc.
Step 2: List golden signals per service
For each service, define the four REDs:
- Rate — requests per second
- Errors — 4xx / 5xx rate
- Duration — p50, p95, p99
- Saturation — queue depth, CPU, memory headroom
For data stores, define USE:
- Utilization — % busy
- Saturation — wait queue
- Errors — counts
Step 3: Define SLOs and alert thresholds
For each user-facing endpoint:
- SLO (e.g., "99.9% of requests succeed within 500ms p95 over 30 days")
- Page-worthy threshold (burning the budget — alert the on-call)
- Ticket-worthy threshold (degraded — file a ticket, fix this week)
- Never-alert noise (background warnings, expected churn)
Every alert MUST link to a runbook (see incident-response-and-postmortems).
Step 4: Establish log levels and structure
| Level | When |
|---|
| ERROR | Failure requiring human attention |
| WARN | Anomaly that retried or recovered |
| INFO | State transitions: started, completed, deployed |
| DEBUG | High-volume internal detail; off in production by default |
Structured JSON only. Fixed top-level fields: timestamp, level, service, trace_id, span_id, user_id (hashed if PII-sensitive), message, error.kind (matching error-handling-and-resilience taxonomy).
Step 5: Design dashboards
One dashboard per service, with a standard layout:
- Top row: SLO compliance + error rate + latency p95
- Middle: RED signals broken down by endpoint
- Bottom: dependency latencies and saturation
Plus one "user journey" dashboard per critical path from .forge/testing-strategy.md.
Step 6: Configure trace sampling
- Head-based 1-10% baseline
- Tail-based 100% for errors, anomalous latency
- Always-sample for traces tagged
priority=high (e.g., paying customer endpoints)
Step 7: Log retention + PII redaction policy
- INFO/DEBUG retention: 7-14 days
- ERROR retention: 90 days
- Audit log retention: per compliance (1+ years)
- PII redaction rules: list every field that must be hashed, redacted, or excluded entirely (cross-reference
security-and-compliance skill's PII inventory)
Step 8: Header
Prepend a forge:meta header to .forge/observability.md (generated_by: observability, generated_at: <ISO 8601 UTC with Z>, depends_on: [.forge/architecture.md] — paths only, never hashes, generated_from: {.forge/architecture.md: <upstream content_hash AT generation time>}, content_hash: <sha256 first 8 of THIS file's body>). See forge-dependency-graph.
Verification
