Observability discipline: structured logging, metrics instrumentation, distributed tracing, and signal correlation. Invoke whenever task involves any interaction with observability concerns — adding logging, designing metrics, instrumenting traces, correlating signals, reviewing instrumentation, or understanding when to use which pillar.
Observability discipline: structured logging, metrics instrumentation, distributed tracing, and signal correlation. Invoke whenever task involves any interaction with observability concerns — adding logging, designing metrics, instrumenting traces, correlating signals, reviewing instrumentation, or understanding when to use which pillar.
Observability
If you cannot ask arbitrary questions about your system's behavior from the outside, your system is not observable —
it is merely monitored.
The Three Pillars
Logs — What Happened
Logs are timestamped, discrete event records. They capture what happened at a specific moment: an error thrown, a
user action, a configuration loaded, a connection refused.
Use logs when you need:
Rich diagnostic context for a specific event
Debugging information with full error details and stack traces
Audit trails of who did what and when
Record of discrete state transitions
Logs are poor at:
Showing aggregate system health (use metrics)
Tracing request flow across services (use traces)
High-frequency numeric trends (too expensive at volume)
Metrics — How Is It Doing
Metrics are numeric measurements aggregated over time. They capture how the system is performing as quantitative
time series: request rates, error percentages, latencies, queue depths, resource utilization.
Use metrics when you need:
Real-time health signals and alerting
Trend analysis over hours, days, weeks
Capacity planning and saturation monitoring
Pre-aggregated data that scales cheaply regardless of traffic
Metrics are poor at:
Explaining something is broken (use logs)
why
Showing the path of a single request (use traces)
Storing per-event detail (cardinality explosion)
Traces — How Did It Flow
Traces record the causal chain of operations that make up a single request as it propagates through distributed
components. A trace is a tree of spans, where each span represents one unit of work (an HTTP call, a database query,
a queue publish).
Use traces when you need:
End-to-end latency breakdown across services
Dependency mapping and bottleneck identification
Understanding the path a failing request took
Correlating work across process and network boundaries
Traces are poor at:
Aggregate health monitoring (use metrics)
Detailed per-event diagnostics on a single node (use logs)
Cheap, long-term trend storage (traces are expensive at 100% sampling)
Choosing the Right Signal
"Is the system healthy right now?" — Metrics
"Why did this specific request fail?" — Traces + Logs
"What happened at 03:14 on node-7?" — Logs
"Where is the bottleneck in checkout flow?" — Traces
"Are error rates increasing over the last hour?" — Metrics
"What was the full stack trace of that exception?" — Logs
"Which downstream service is slow?" — Traces
"How much headroom does the database have?" — Metrics
Structured Logging
Always Structured
Emit logs as structured records (JSON or equivalent key-value format) with a consistent schema. Unstructured string logs
are for local development only. Structured logs are machine-parseable, indexable, and filterable at scale.
Log Levels
Use levels consistently. Agree on what each level means across the team.
FATAL/CRITICAL — Process cannot continue; about to crash. Alerting: Page immediately
WARN — Unexpected condition; system compensated. Alerting: Monitor trend
INFO — Significant business or lifecycle event. Alerting: Dashboard
DEBUG — Diagnostic detail for developers. Alerting: Never in production by default
TRACE — Extremely verbose step-by-step flow. Alerting: Never in production
Rules:
Production defaults to INFO or above. DEBUG/TRACE are off unless explicitly enabled for a bounded investigation
window.
WARN is not a dumping ground. If it never leads to action, it is noise — downgrade to DEBUG or remove it.
ERROR means something is broken. Expected conditions (404 for missing resources, validation failures from bad input)
are not errors — log at INFO with a status field.
Log level must be configurable at runtime without restarts.
Structured Fields
Every log record should include these baseline fields:
timestamp: ISO 8601, UTC
level: Severity (ERROR, WARN, INFO, ...)
message: Human-readable summary of the event
service: Service name emitting the log
version: Service version / build / commit SHA
trace_id: Distributed trace ID (if in request context)
Request/response bodies containing user-submitted personal data
When user identifiers are needed, log opaque IDs (user_id), not email addresses or names. If regulations (GDPR, HIPAA)
apply, verify logged fields comply. When in doubt, omit the field.
Logging at Boundaries
At application startup:
INFO: service name, version, loaded configuration (without secrets), listen address
WARN: degraded mode (e.g., fallback to local cache because Redis is unreachable)
ERROR/FATAL: unrecoverable startup failures
Per incoming request:
INFO: method, path (scrubbed of PII), status code, duration, request dimensions (tenant, region)
WARN/ERROR: only for unexpected exceptions; catch at the top-level handler
Per outgoing dependency call:
INFO or DEBUG: target service, operation, status, duration
ERROR: failures in dependent services (Redis, database, queue, etc.)
Log Once, at the Right Level
Log a raised exception once. Do not catch-log-rethrow at every layer. Let exceptions propagate to the top-level
handler, which logs with full context. Log and rethrow only when adding context that would otherwise be lost.
Metrics
Metric Types
Counter — Monotonically increasing; resets on restart. Use for totals: requests, errors, bytes sent
Gauge — Arbitrary value; goes up and down. Use for snapshots: queue depth, memory usage, connections
Histogram — Client-side aggregation into buckets. Use for distributions: request latency, payload size
Summary — Client-side quantile calculation. Use for pre-computed percentiles (less flexible than histogram)
Rules:
Use counters for events that accumulate. Derive rates with rate() / increase() — never store pre-computed rates.
Use gauges for current-state snapshots. Never rate() a gauge.
Use histograms for latency and size distributions. Histograms enable percentile calculation across instances;
summaries do not aggregate.
Export timestamps as Unix epoch seconds, not "time since" values.
Initialize all metrics with zero at startup to avoid missing-metric problems.
What to Measure
The Four Golden Signals (Google SRE)
For every user-facing service, measure these four:
Latency — Time to serve a request. Example: http_request_duration_seconds histogram
Traffic — Demand on the system. Example: http_requests_total counter by method/path
Errors — Rate of failed requests. Example: http_requests_total{status=~"5.."}
Saturation — How "full" the service is. Example: CPU usage, memory, queue depth, thread pool
Distinguish successful latency from error latency. A fast 500 is not good latency. Track both.
RED Method (Request-Centric)
For every microservice:
Rate — requests per second
Errors — failed requests per second
Duration — distribution of request latency
RED is a focused subset of the golden signals, optimized for request-driven services.
USE Method (Resource-Centric)
For every resource (CPU, memory, disk, network, thread pool):
Utilization — percentage of capacity in use
Saturation — backlog / queue depth
Errors — resource-level error count
RED tells you what is degraded from the user's perspective. USE tells you why at the infrastructure level. Use both
together.
Batch jobs — Last successful completion time, job duration, records processed, exit status
Caches — Hit rate, miss rate, eviction count, latency to backend on miss
Thread/connection pools — Pool size, active count, queue length, wait time
Metric Naming
Metric names should be self-documenting. Follow these conventions:
Prefix with namespace.myapp_http_requests_total, not requests_total.
Use base units. Seconds (not milliseconds), bytes (not megabytes), ratio 0-1 (not percentage 0-100).
Suffix with unit._seconds, _bytes, _total (for unit-less counters).
One metric, one unit, one quantity. Never mix request size with request duration in the same metric.
snake_case.http_request_duration_seconds, not httpRequestDurationSeconds.
Good
Bad
http_request_duration_seconds
request_latency (no unit, ambiguous)
http_requests_total
http_responses_500_total (use labels)
node_memory_usage_bytes
memory_mb (not base unit)
process_cpu_seconds_total
cpu_percent (use ratio 0-1)
Labels and Cardinality
Labels add dimensions to a metric. Every unique combination of label values creates a separate time series.
Good labels (bounded, low cardinality):
method (GET, POST, PUT, DELETE)
status_code (200, 404, 500 — or class: 2xx, 4xx, 5xx)
service, region, version
Dangerous labels (unbounded, high cardinality):
user_id, email, session_id
request_path with dynamic segments (/users/12345)
error_message (arbitrary strings)
Rules:
Keep label cardinality below 10 values per label for most metrics.
If a label can grow unbounded, it does not belong on a metric. Log it instead.
Use labels instead of encoding dimensions in the metric name. http_requests_total{method="GET"}, not
http_get_requests_total.
Ensure sum() or avg() across all label values is meaningful. If not, split into separate metrics.
Percentiles and Tail Latency
Averages hide outliers. A service with 100ms average latency may have 1% of requests taking 5 seconds. That 1% tail can
dominate user experience when users hit multiple services per page load.
Always track p50, p90, p99 latency at minimum.
Use histograms with exponentially distributed bucket boundaries (e.g., 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s,
2.5s, 5s, 10s).
Alert on p99, not mean. Mean latency alerts miss tail degradation.
Distributed Tracing
Core Concepts
Trace — End-to-end record of a single request across all services
Span — One unit of work within a trace (HTTP call, DB query, function)
Root span — First span in a trace; has no parent
Child span — Span nested under a parent; represents a sub-operation
Span context — Immutable bag of trace_id + span_id + flags, propagated across boundaries
Span attributes — Key-value metadata on a span (http.method, db.statement)
Span events — Timestamped annotations within a span's lifetime
Span links — Causal references between spans in different traces
Span Kinds
Client — Outgoing synchronous call. Example: HTTP request to another service
Server — Incoming synchronous call. Example: Handling an HTTP request
Producer — Creates async work. Example: Publishing to a message queue
Consumer — Processes async work. Example: Consuming from a message queue
Internal — No network boundary. Example: In-process function instrumentation
Context Propagation
Context propagation connects spans across process boundaries into a single trace. Without it, you get disconnected
spans, not traces.
Rules:
Propagate context on every outgoing call. HTTP headers (W3C Trace Context or B3), message metadata, gRPC metadata
— every cross-process boundary must carry trace context.
Extract context on every incoming call. Extract trace context and create a child span under the propagated parent.
Use W3C Trace Context (traceparent/tracestate) as the default propagation format unless the ecosystem requires
otherwise (e.g., legacy B3).
Never generate a new trace ID when continuing an existing trace. A new trace ID means a broken trace.
service.name, service.version: All spans (set on resource)
Use semantic conventions for attribute names rather than inventing
custom ones. Consistent naming enables cross-service analysis.
Span Status
Unset — Completed without error (default). When: most successful operations
Error — Operation failed. When: server errors, exceptions
Ok — Explicitly marked successful. When: only when you need to override ambiguity
Leave status as Unset for normal success. Set Error only for actual failures. Do not set Error for client errors
like 404 on a server span — the server operated correctly.
Sampling
At high traffic volumes, tracing 100% of requests is expensive. Sampling reduces cost while preserving signal. |
Strategy | How It Works | Trade-off | | ------------------------ |
------------------------------------------------------ | ---------------------------------------------- | |
Head-based | Decide at trace start whether to sample | Simple; may miss rare errors | | Tail-based | Decide
after trace completes based on content | Catches errors; needs buffering infrastructure | | Always-on for errors |
Sample 100% of error traces, probabilistic for success | Good default balance |
Rules:
Never drop error traces. If cost is a concern, sample successful traces at a lower rate but keep 100% of error and
high-latency traces.
Sample at the entry point (head) and propagate the decision. Each service deciding independently creates partial
traces.
Start with a low sampling rate (1-10%) and increase based on need, not the reverse.
Connecting the Pillars
The three pillars become powerful when correlated. An alert fires on a metric → you find the offending trace → the trace
points to a span → the span's logs reveal the root cause.
Correlation Keys
trace_id — Links logs and spans to the same trace. Where: logs, span context
span_id — Links a log to the exact span that produced it. Where: logs, span context
request_id — Correlates all work for one inbound request. Where: logs, HTTP headers
service.name + service.version — Groups telemetry by source. Where: all signals
Rules:
Embed trace_id and span_id in every log record emitted within a request context — the primary bridge between logs
and traces.
Use a correlation/request ID assigned at the edge (API gateway, load balancer) and propagated to all downstream
services.
Attach exemplars to metrics. An exemplar is a trace_id on a specific metric observation, enabling drill-down from
a metric spike to a representative trace.
The Correlation Workflow
Metrics dashboard shows error rate spike
→ Filter by service + time window
→ Find exemplar trace_id on the error counter
→ Open trace in tracing UI
→ Identify the failing span (database timeout)
→ Search logs by trace_id for full error details
→ Root cause: connection pool exhausted
Metric-to-Trace Exemplars
Exemplars attach a trace_id sample to a metric data point. When you see a latency spike on a histogram, the exemplar
gives you a concrete trace to investigate rather than guessing.
Attach exemplars to histogram observations for latency metrics.
Attach exemplars to counter increments for error metrics.
Not every metric point needs an exemplar — one per scrape interval is sufficient.
Trace-to-Log Linking
When viewing a trace, each span should link to its logs. When viewing a log, the trace_id should link back to the full
trace. This bidirectional linking is the backbone of incident investigation.
Anti-Patterns
Logging everything at DEBUG in production — Disk/cost explosion, noise buries signal. Fix: default to INFO; enable
DEBUG temporarily per-component
catch (err) { log(err); throw err; } at every layer — Same error logged N times across the call stack. Fix: log
once at the top-level handler
Metrics with unbounded label cardinality — Time series explosion; monitoring system degrades. Fix: use bounded
labels; move high-cardinality data to logs
Encoding dimensions in metric names — Cannot aggregate; proliferates metrics. Fix: use labels:
requests_total{method="GET"}
Averaging latency for alerting — Hides tail latency; misses degradation for minority of users. Fix: alert on p99
from histograms
Missing trace context propagation — Broken traces; spans from different services are disconnected. Fix: propagate
context on every cross-process call
Sampling each service independently — Partial traces — some spans sampled, some dropped. Fix: decide at head,
propagate sampling decision
Logging PII / secrets — Compliance violations, security risk. Fix: audit log fields; log opaque IDs, never raw PII
Alert on every metric wiggle — Alert fatigue; team ignores pages. Fix: alert on symptoms (golden signals), not
causes; require actionability
Treating WARN as a soft ERROR — WARN becomes noise nobody reads. Fix: WARN = system compensated but situation is
unusual; ERROR = broken
Storing pre-computed rates instead of counters — Cannot re-aggregate over different windows. Fix: store raw
counters; derive rates at query time
No baseline metrics for new services — Cannot tell if behavior is normal or degraded. Fix: instrument golden
signals from day one, before first deploy
Application
When Writing Code
Instrument from the start. Add golden signal metrics, structured logging, and trace context propagation before the
first production deploy — not after the first incident.
Follow the conventions silently. Apply structured logging, metric naming, and tracing patterns without narrating
each rule.
If the codebase has existing patterns, follow them. Consistency beats theoretical correctness. Flag divergences
once, then move on.
Choose the right pillar. Before adding instrumentation, ask: "Is this a metric, a log, or a span?" Use the
decision table above.
Connect the signals. Every log in a request context must carry trace_id and span_id. Every error metric should
have an exemplar.
When Reviewing Code
Check that new endpoints/operations have golden signal coverage. Missing metrics on a new endpoint is a review
blocker.
Verify structured logging. Unstructured log.Print("something happened") in production code should be flagged
with the fix inline.
Check log levels. Expected client errors logged as ERROR, or debug noise left on at INFO, are common mistakes.
Verify trace context propagation. Any new outgoing HTTP/gRPC/queue call must propagate trace context. Missing
propagation breaks traces.
Check label cardinality. New metric labels must be bounded. Flag unbounded labels (user IDs, free-text)
immediately.
No sensitive data in logs or span attributes. Passwords, tokens, PII in telemetry is a security and compliance
defect.
Bad review comment:
"According to observability best practices, you should consider
adding structured logging with appropriate fields..."
Good review comment:
"Missing trace_id in log context — requests through this handler
won't correlate to traces. Add ctx.TraceID() to the logger fields."
Integration
This skill provides observability discipline alongside other skills:
Observability (this skill) — What to log, measure, and trace
Tool-specific skills (Prometheus, StatsD, OTel) — How to implement with a specific technology
The coding skill governs workflow. This skill governs observability design decisions. Tool-specific skills govern
implementation details for their respective technologies.
