observability-instrumentation

name	observability-instrumentation
description	Instrument services for observability — structured logging, distributed tracing, metrics naming conventions, and the three-pillar model. Use when asked about "add logging", "structured logs", "distributed tracing", "trace ID", "metrics", "Prometheus", "OpenTelemetry", "why can't I debug production issues", or "correlate logs across services". Do NOT use for: SLO/alerting design — use `slo-design`. Do NOT use for: log storage/infrastructure provisioning.
origin	yamtam-original
license	MIT © 2025 Vũ Văn Tâm
version	1.0.0
compatibility	Any backend service. Examples use OpenTelemetry + Prometheus notation.

When to Use

• Use when: production issues are hard to diagnose because logs are unstructured
• Use when: a request spans multiple services and you can't follow the path
• Use when: adding metrics to a new service before its first deploy
• Use when: setting up OpenTelemetry for a project
• Do NOT use for: frontend performance monitoring — use web-performance

---

The Three Pillars

Pillar	What it answers	Tool examples
Logs	What happened at a point in time	Loki, CloudWatch, Datadog Logs
Metrics	How much / how fast / how often over time	Prometheus, CloudWatch Metrics
Traces	How a request flowed across services	Jaeger, Tempo, AWS X-Ray

All three are needed. Metrics tell you something is wrong; logs tell you what; traces tell you where.

---

Structured Logging

Every log line must be machine-parseable JSON. No free-text concatenation.

Required fields on every log line

{
  "timestamp": "2025-05-21T10:00:00.123Z",
  "level":     "INFO",
  "service":   "order-service",
  "trace_id":  "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id":   "00f067aa0ba902b7",
  "message":   "Order placed successfully",
  "order_id":  "ord-123",
  "user_id":   "usr-456",
  "duration_ms": 42
}

Log levels — use consistently

Level	Use
ERROR	Something failed; human action may be needed
WARN	Degraded behavior; investigation warranted
INFO	Normal significant events (request received, order placed)
DEBUG	Detailed diagnostic data — disable in production by default

Rules:

• Never log PII (email, name, card number, tokens) — log IDs only
• Never interpolate secrets into log messages
• Include trace_id on every line — enables cross-service correlation

---

Distributed Tracing

Concepts

Trace:  one request's full journey across all services
Span:   one operation within a trace (service call, DB query, external API)
Parent span: the span that initiated a child span

Trace:   [order-service: handle POST /orders] (root span)
  └── Span: [payment-service: charge card]
  └── Span: [inventory-service: reserve stock]
       └── Span: [DB: UPDATE inventory]

Propagation — pass trace context in HTTP headers

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             version-traceId-spanId-flags

Use W3C TraceContext standard. Always forward traceparent on all outbound calls.

OpenTelemetry setup (minimal)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider()
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("order-service")

with tracer.start_as_current_span("place-order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("user.id", user_id)
    # ... do work

---

Metrics Naming Conventions

Follow Prometheus naming conventions — other systems largely adopt the same.

{namespace}_{subsystem}_{name}_{unit}

Examples:
  http_requests_total              (counter — always _total suffix)
  http_request_duration_seconds    (histogram — use base units: seconds not ms)
  db_connections_active            (gauge — current state)
  order_processing_errors_total    (counter)
  cache_hit_ratio                  (gauge — 0.0 to 1.0)

Metric types

Type	Use	Example
Counter	Monotonically increasing count	requests, errors, orders
Gauge	Current value up or down	active connections, queue depth
Histogram	Distribution of values	request duration, payload size
Summary	Like histogram, client-side quantiles	legacy — prefer histogram

Mandatory labels (cardinality warning)

http_requests_total{method="POST", status="200", endpoint="/orders"}

Never use user IDs, order IDs, or any high-cardinality value as a label — creates metric explosion.

---

Instrumentation Checklist

Every production service must have:

□ Structured JSON logs with timestamp, level, service, trace_id on every line
□ trace_id injected into all log lines (MDC / context propagation)
□ Outbound HTTP calls: forward traceparent header
□ Inbound requests: extract trace context or start new root span
□ Metrics: request count, error count, duration histogram per endpoint
□ Alerts wired to error rate and latency metrics (see slo-design)
□ No PII in logs — IDs only

---

Anti-Fake-Pass Rules

Before claiming a service is observable, you MUST show:

• Logs are structured JSON — no free-text concatenation
• trace_id present on every log line and propagated to downstream calls
• At least 3 metrics: request count, error count, latency histogram
• OpenTelemetry (or equivalent) trace spans on outbound calls and DB queries
• PII verified absent from log fields (spot-check 5 representative log lines)

Reference: gates/anti-fake-pass-gate.md