observability

name	observability
description	Use when adding structured logging, instrumenting Prometheus metrics, wiring up distributed tracing with OpenTelemetry, writing alerting rules, defining SLOs and error budgets, or building a Grafana golden-signals dashboard.

Observability

Observability is the practice of instrumenting systems so you can understand their internal state from external outputs — logs, metrics, and traces.

When to Activate

• Adding logging to a new service or endpoint
• Setting up Prometheus metrics for a service
• Implementing distributed tracing across services
• Writing alerting rules or on-call runbooks
• Defining SLOs and error budgets for a service
• Building a Grafana dashboard for a service
• Debugging a production issue using logs, metrics, or traces

---

The Three Pillars

Pillar	Question it answers	Best tool	When to reach for it
Logs	"What happened?"	structlog, pino, slog	Debugging specific errors, audit trails
Metrics	"How is the system performing?"	Prometheus	Trending, alerting, capacity
Traces	"Why is this slow / where did it fail?"	OpenTelemetry	Latency debugging, distributed request flow

OpenTelemetry is the unifying standard across all three pillars: one SDK for logs, metrics, and traces, vendor-agnostic, with exporters to any backend (Grafana, Datadog, Honeycomb, Jaeger, etc.).

---

Structured Logging

Always emit logs as JSON. Human-readable plaintext is fine in development, but production logs must be machine-parseable. Never log sensitive data (PII, card numbers, passwords, tokens).

Mandatory Fields

Every log line must include:

Field	Type	Example
timestamp	ISO 8601	2024-01-15T10:30:00.000Z
level	string	INFO
service	string	payment-service
trace_id	string	abc123...
span_id	string	def456...
request_id	string	UUID per HTTP request
message	string	Human-readable description

Log Levels

Level	When to use	Example
DEBUG	Verbose dev-only detail	"Entering validatePayment()"
INFO	Normal operational events	"Payment processed for order 123"
WARN	Unexpected but recoverable	"Retry attempt 2/3 for order 123"
ERROR	Failure that needs attention	"Payment declined: card expired"
FATAL/CRITICAL	Service cannot continue	"DB connection pool exhausted"

Rule: use INFO in production, DEBUG only in development. WARN does not page on-call. ERROR does.

Code Examples

Python with structlog:

import structlog

logger = structlog.get_logger()

# Configure once at startup
structlog.configure(
    processors=[
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

# Usage — bind context, then log
log = logger.bind(service="payment-service", request_id=request_id)
log.info("payment.processing", order_id=order_id, amount=amount)
log.error("payment.failed", order_id=order_id, error=str(e))

TypeScript with pino:

import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: { service: 'payment-service' },
});

const requestLogger = logger.child({ requestId, traceId });
requestLogger.info({ orderId, amount }, 'payment.processing');
requestLogger.error({ orderId, err }, 'payment.failed');

Go with slog (stdlib, Go 1.21+):

import "log/slog"

logger := slog.With(
    slog.String("service", "payment-service"),
    slog.String("request_id", requestID),
)

logger.Info("payment.processing",
    slog.String("order_id", orderID),
    slog.Float64("amount", amount),
)
logger.Error("payment.failed",
    slog.String("order_id", orderID),
    slog.String("error", err.Error()),
)

Correlation ID Propagation

Correlation IDs let you follow a single request across multiple services and log entries.

• Generate request_id at the edge (API gateway or the first handler to receive the request)
• Pass it downstream via HTTP headers: X-Request-ID or X-Correlation-ID
• Include the ID in all log lines for the duration of that request
• Include the ID in error responses so users can report it to support
• In async flows (message queues, event streams): embed correlation_id in the message payload itself, not just headers

Middleware pattern (Express/Node):

import { v4 as uuidv4 } from 'uuid';

app.use((req, res, next) => {
  const requestId = req.headers['x-request-id'] as string ?? uuidv4();
  req.requestId = requestId;
  res.setHeader('X-Request-ID', requestId);
  req.log = logger.child({ requestId });
  next();
});

Middleware pattern (Python/FastAPI):

import uuid
from starlette.middleware.base import BaseHTTPMiddleware

class CorrelationIDMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        request_id = request.headers.get("x-request-id") or str(uuid.uuid4())
        with structlog.contextvars.bound_contextvars(request_id=request_id):
            response = await call_next(request)
            response.headers["x-request-id"] = request_id
            return response

---

Metrics with Prometheus

Prometheus is a pull-based metrics system: your service exposes a /metrics endpoint and Prometheus scrapes it on a configured interval (typically 15–30 seconds).

Metric Types

Type	Use for	Example
Counter	Things that only go up (requests, errors)	http_requests_total
Gauge	Values that go up and down	active_connections, memory_bytes
Histogram	Distribution of values (latency, size)	request_duration_seconds
Summary	Like histogram, but calculates quantiles client-side	Avoid — prefer Histogram

Prefer Histogram over Summary. Histograms can be aggregated across instances in PromQL; Summaries cannot.

Naming Conventions

Pattern: <namespace>_<subsystem>_<name>_<unit>

• payment_service_http_requests_total — counter; _total suffix implies count
• payment_service_db_query_duration_seconds — histogram; always include unit in the name
• payment_service_cache_hit_ratio — gauge; ratio 0–1, no unit suffix needed
• payment_service_queue_depth — gauge; current number of items in queue

Never include label values in metric names. Use labels instead:

# WRONG
http_500_errors_total

# RIGHT
http_errors_total{status="500"}

Label Cardinality Trap

Labels multiply time series. A metric with two labels each having 10 values creates 100 series. High-cardinality labels (user IDs, order IDs, full URL paths with path parameters) can create millions of series, crashing Prometheus.

# BAD — creates a new time series for every unique user ID
http_requests_total.labels(user_id=user.id).inc()

# GOOD — low cardinality labels only
http_requests_total.labels(method="POST", endpoint="/payments", status="200").inc()

Safe label values: HTTP method, endpoint (normalized, no IDs), status code, region, environment, service version.

RED Method (for services)

Use RED to define the minimum set of metrics for any service:

• Rate: requests per second

  rate(http_requests_total[5m])
  

• Errors: error rate as a fraction of total traffic

  rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m])
  

• Duration: latency at the 95th percentile

  histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))
  

USE Method (for resources)

Use USE to define metrics for infrastructure and resource-level components:

• Utilization: percentage of time the resource is busy (e.g., CPU%, disk I/O%)
• Saturation: work that is queued because it cannot be served immediately (e.g., run queue length)
• Errors: error events from the resource (e.g., disk I/O errors, network drops)

Code Examples

Python with prometheus_client:

from prometheus_client import Counter, Histogram, start_http_server

REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)
REQUEST_DURATION = Histogram(
    'request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5]
)

# Usage
with REQUEST_DURATION.labels(method='POST', endpoint='/payments').time():
    result = process_payment()
REQUEST_COUNT.labels(method='POST', endpoint='/payments', status='200').inc()

# Expose metrics endpoint
start_http_server(8000)

TypeScript with prom-client:

import { Counter, Histogram, collectDefaultMetrics, register } from 'prom-client';

collectDefaultMetrics({ prefix: 'payment_service_' });

const requestCount = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'endpoint', 'status'],
});

const requestDuration = new Histogram({
  name: 'request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'endpoint'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5],
});

// Expose metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Go with prometheus/client_golang:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requestCount = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    }, []string{"method", "endpoint", "status"})

    requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "request_duration_seconds",
        Help:    "HTTP request duration",
        Buckets: prometheus.DefBuckets,
    }, []string{"method", "endpoint"})
)

// Expose metrics endpoint
http.Handle("/metrics", promhttp.Handler())

---

Distributed Tracing with OpenTelemetry

Concepts

• Trace: the complete end-to-end journey of a single request, identified by a unique trace_id
• Span: a single named unit of work within a trace (has a start time, end time, status, and attributes); spans form a tree
• Context propagation: the mechanism for passing trace_id and span_id across service boundaries via HTTP headers; use the W3C traceparent header (standardized, supported everywhere)
• Collector: a sidecar or standalone service (otel-collector) that receives span data, batches it, and forwards it to a backend (Jaeger, Tempo, etc.)

Auto-Instrumentation vs Manual Spans

• Auto-instrumentation: instrument HTTP frameworks, DB drivers, gRPC clients automatically — get traces with zero code changes (install the SDK + one setup file)
• Manual spans: wrap custom business logic to make it visible in traces (e.g., "processing payment", "calling fraud check API", "serializing response")

When in doubt: start with auto-instrumentation to get baseline visibility, then add manual spans where you need business context.

Setup (Python)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Bootstrap once at startup
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("payment-service")

# Manual span around business logic
with tracer.start_as_current_span("process_payment") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("payment.amount", amount)
    span.set_attribute("payment.currency", currency)
    result = charge_card(card_token, amount)
    span.set_attribute("payment.result", result.status)

Setup (TypeScript)

import { trace, SpanStatusCode, context } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';

// Bootstrap once at startup
const provider = new NodeTracerProvider();
provider.addSpanProcessor(
  new BatchSpanProcessor(
    new OTLPTraceExporter({ url: 'http://otel-collector:4317' })
  )
);
provider.register();

const tracer = trace.getTracer('payment-service');

// Manual span with error handling
async function processPayment(orderId: string, amount: number) {
  const span = tracer.startSpan('process_payment');
  return context.with(trace.setSpan(context.active(), span), async () => {
    span.(, orderId);
    span.(, amount);
     {
       result =  (orderId, amount);
      span.({ : . });
       result;
    }  (err) {
      span.(err  );
      span.({ : ., : (err  ). });
       err;
    }  {
      span.();
    }
  });
}

Setup (Go)

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func initTracer(ctx context.Context) (*trace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("payment-service"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

// Usage
tracer := otel.Tracer("payment-service")
ctx, span := tracer.Start(ctx, "process_payment")
defer span.End()
span.SetAttributes(
    attribute.String("order.id", orderID),
    attribute.Float64("payment.amount", amount),
)

Sampling Strategies

Strategy	Description	Best for
Head sampling (always on)	Sample 100% of traces	Development, test environments
Head sampling (probabilistic)	Sample X% of traces at trace start	Steady-state production traffic cost control
Head sampling (never)	Drop all traces	When tracing a noisy low-value service
Tail sampling	Sample based on outcome (errors, slow spans)	Catch 100% of errors while sampling fast successes
Parent-based	Follow the upstream caller's sampling decision	Microservices (honor the decision made at the edge)

Recommended production setup: parent-based at 10% for happy path + tail sampling to capture 100% of errors and traces with p99+ latency.

---

Alerting Design

Symptom-Based vs Cause-Based

Alert on symptoms (what users experience), not causes (what resources are doing):

Type	Example	Why
Symptom (preferred)	"5xx error rate > 1% for 5 minutes"	Users are directly affected right now
Cause (avoid as primary)	"CPU usage > 80%"	CPU can spike without user impact; often too noisy
Cause (use for capacity)	"Disk > 85% full"	Predictive; gives time to act before impact

Alert Design Rules

• Every alert must link to a runbook that explains diagnosis and remediation steps
• severity: critical means page someone immediately — it is happening now and users are affected
• severity: warning means investigate during business hours — degraded but not yet user-impacting
• Avoid alerting on individual metrics; alert on SLO burn rate where possible
• Every alert should be actionable: if you cannot do anything about it, do not alert on it
• Add a for: duration to avoid flapping on transient spikes (5 minutes is a safe default)

Prometheus Alerting Rule Example

groups:
  - name: payment-service
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{service="payment-service",status=~"5.."}[5m])
          / rate(http_requests_total{service="payment-service"}[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
          service: payment-service
          team: payments
        annotations:
          summary: "High error rate on payment-service"
          description: "Error rate is {{ $value | humanizePercentage }} over last 5 minutes"
          runbook: "https://wiki.example.com/runbooks/payment-service#high-error-rate"

      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99,
            rate(request_duration_seconds_bucket{service="payment-service"}[5m])
          ) > 2.0
        for: 5m
        labels:
          severity: warning
          service: payment-service
          team: payments
        annotations:
          summary: "P99 latency above 2s on payment-service"
          description: "P99 is "
           

---

SLOs and Error Budgets

Hierarchy

• SLI (Service Level Indicator): the specific metric being measured (e.g., fraction of HTTP requests returning 2xx, p99 latency)
• SLO (Service Level Objective): the internal target for an SLI (e.g., 99.9% of requests return 2xx over a rolling 30-day window)
• SLA (Service Level Agreement): the contractual commitment with defined penalties if breached; always set SLA below SLO to leave buffer

Writing a Good SLO

An SLO must specify:

1. The SLI being measured
2. The target percentage
3. The time window (rolling 30 days is standard)
4. What counts as "good" vs "bad" (the success criterion)

Examples:

• "99.9% of payment API requests return HTTP 2xx over a rolling 30-day window"
• "99% of payment API requests complete in under 500ms over a rolling 30-day window"
• "The payment service is reachable from all regions 99.95% of the time over 30 days"

Availability SLO Math

SLO	Downtime per year	Downtime per month
99%	87.6 hours	7.3 hours
99.9%	8.7 hours	43.8 minutes
99.95%	4.4 hours	21.9 minutes
99.99%	52.6 minutes	4.4 minutes
99.999%	5.3 minutes	26 seconds

Most internal services target 99.9%. Customer-facing critical paths target 99.95%–99.99%. Do not set SLOs higher than you can actually achieve; an SLO you always exceed has no meaning.

Error Budget Policy

• Error budget = 1 − SLO (e.g., for 99.9% availability, 0.1% of requests can fail over 30 days)
• Track remaining error budget in a Grafana dashboard so teams see how much headroom they have

Budget consumed	Action
< 50%	Normal — continue shipping
50–75%	Review recent deployments; tighten testing
75–99%	Freeze non-critical deployments; prioritize reliability work
100% (exhausted)	All hands on reliability; no new features until budget recovers

Burn Rate Alerting

Burn rate measures how fast you are spending your error budget relative to the expected pace.

• If SLO is 99.9% over 30 days, you have ~43.8 minutes of downtime budget per month
• A burn rate of 1.0 means you are consuming budget at exactly the sustainable rate
• A burn rate of 14x means you will exhaust your monthly budget in ~3 days

# 1-hour burn rate for a 99.9% SLO
(
  1 - (
    rate(http_requests_total{status=~"2.."}[1h])
    / rate(http_requests_total[1h])
  )
) / (1 - 0.999)

Recommended burn rate alerts:

Burn rate	Window	Severity	Action
> 14x	1 hour	critical	Page immediately
> 6x	6 hours	critical	Page immediately
> 3x	1 day	warning	Ticket — investigate today
> 1x	3 days	warning	Ticket — investigate this week

---

Dashboard Design (Grafana)

Golden Signals Dashboard Template

Every service should have a single dashboard with at minimum four panels, one per golden signal:

1. Traffic — request rate (RPS) over time

   sum(rate(http_requests_total{service="$service"}[5m])) by (endpoint)
   

2. Latency — p50, p95, and p99 response time on one panel

   histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket{service="$service"}[5m])) by (le))
   histogram_quantile(0.95, sum(rate(request_duration_seconds_bucket{service="$service"}[5m])) by (le))
   histogram_quantile(0.50, sum(rate(request_duration_seconds_bucket{service="$service"}[5m])) by (le))
   

3. Errors — error rate as a percentage of total traffic

   sum(rate(http_requests_total{service="$service",status=~"5.."}[5m]))
   / sum(rate(http_requests_total{service="$service"}[5m]))
   

4. Saturation — CPU%, memory%, and connection pool usage on one panel

   avg(container_cpu_usage_seconds_total{pod=~"$service.*"}) by (pod)
   

Useful Panel Types

Type	Best for
Time series	Metrics over time — use for most panels
Stat	Single current value (uptime %, current error count)
Gauge	Percentage of a max (disk usage, pool utilization)
Table	Top N slow endpoints, recent error breakdown by status
Heatmap	Latency distribution from a histogram metric
Logs panel	Log lines from Loki shown alongside matching metrics
Bar chart	Comparison across services or endpoints at a point in time

Dashboard Organization Tips

• Add a row per service tier (frontend, API, database, queue)
• Use template variables ($service, $env, $region) so dashboards are reusable
• Link each panel's title to the relevant runbook or alert rule
• Set the default time range to "Last 3 hours" with a 30-second auto-refresh for live debugging
• Pin the error budget remaining as a Stat panel at the top of every service dashboard

---

Red Flags

• High-cardinality label in a Prometheus metric — adding user_id, order_id, or an un-normalized URL path as a label creates millions of time series and can crash Prometheus under memory pressure; only use low-cardinality labels
• Alerting on CPU or memory utilization as a primary signal — CPU at 80% may have zero user impact while a queue backlog at 10% causes data loss; alert on symptoms (error rate, latency) and use resource metrics only for capacity forecasting
• Alert with no for: duration — an alert that fires on the first data point trips on transient 1-second spikes; always add for: 5m or similar to require the condition to be sustained before paging
• Logging at DEBUG level in production — debug logs from a high-traffic service generate gigabytes per hour, burying actionable signals and inflating log storage costs; configure LOG_LEVEL=INFO in production
• Generating a new request_id at each service hop — if each service creates its own ID, you cannot correlate log lines across service boundaries; generate once at the edge and propagate via X-Request-ID or the W3C traceparent header
• Using Summary instead of Histogram for latency metrics — summaries calculate quantiles client-side and cannot be aggregated across multiple instances in PromQL; histograms aggregate correctly across replicas
• SLO defined as "99.9% uptime" without specifying the SLI or success criterion — "uptime" is ambiguous; a well-formed SLO specifies the SLI (fraction of HTTP 2xx responses), the target (99.9%), and the window (rolling 30 days)
• Span attributes containing PII (email, card number, user name) — trace data is often forwarded to third-party backends (Datadog, Honeycomb) with relaxed data retention; scrub or hash sensitive fields before attaching them to spans

Checklist

• All log lines include timestamp, level, service, trace_id, span_id, and request_id
• Log level is INFO in production; DEBUG only in development; never log sensitive data
• Correlation ID is generated at the edge and propagated through all downstream services and async queues
• RED metrics (rate, errors, duration) are instrumented on every service entry point
• Prometheus metric labels have low cardinality — no user IDs, order IDs, or raw URL paths
• Histogram buckets are tuned to the expected latency range of the service
• OpenTelemetry traces are configured with an OTLP exporter pointing at a collector
• Context propagation uses the W3C traceparent header across all HTTP and gRPC calls
• Every alert has a severity label, a for: duration, and a runbook annotation
• SLOs are defined with an explicit SLI, target percentage, and rolling time window
• Error budget policy is documented: what the team does when 75%, 100% of budget is consumed
• Grafana dashboard includes all four golden signals (traffic, latency, errors, saturation) with template variables