production-monitoring

# Production Monitoring and Observability This skill encodes battle-tested observability patterns for production services. Every recommendation comes from real incidents — the ones where you stared at a dashboard that showed nothing useful while users were screaming. Observability is not a feature you bolt on after launch. It is the foundation you build on from day one. --- ## 1. The Three Pillars of Observability Observability is not "having logs." It is the ability to ask arbitrary questions about your system's behavior without deploying new code. The three pillars work together — none is sufficient alone. | Pillar | What It Tells You | Example | |--------|-------------------|---------| | **Logs** | *What happened* — discrete events with context | "User X login failed: expired token" | | **Metrics** | *How the system behaves now* — aggregated numbers over time | "p99 latency is 450ms and rising" | | **Traces** | *Why something is slow* — a request's journey across services | "Postgres query in user-service took 2.3s" | **How they connect:** An alert fires on a **metric** (error rate > 1%). You filter **logs** by the time window to see what errors occurred. You grab a trace ID from the logs and follow the **trace** to the slow service. You fix it and verify the **metric** recovers. Without all three, you are flying blind. --- ## 2. Structured Logging Unstructured logs (`print("something went wrong")`) are useless in production. You cannot filter, aggregate, or dashboard them. ### Python: structlog Setup ```python import structlog, logging def configure_logging(environment: str) -> None: processors: list[structlog.types.Processor] = [ structlog.contextvars.merge_contextvars, structlog.processors.add_log_level, structlog.processors.StackInfoRenderer(), structlog.processors.TimeStamper(fmt="iso"), structlog.processors.format_exc_info, ] renderer = (structlog.processors.JSONRenderer() if environment == "production" else structlog.dev.ConsoleRenderer()) structlog.configure( processors=[*processors, structlog.stdlib.ProcessorFormatter.wrap_for_formatter], logger_factory=structlog.stdlib.LoggerFactory(), wrapper_class=structlog.stdlib.BoundLogger, cache_logger_on_first_use=True, ) formatter = structlog.stdlib.ProcessorFormatter( processors=[structlog.stdlib.ProcessorFormatter.remove_processors_meta, renderer], ) handler = logging.StreamHandler() handler.setFormatter(formatter) root = logging.getLogger() root.handlers.clear() root.addHandler(handler) root.setLevel(logging.INFO) # Silence noisy libraries for lib in ("uvicorn.access", "httpx", "sqlalchemy.engine"): logging.getLogger(lib).setLevel(logging.WARNING) ``` ### Node.js: pino Setup ```typescript import pino from "pino"; const logger = pino({ level: process.env.LOG_LEVEL || "info", transport: process.env.NODE_ENV !== "production" ? { target: "pino-pretty", options: { colorize: true } } : undefined, base: { service: process.env.SERVICE_NAME || "my-service" }, redact: ["req.headers.authorization", "req.headers.cookie", "*.password", "*.token"], }); export default logger; ``` ### Log Levels Discipline Log levels are a contract with your on-call engineers, not a suggestion. | Level | Meaning | Alert? | Example | |-------|---------|--------|---------| | **ERROR** | Needs human attention. An alert should fire. | Yes | Database connection failed, payment processing error, unhandled exception | | **WARNING** | Something unexpected happened but was handled. | No | Rate limit hit, cache miss fallback, deprecated API called | | **INFO** | Business events. The happy path. | No | User created, order placed, deployment started | | **DEBUG** | Developer diagnostics. Never in production. | No | SQL query text, request/response bodies, internal state | **Rules:** - If nobody will read it, do not log it - If it is ERROR, there must be a corresponding alert. Otherwise it is WARNING - DEBUG logs in production are a performance tax with zero value — disable them ### NEVER Log / ALWAYS Include **NEVER log:** passwords, tokens/authorization headers, credit card numbers, SSNs/PII, raw request bodies (may contain secrets). **ALWAYS include in every log line:** ```python logger.info("order_placed", request_id="req-abc123", # Ties to HTTP request trace_id="trace-def456", # Ties to distributed trace user_id="user-789", # Who triggered this order_id="order-012", # What business entity amount=99.99, currency="USD", service="order-service", # Which service emitted this ) ``` ### Correlation IDs Across Services Every request gets a unique ID at the edge. Pass it downstream in headers. Include it in every log line. ```python import uuid, structlog from starlette.types import ASGIApp, Receive, Scope, Send class CorrelationIDMiddleware: def __init__(self, app: ASGIApp) -> None: self.app = app async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: if scope["type"] not in ("http", "websocket"): await self.app(scope, receive, send) return headers = dict(scope.get("headers", [])) request_id = (headers.get(b"x-request-id", b"").decode() or headers.get(b"x-correlation-id", b"").decode() or str(uuid.uuid4())) structlog.contextvars.clear_contextvars() structlog.contextvars.bind_contextvars(request_id=request_id) async def send_with_id(message): if message["type"] == "http.response.start": h = list(message.get("headers", [])) h.append((b"x-request-id", request_id.encode())) message["headers"] = h await send(message) await self.app(scope, receive, send_with_id) # Propagate to downstream services async def call_downstream(client: httpx.AsyncClient, url: str): rid = structlog.contextvars.get_contextvars().get("request_id", "unknown") return await client.get(url, headers={"X-Request-ID": rid}) ``` --- ## 3. OpenTelemetry (OTEL) Setup OpenTelemetry is the vendor-neutral standard. Instrument once, export to Jaeger, Tempo, Datadog, or any OTLP backend. ### Python Dependencies ``` opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-grpc opentelemetry-instrumentation-fastapi, -sqlalchemy, -httpx, -redis ``` ### Complete FastAPI Integration ```python from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor from opentelemetry.instrumentation.redis import RedisInstrumentor def configure_tracing(service_name: str, service_version: str, otlp_endpoint: str) -> None: resource = Resource.create({ SERVICE_NAME: service_name, SERVICE_VERSION: service_version, "deployment.environment": os.getenv("ENVIRONMENT", "development"), }) provider = TracerProvider(resource=resource) provider.add_span_processor(BatchSpanProcessor( OTLPSpanExporter(endpoint=otlp_endpoint, insecure=True) )) trace.set_tracer_provider(provider) # Auto-instrument: creates spans for every request, SQL query, HTTP call, Redis command SQLAlchemyInstrumentor().instrument() HTTPXClientInstrumentor().instrument() RedisInstrumentor().instrument() @asynccontextmanager async def lifespan(app: FastAPI): configure_tracing("order-service", settings.app_version, settings.otlp_endpoint) yield trace.get_tracer_provider().shutdown() app = FastAPI(title="Order Service", lifespan=lifespan) FastAPIInstrumentor.instrument_app(app) ``` ### Manual Spans for Business Logic Auto-instrumentation covers libraries. The most valuable spans are on your business logic. ```python tracer = trace.get_tracer(__name__) async def process_order(order_id: str, user_id: str) -> Order: with tracer.start_as_current_span("process_order", attributes={"order.id": order_id, "user.id": user_id}) as span: with tracer.start_as_current_span("validate_inventory"): if not await check_inventory(order_id): span.set_status(trace.StatusCode.ERROR, "Insufficient inventory") raise InsufficientInventoryError(order_id) with tracer.start_as_current_span("charge_payment") as ps: payment = await charge_payment(order_id) ps.set_attribute("payment.amount", payment.amount) span.add_event("order_completed", attributes={"order.total": payment.amount}) return order ``` ### Context Propagation (W3C TraceContext) Instrumented HTTP clients inject `traceparent` headers automatically. For non-instrumented clients: ```python from opentelemetry.propagate import inject headers = {} inject(headers) # Adds traceparent + tracestate response = await some_client.get(url, headers=headers) ``` ### Exporter Configuration ```bash # Jaeger: docker run -d -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 # Jaeger / Tempo / OTEL Collector ``` --- ## 4. Metrics (RED Method) The RED method gives you the three metrics that matter most for request-driven services. If you measure nothing else, measure these. - **R**ate — requests per second (throughput) - **E**rrors — error rate as a percentage (4xx and 5xx) - **D**uration — latency distribution (p50, p95, p99) ### Prometheus Client Setup (Python) ```python from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST from starlette.types import ASGIApp, Receive, Scope, Send from starlette.responses import Response import time, re REQUEST_COUNT = Counter("http_requests_total", "Total HTTP requests", ["method", "endpoint", "status_code"]) REQUEST_DURATION = Histogram("http_request_duration_seconds", "Request duration", ["method", "endpoint"], buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]) REQUESTS_IN_PROGRESS = Gauge("http_requests_in_progress", "In-flight requests", ["method", "endpoint"]) class PrometheusMiddleware: def __init__(self, app: ASGIApp) -> None: self.app = app async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: if scope["type"] != "http": await self.app(scope, receive, send) return method, path = scope["method"], scope["path"] # Normalize: /users/123 -> /users/{id} to prevent cardinality explosion endpoint = re.sub(r"/\d+", "/{id}", re.sub( r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", "{id}", path)) REQUESTS_IN_PROGRESS.labels(method=method, endpoint=endpoint).inc() start, status_code = time.perf_counter(), 500 async def send_wrapper(message): nonlocal status_code if message["type"] == "http.response.start": status_code = message["status"] await send(message) try: await self.app(scope, receive, send_wrapper) finally: REQUEST_COUNT.labels(method=method, endpoint=endpoint, status_code=status_code).inc() REQUEST_DURATION.labels(method=method, endpoint=endpoint).observe(time.perf_counter() - start) REQUESTS_IN_PROGRESS.labels(method=method, endpoint=endpoint).dec() # Scrape target for Prometheus async def metrics_endpoint(request): return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST) ``` ### Custom Business Metrics Technical metrics tell you the system is healthy. Business metrics tell you the *business* is healthy. ```python ORDERS_PLACED = Counter("orders_placed_total", "Total orders", ["payment_method", "region"]) ORDER_VALUE = Histogram("order_value_dollars", "Order value", buckets=[10, 25, 50, 100, 250, 500, 1000, 5000]) ACTIVE_USERS = Gauge("active_users_current", "Currently active users") ``` ### Cardinality Awareness High cardinality kills Prometheus. This is the #1 Prometheus misconfiguration. ```python # DANGEROUS — user_id has millions of values = millions of time series = OOM Counter("http_requests_total", "...", ["method", "endpoint", "user_id"]) # cardinality bomb # SAFE — all labels have bounded values