logging-sucks

name	logging-sucks
description	Use when adding, refactoring, or reviewing logging code and callsites across a codebase. Ensures structured, queryable, context-rich logging.
license	MIT
compatibility	opencode
metadata	{"source":"https://loggingsucks.com/","author":"Boris Tane"}

This skill is adapted from "Logging sucks. And here's how to make it better." by Boris Tane.

When helping with logging, observability, or debugging strategies, first identify the current logger, transport, schema, and deployment/runtime constraints. Then follow these principles:

Core Philosophy

• Logs are optimized for querying, not writing — always design with debugging in mind
• Context is everything — a log without correlation IDs is useless in distributed systems
• Logs are for humans during incidents, not just for compliance or "just in case"
• If you can't filter and search your logs effectively, they provide zero value
• Mental model shift: Log what happened to this request, not what your code is doing

Wide Events / Canonical Log Lines

Instead of scattering 10-20 log lines throughout a request, emit one comprehensive event per request per service. This is the most important concept for effective logging.

• Build the event object throughout the request lifecycle
• Enrich it with context as you process (user info, business data, feature flags)
• Emit once at the end with all context attached
• Include 30-50+ fields containing everything useful for debugging

Example wide event structure:

{
  "timestamp": "2025-01-15T10:23:45.612Z",
  "request_id": "req_8bf7ec2d",
  "trace_id": "abc123",
  "service": "checkout-service",
  "method": "POST",
  "path": "/api/checkout",
  "status_code": 500,
  "duration_ms": 1247,
  "user": {
    "id": "user_456",
    "subscription": "premium",
    "account_age_days": 847
  },
  "cart": {
    "id": "cart_xyz",
    "item_count": 3,
    "total_cents": 15999
  },
  "error": {
    "type": "PaymentError",
    "code": "card_declined",
    "message": "Card declined by issuer"
  },
  "feature_flags": {
    "new_checkout_flow": true
  }
}

This enables queries like: "Show all checkout failures for premium users where new_checkout_flow was enabled, grouped by error code."

Structured Logging Requirements

• Always use key-value pairs (JSON) instead of string interpolation
• Bad: "Payment failed for user 123"
• Good: {"event": "payment_failed", "user_id": "123", "reason": "insufficient_funds", "amount": 99.99}
• Structured logs are machine-parseable, enabling aggregation, alerting, and dashboards

Required Fields for Every Log Event

• timestamp — RFC3339 with timezone (e.g., 2025-01-24T20:00:00Z)
• level — debug, info, warn, error (be consistent, don't invent new levels)
• event — machine-readable event name, snake_case (e.g., user_login_success)
• request_id or trace_id — for correlating logs across a single request
• service — which service/application emitted this log
• environment — prod, staging, dev

Examples of High-Cardinality Fields (always include when available)

• user_id, org_id, account_id — who is affected
• request_id, trace_id, span_id — for distributed tracing
• order_id, transaction_id, job_id — domain-specific identifiers

These fields are what make logs actually queryable during incidents. Without them, you're grepping through millions of lines blindly.

Look for opportunities for high-cardinality fields that can help you identify the root cause of an issue quickly.

Context Propagation

• Pass trace/request IDs through all service boundaries (HTTP headers, message queues, etc.)
• Downstream services must inherit correlation IDs from upstream
• Use middleware or interceptors to automatically inject context into every log
• For async jobs, store and restore the original request context

Log Levels — Use Them Correctly

• debug — Verbose details for local development, usually disabled in production
• info — Normal operations worth recording (user actions, job completions, deploys)
• warn — Something unexpected happened but the system handled it (retries, fallbacks)
• error — Something failed and likely needs human attention (exceptions, failed requests)

Don't log errors for expected conditions (e.g., user enters wrong password)

What to Log

• Request entry and exit points (with duration)
• State transitions (order created → paid → shipped)
• External service calls (with latency and response codes)
• Authentication and authorization events
• Background job starts, completions, and failures
• Retry attempts and circuit breaker state changes

What NOT to Log

• Sensitive data (passwords, tokens, PII, credit card numbers, session cookies, API keys, OAuth codes, private URLs)
• Logs inside tight loops (will generate millions of useless entries)
• Success cases that provide no debugging value
• Redundant information already captured by infrastructure (load balancer logs, etc.)

Language Examples

Runtime	Preferred pattern
Node/TypeScript	Use structured loggers such as pino, winston JSON format, or platform-native structured logs; pass context objects, not interpolated strings.
Python	Use structlog or logging with JSON formatter and extra fields.
Go	Use log/slog with typed attributes and request-scoped context.
Cloudflare Workers	Emit JSON via console.log(JSON.stringify(event)) only after redacting secrets and keeping payload sizes bounded.

Naming Conventions

• Be consistent across all services — agree on field names as a team
• Use snake_case for field names: user_id, not userId or user-id
• Use past-tense verbs for events: payment_completed, not complete_payment
• Prefix events by domain when helpful: auth.login_failed, billing.invoice_created

Performance Considerations

• Avoid logging inside hot paths unless absolutely necessary
• Buffer and batch log writes to reduce I/O overhead
• Consider log levels that can be changed at runtime without redeploying

Sampling Strategy (Tail Sampling)

Use tail sampling — make the sampling decision after the request completes based on its outcome:

1. Always keep errors — 100% of 5xx status codes, exceptions, and failures
2. Always keep slow requests — anything above your p99 latency threshold
3. Always keep specific users — VIP customers, internal testing accounts, flagged sessions
4. Randomly sample the rest — happy, fast requests get sampled at 1-5%

This ensures you never lose the events that matter during incidents while keeping costs manageable.

Review Workflow

When reviewing logging changes:

1. Identify existing field names and logger conventions.
2. Check whether logs answer who/what/when/where/how long/why failed.
3. Check correlation propagation across HTTP, queues, jobs, and external calls.
4. Check redaction and high-cardinality identifiers.
5. Flag noisy logs, tight-loop logs, string interpolation, missing request IDs, and secrets.

During Incidents

• Your logs should answer: Who was affected? What failed? When? Why?
• If you can't answer these within 5 minutes of querying, your logging strategy needs work
• Post-incident: add the logs you wished you had

Common Misconceptions

• Structured logging != wide events — JSON logs with 5 fields scattered across 20 lines are still useless. Wide events are a philosophy: one comprehensive event per request.
• OpenTelemetry won't save you — OTel is a delivery mechanism, not a strategy. It doesn't decide what to log or add business context. You still need to deliberately instrument with wide events.
• High cardinality is only expensive on legacy systems — Modern columnar databases (ClickHouse, BigQuery) are designed for high-cardinality, high-dimensionality data.