logging-observability

license	Apache-2.0
name	logging-observability
description	Structured logging, distributed tracing, and metrics for production applications. [What: OpenTelemetry setup, log level strategy, correlation IDs, SLI/SLO alerting thresholds, Grafana dashboard design, PagerDuty integration] [When: setting up production logging, adding observability to a service, debugging distributed systems, designing alerting, implementing traces/metrics/logs] [Keywords: logging, observability, OpenTelemetry, OTel, structured logs, distributed tracing, correlation ID, metrics, Grafana, Prometheus, PagerDuty, Winston, Pino, structlog, log levels, SLI, SLO, alerting] NOT for application performance profiling (use a profiler), load testing, or database query optimization.
allowed-tools	Read,Write,Edit,Bash(npm:*,npx:*,pip:*,docker:*)
argument-hint	[service description] [stack: node|python|go|java] [current problem: no-logging|no-tracing|alert-fatigue|pii-leak]
metadata	{"category":"Code Quality & Testing","pairs-with":[{"skill":"api-architect","reason":"API request tracing and correlation IDs"},{"skill":"devops-automator","reason":"Deploying collectors and dashboards"},{"skill":"background-job-orchestrator","reason":"Distributed job observability"}],"tags":["observability","logging","tracing","metrics","opentelemetry","monitoring"]}
category	DevOps & Infrastructure
tags	["logging","observability","monitoring","tracing","debugging"]

Logging & Observability

Structured logging, distributed tracing, and metrics for production systems. Covers the full observability stack from log formatting to alert routing.

Decision Points

1. Log Level Assignment by Event Type

Event occurs → 
  ├── System failure?
      ├── YES → Service cannot continue? 
          ├── YES → FATAL (page immediately)
          └── NO → ERROR (operation failed, will retry)
      └── NO → Unexpected condition?
          ├── YES → WARN (circuit breaker, deprecation)
          └── NO → Business event?
              ├── YES → INFO (user action, payment processed)
              └── NO → Debug helper?
                  ├── YES → DEBUG (DB queries, cache hits)
                  └── NO → TRACE (spans, fine-grained flow)

2. Observability Stack Choice by Scale

Request volume →
  ├── < 1000/min → Structured logs + simple metrics
  ├── < 10k/min → Add distributed tracing (10% sampling)
  ├── < 100k/min → Full OTel + head-based sampling
  └── > 100k/min → Tail-based sampling + cardinality limits

3. Alert Threshold Setting

SLI established →
  ├── User-facing service?
      ├── YES → Start with 99% SLO (44min/month error budget)
      └── NO → Start with 95% SLO (36hr/month error budget)
  └── Historical data available?
      ├── YES → Set threshold at 95th percentile of normal operation
      └── NO → Set conservative threshold, tune weekly for 1 month

See references/alerting-patterns.md for full SLI/SLO vocabulary, error budget tables, and PagerDuty alert routing recipes.

4. Trace Sampling Decision

Performance impact →
  ├── Latency sensitive service?
      ├── YES → 1-5% sampling rate
      └── NO → 10-20% sampling rate
  └── Error debugging needed?
      ├── YES → Always sample errors (status=error)
      └── NO → Uniform probability sampling

5. PII Handling Strategy

Field contains sensitive data →
  ├── Required for debugging?
      ├── YES → Hash or tokenize (preserve cardinality)
      └── NO → Complete redaction
  └── Regulatory compliance?
      ├── GDPR/CCPA → Allowlist approach only
      └── PCI → Redact payment fields specifically

Failure Modes

1. Alert Fatigue

• Symptom: Teams ignore pages; alerts stay open for hours
• Diagnosis: Alert-to-incident ratio > 3:1, or SLI threshold too sensitive
• Fix: Raise threshold by 10% increments until alerts correlate with real user impact