Logging & Observability
Structured logging, distributed tracing, and metrics for production systems. Covers the full observability stack from log formatting to alert routing.
Decision Points
1. Log Level Assignment by Event Type
Event occurs →
├── System failure?
├── YES → Service cannot continue?
├── YES → FATAL (page immediately)
└── NO → ERROR (operation failed, will retry)
└── NO → Unexpected condition?
├── YES → WARN (circuit breaker, deprecation)
└── NO → Business event?
├── YES → INFO (user action, payment processed)
└── NO → Debug helper?
├── YES → DEBUG (DB queries, cache hits)
└── NO → TRACE (spans, fine-grained flow)
2. Observability Stack Choice by Scale
Request volume →
├── < 1000/min → Structured logs + simple metrics
├── < 10k/min → Add distributed tracing (10% sampling)
├── < 100k/min → Full OTel + head-based sampling
└── > 100k/min → Tail-based sampling + cardinality limits
3. Alert Threshold Setting
SLI established →
├── User-facing service?
├── YES → Start with 99% SLO (44min/month error budget)
└── NO → Start with 95% SLO (36hr/month error budget)
└── Historical data available?
├── YES → Set threshold at 95th percentile of normal operation
└── NO → Set conservative threshold, tune weekly for 1 month
See references/alerting-patterns.md for full SLI/SLO vocabulary, error budget tables, and PagerDuty alert routing recipes.
4. Trace Sampling Decision
Performance impact →
├── Latency sensitive service?
├── YES → 1-5% sampling rate
└── NO → 10-20% sampling rate
└── Error debugging needed?
├── YES → Always sample errors (status=error)
└── NO → Uniform probability sampling
5. PII Handling Strategy
Field contains sensitive data →
├── Required for debugging?
├── YES → Hash or tokenize (preserve cardinality)
└── NO → Complete redaction
└── Regulatory compliance?
├── GDPR/CCPA → Allowlist approach only
└── PCI → Redact payment fields specifically
Failure Modes
1. Alert Fatigue
- Symptom: Teams ignore pages; alerts stay open for hours
- Diagnosis: Alert-to-incident ratio > 3:1, or SLI threshold too sensitive
- Fix: Raise threshold by 10% increments until alerts correlate with real user impact
2. PII Leakage
- Symptom: Compliance audit flags personal data in logs
- Diagnosis: Search logs for patterns like
"password":, "ssn":, credit card regex
- Fix: Implement allowlist-only logging; redact at logger config level, not call sites
3. Trace Orphaning
- Symptom: Spans appear disconnected; can't follow requests end-to-end
- Diagnosis: Missing
traceparent header propagation on outbound HTTP calls
- Fix: Auto-instrument HTTP clients or manually inject context headers
4. Log-and-Throw Duplication
- Symptom: Same error appears 2-5 times with identical trace IDs
- Diagnosis: Error logged at every stack frame, not just handling boundary
- Fix: Log only where you decide what to do with the error (usually HTTP boundary)
5. Cardinality Explosion
- Symptom: Metrics storage costs spike; query performance degrades
- Diagnosis: Label values exceed 1000 unique values per metric
- Fix: Replace high-cardinality labels (user_id) with bucketed versions (user_tier)
Worked Examples
End-to-End: Distributed Payment Failure Trace
Scenario: Payment service returning 500s sporadically. Need to trace through API Gateway → Payment Service → Bank API.
Step 1: Trace ID Recovery
grep -A5 -B5 "payment_failed" /var/log/api-gateway.log | grep "14:3[0-9]"
Step 2: Cross-Service Trace Following
kubectl logs payment-service | grep "abc123def456"
curl -H "X-Trace-ID: abc123def456" https://bank-api/logs
Decision Point: Sampling trade-off encountered
- Expert notices: Only 10% of traces sampled, but this error trace was captured
- Novice misses: Would increase sampling to 100%, causing performance impact
- Expert decision: Enable error-based sampling (always trace when status=error)
Step 3: Root Cause Analysis
logger.error({
trace_id,
bank_response_code: 502,
bank_error: "insufficient_funds",
our_retry_count: 3
}, "payment_processing_failed");
Resolution: Bank API returns 502 for business logic errors (insufficient funds). Change error handling to return 400 instead of retrying on 502.
Setting Up Structured Logging
Node.js Payment Service Implementation:
import pino from 'pino';
const logger = pino({
level: process.env.LOG_LEVEL ?? 'info',
redact: {
paths: ['req.headers.authorization', 'body.cardNumber', '*.ssn'],
censor: '[REDACTED]'
}
});
export function correlationMiddleware(req, res, next) {
const traceId = req.headers['x-trace-id'] ?? randomUUID();
res.setHeader('x-trace-id', traceId);
requestContext.run({ traceId }, () => {
logger.info({
traceId,
method: req.method,
path: req.path,
userAgent: req.headers['user-agent']
}, 'request_received');
next();
});
}
See references/opentelemetry-setup.md for complete OTel SDK initialization across Node.js, Python, and Go, plus collector deployment and trace propagation configuration.
Failure Modes
Alert Noise Syndrome
- Detection: If alert-to-incident ratio exceeds 3:1, threshold too low
- Cause: SLI thresholds set at 90th percentile instead of 95th percentile
- Fix: Raise threshold by 10% increments until alerts predict real user impact
Schema Drift
- Detection: Dashboard queries break after service deployments
- Cause: Log field names change without coordinated dashboard updates
- Fix: Treat log schema as API contract; version field names explicitly
Sampling Blind Spots
- Detection: Cannot find traces for reported user issues
- Cause: Uniform sampling misses rare but critical error paths
- Fix: Implement intelligent sampling (always sample errors, high-value users)
Quality Gates
Bundled Assets
NOT-FOR Boundaries
This skill handles: Production observability, structured logging, distributed tracing, alerting strategy
Delegate elsewhere:
- Application performance profiling → Use
performance-optimization skill instead
- Load testing and capacity planning → Use
infrastructure-scaling skill
- Database query optimization → Use
database-architect skill
- Security event monitoring → Use
security-architect skill
- Cost optimization for observability tools → Use
cost-optimization skill
