observability-learn

name	observability-learn
source	botcore
description	Guides implementation of structured logging, metrics collection, and distributed tracing for production services. Covers the three pillars of observability (logs, metrics, traces), OpenTelemetry integration, correlation ID propagation, alerting thresholds, and common anti-patterns. Use when adding monitoring to a service, debugging production issues, improving error visibility, setting up alerting, or instrumenting business metrics. Triggers: observability, logging, metrics, tracing, monitoring, OpenTelemetry, alerts, instrumentation, structured logs, correlation ID.
version	1.0.0
triggers	["observability","logging","metrics","tracing","monitoring","OpenTelemetry","alerts","instrumentation","structured logs","correlation ID"]
portable	true

Implementing Observability

Best practices for structured logging, metrics collection, and distributed tracing in production services.

Capabilities

1. Design structured logging with proper log levels and JSON output
2. Implement correlation ID propagation across service boundaries
3. Instrument services with counters, gauges, and histograms
4. Set up OpenTelemetry for automatic and manual instrumentation
5. Define alerting thresholds using RED and USE methodologies
6. Identify and avoid common observability anti-patterns
7. Build dashboards covering rate, errors, and duration

Core Principles

• Three Pillars -- Observability rests on logs (event records), metrics (numeric measurements), and traces (request flow). Each pillar serves a distinct debugging purpose; all three are needed for full visibility.
• Structure Over Strings -- Always emit structured JSON logs rather than interpolated strings. Structured logs are searchable, parseable, and machine-readable.
• Correlation Everywhere -- Every request must carry a correlation ID that propagates across all downstream services and appears in every log entry, metric label, and trace span.
• Bounded Cardinality -- Metric labels must use bounded value sets. Unbounded labels (user IDs, request paths with parameters) cause metric explosion and storage issues.
• Actionable Alerts Only -- Every alert must have a clear owner, a runbook, and a required action. If an alert fires and nobody needs to act, remove it.
• Sensitive Data Exclusion -- Never log PII, secrets, tokens, or credentials. Redact or mask sensitive fields before they reach the logging pipeline.

Quick Reference

Log Levels

Level	Use For
error	Failures requiring immediate attention
warn	Recoverable issues, deprecations
info	Business events (user created, payment processed)
debug	Development troubleshooting
trace	Detailed execution flow

Metric Types

Type	Behavior	Example
Counter	Monotonic, always increases	ordersTotal.inc({ status: "completed" })
Gauge	Can increase or decrease	activeConnections.set(pool.size)
Histogram	Distribution of values	requestDuration.observe(elapsed)

Methodology Frameworks

Framework	Applies To	Measures
RED	Services	Rate, Errors, Duration
USE	Resources	Utilization, Saturation, Errors
Business	Product	Signups, Orders, Revenue

Workflow

1. Structured Logging

// BAD: Unstructured string interpolation
console.log(`User ${userId} failed to login: ${error}`);

// GOOD: Structured JSON with context
logger.error({
  event: "login_failed",
  userId,
  error: error.message,
  stack: error.stack,
  requestId: ctx.requestId,
});

2. Correlation ID Propagation

// Middleware to inject and propagate correlation ID
app.use((req, res, next) => {
  req.correlationId = req.headers['x-correlation-id'] || crypto.randomUUID();
  res.setHeader('x-correlation-id', req.correlationId);
  next();
});

// Include in every log entry
logger.info({ correlationId: req.correlationId, event: "order_placed" });

3. OpenTelemetry Setup

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'my-service',
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

4. Alerting Thresholds

Alert Type	Threshold	Response
Error rate	> 1% for 5 min	Page on-call
Latency p99	> 2s for 10 min	Investigate
Saturation	> 80% for 15 min	Scale up

Anti-Patterns to Avoid

Avoid	Instead
Logging PII or secrets	Redact sensitive data before logging
High-cardinality metric labels	Use bounded label values only
Logging in hot paths without sampling	Sample or aggregate high-frequency events
Noisy or non-actionable alerts	Ensure every alert has an owner and runbook

Checklist

• Structured JSON logging configured with appropriate log levels
• Correlation ID generated at ingress and propagated to all downstream calls
• Error context captured including stack traces and request metadata
• Key business metrics instrumented (signups, orders, revenue)
• RED metrics instrumented for all services (rate, errors, duration)
• USE metrics instrumented for critical resources (CPU, memory, connections)
• Health check endpoints exposed and monitored
• Alerting rules configured with clear thresholds and response procedures
• Dashboards created for RED metrics and business KPIs
• Sensitive data redacted from all log output
• Metric label cardinality verified as bounded
• OpenTelemetry or equivalent tracing SDK initialized

When to Escalate

• Infrastructure team -- When setting up centralized log aggregation, metric storage backends (Prometheus, Datadog), or tracing collectors (Jaeger, Zipkin) at the platform level.
• Security team -- When uncertain whether a data field constitutes PII or when logging requirements conflict with compliance obligations (GDPR, HIPAA).
• SRE / On-call team -- When defining SLOs, error budgets, or paging thresholds that affect incident response workflows.
• Architecture team -- When introducing a new observability vendor or making cross-cutting changes to the instrumentation strategy across multiple services.