Monitoring and Observability
Production systems require visibility into their behavior. Observability goes beyond simple monitoring by enabling you to ask arbitrary questions about system state using logs, metrics, and traces. This guide covers instrumentation, collection, visualization, alerting, and the operational patterns that prevent alert fatigue while keeping systems reliable.
The Three Pillars
| Pillar | What It Captures | Best For | Key Tools |
|---|
| Logs | Discrete events with context | Debugging specific requests, audit trails | ELK, Loki, CloudWatch Logs |
| Metrics | Numeric measurements over time | Trends, thresholds, capacity planning | Prometheus, Datadog, CloudWatch Metrics |
| Traces | Request flow across services | Latency breakdown, dependency mapping | Jaeger, Tempo, X-Ray |
| Question | Signal |
|---|
| "Why did this request fail?" | Logs (event detail) + Traces (call chain) |
| "Is error rate increasing?" | Metrics (counters over time) |
| "Which service is slow?" | Traces (span timing) |
| "What happened at 3:42 AM?" | Logs (timestamped events) |
| "Are we within SLO budget?" | Metrics (error ratio, latency percentiles) |
| "How do services depend on each other?" | Traces (service graph) |
OpenTelemetry SDK Setup
OpenTelemetry provides a vendor-neutral API for emitting all three signals. Instrument once, export anywhere.
Node.js Auto-Instrumentation
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
const sdk = new NodeSDK({
resource: new Resource({
[ATTR_SERVICE_NAME]: 'order-service',
[ATTR_SERVICE_VERSION]: process.env.APP_VERSION || '0.0.0',
}),
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter(),
exportIntervalMillis: 15000,
}),
instrumentations: [
getNodeAutoInstrumentations({
'@opentelemetry/instrumentation-fs': { enabled: false },
'@opentelemetry/instrumentation-http': {
ignoreIncomingPaths: ['/healthz', '/readyz', '/metrics'],
},
}),
],
});
sdk.start();
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));
Manual Span Creation
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('order-service', '1.0.0');
async function processOrder(orderId: string): Promise<Order> {
return tracer.startActiveSpan('processOrder', async (span) => {
try {
span.setAttribute('order.id', orderId);
const order = await tracer.startActiveSpan('db.fetchOrder', async (dbSpan) => {
dbSpan.setAttribute('db.system', 'postgresql');
const result = await db.orders.findById(orderId);
dbSpan.end();
return result;
});
span.addEvent('order.validated', { 'order.total': order.total });
await chargePayment(order);
span.setStatus({ code: SpanStatusCode.OK });
return order;
} (error) {
span.({ : ., : error. });
span.(error);
error;
} {
span.();
}
});
}
Prometheus Metrics
Prometheus uses a pull model -- it scrapes HTTP endpoints at intervals.
| Type | Use Case | Example |
|---|
| Counter | Cumulative count (only goes up) | Total requests, errors, bytes sent |
| Gauge | Current value (goes up and down) | Active connections, queue depth |
| Histogram | Distribution of values in buckets | Request latency, response size |
| Summary | Quantiles calculated client-side | Legacy use -- prefer histograms |
Express Application Metrics
import express from 'express';
import promClient from 'prom-client';
const app = express();
promClient.collectDefaultMetrics({ prefix: 'orderservice_' });
const httpRequestDuration = new promClient.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'route', 'status_code'],
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});
const httpRequestsTotal = new promClient.Counter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'route', 'status_code'],
});
app.use((req, res, next) => {
const end = httpRequestDuration.startTimer();
res.on('finish', () => {
labels = { : req., : req.?. || req., : res..() };
(labels);
httpRequestsTotal.(labels);
});
();
});
app.(, (req, res) => {
res.(, promClient..);
res.( promClient..());
});
Grafana Dashboard
Define dashboards as JSON for version control. Key panels for any service overview:
{
"dashboard": {
"title": "Order Service Overview",
"refresh": "30s",
"panels": [
{
"title": "Request Rate",
"type": "timeseries",
"targets": [{
"expr": "sum(rate(http_requests_total{service=\"order-service\"}[5m])) by (status_code)",
"legendFormat": "{{status_code}}"
}],
"fieldConfig": { "defaults": { "unit": "reqps" } }
},
{
"title": "P99 Latency",
SLO-Based Alerting
Alert on Service Level Objectives, not raw thresholds. This ties alerting to user-visible impact.
| Concept | Definition | Example |
|---|
| SLI | Service Level Indicator -- a measured metric | Proportion of requests < 300ms |
| SLO | Service Level Objective -- target for the SLI | 99.9% of requests < 300ms |
| Error Budget | Allowed failures = 1 - SLO | 0.1% can be slow (43 min/month) |
| Burn Rate | How fast you consume error budget | 2x = burning twice as fast as sustainable |
Multi-Window Burn Rate Alerts
groups:
- name: slo-burn-rate
rules:
- alert: HighErrorBudgetBurnRate_Fast
expr: |
(
sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[1h]))
/ sum(rate(http_requests_total{service="order-service"}[1h]))
) > (14.4 * 0.001)
and
(
sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[5m]))
/ sum(rate(http_requests_total{service="order-service"}[5m]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "Burning error budget 14.4x faster than sustainable"
runbook: "https://wiki.internal/runbooks/order-service-high-errors"
- alert: HighErrorBudgetBurnRate_Slow
expr: |
(
sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[6h]))
/ sum(rate(http_requests_total{service="order-service"}[6h]))
) > (1.0 * 0.001)
for: 30m
labels:
severity: warning
annotations:
summary: "Slowly burning error budget"
- alert: LatencySLOBreach
Log Aggregation Patterns
Structured Logging with Trace Correlation
import pino from 'pino';
import { context, trace } from '@opentelemetry/api';
const logger = pino({
level: process.env.LOG_LEVEL || 'info',
mixin() {
const span = trace.getSpan(context.active());
if (span) {
const { traceId, spanId } = span.spanContext();
return { traceId, spanId };
}
return {};
},
redact: {
paths: ['req.headers.authorization', 'password', 'ssn', 'creditCard'],
censor: '[REDACTED]',
},
});
logger.info({ orderId, userId, total: order.total }, 'Order placed successfully');
Loki Query Examples
# Find errors for a specific trace
{service="order-service"} |= "error" | json | traceId="abc123def456"
# Error rate by service
sum(rate({service=~".+"} |= "error" [5m])) by (service)
# Slow database queries
{service="order-service"} | json | db_duration_ms > 500
Distributed Tracing with Context Propagation
Client -> API Gateway -> Order Service -> Payment Service -> Database
| | | |
trace_id=abc trace_id=abc trace_id=abc trace_id=abc
span_id=001 span_id=002 span_id=003 span_id=004
parent=001 parent=002 parent=003
import { context, propagation } from '@opentelemetry/api';
async function callPaymentService(order: Order): Promise<PaymentResult> {
const headers: Record<string, string> = {};
propagation.inject(context.active(), headers);
return fetch('https://payment-service/charge', {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...headers },
body: JSON.stringify({ orderId: order.id, amount: order.total }),
}).then(r => r.json());
}
function processMessage(message: QueueMessage) {
const parentContext = propagation.extract(context.active(), message.headers);
context.with(parentContext, {
tracer.(, {
span.();
});
});
}
APM Business Metrics
import { metrics } from '@opentelemetry/api';
const meter = metrics.getMeter('order-service', '1.0.0');
const orderCounter = meter.createCounter('orders_total', {
description: 'Total orders processed',
});
const orderValue = meter.createHistogram('order_value_dollars', {
description: 'Distribution of order values',
});
async function completeOrder(order: Order) {
orderCounter.add(1, { payment_method: order.paymentMethod, region: order.region });
orderValue.record(order.total, { currency: order.currency });
}
Alert Fatigue Prevention
| Strategy | Implementation | Impact |
|---|
| Alert on symptoms, not causes | Alert on error rate, not CPU | Fewer alerts, user-facing focus |
| Multi-window burn rates | Short + long window must both fire | Eliminates transient spikes |
| Severity routing | Critical -> page, Warning -> ticket | Right urgency for right signal |
| Alert deduplication | Group related alerts by service | One alert per incident |
| Regular alert review | Monthly: delete or tune noisy alerts | Continuous improvement |
| Require runbook links | Every alert links to a runbook | Responders know what to do |
| Error budget based | Alert when burning budget, not on every error | Tolerates expected failure rate |
| Inhibition rules | Critical suppresses warning for same service | Reduces duplicate noise |
Anti-Patterns
| Anti-Pattern | Problem | Fix |
|---|
| Alerting on raw thresholds (CPU > 80%) | Alerts on non-issues, misses real problems | Alert on SLOs and user-facing symptoms |
| No structured logging | Logs are unsearchable at scale | Use JSON structured logging with consistent fields |
| Missing trace context in logs | Cannot correlate logs with traces | Inject traceId/spanId via logger mixin |
| High-cardinality metric labels | Prometheus memory explosion, slow queries | Never use userId, requestId, or IP as label values |
| Alerting without runbooks | Responders waste time figuring out what to do | Require runbook URL in every alert annotation |
| Sampling 100% of traces | Storage costs explode, collectors overloaded | Use head-based or tail-based sampling (1-10%) |
| Logging PII in plaintext | Compliance violations (GDPR, HIPAA) | Redact sensitive fields, use structured redaction |
| Dashboard sprawl without ownership | Stale dashboards with broken queries | Assign team owners, review quarterly |
| Monitoring only infrastructure | Misses application-level failures | Add business KPI metrics (orders/sec, revenue) |
| Ignoring metric staleness | Stale metrics give false "all clear" | Alert on absent metrics with absent() function |
| No log retention policy | Storage costs grow indefinitely | Set TTLs: 7d hot, 30d warm, 90d cold, archive |
| Synchronous log shipping | Log pipeline failure blocks application | Use async buffered shipping with local fallback |
| No baseline for normal | Cannot detect anomalies | Record baselines during stable periods |
Observability Maturity Checklist
Level 1: Foundations
Level 2: Instrumentation
Level 3: Alerting
Level 4: Advanced
