monitoring-setup

name	monitoring-setup
description	Add production observability—health checks, metrics, tracing, alerts, and runbooks—when the task is to instrument or operationalize an existing service.

Monitoring Setup

Overview

Add production-grade observability to any service by instrumenting health checks, metrics, tracing, and alerts as a cohesive system. The skill treats monitoring as structured output — not ad-hoc logging — producing files that integrate with standard observability stacks (Prometheus, Grafana, OpenTelemetry, PagerDuty/OpsGenie).

When to use

• The task is to instrument or operationalize an existing service with health checks, metrics, tracing, alerts, or runbooks.
• The user is preparing a system for production readiness, SLOs, or on-call support.
• The deliverable is observability configuration, code instrumentation, and operational guidance.
• The problem is missing monitoring coverage, not analysis of an already-failing monitoring stack.

Do NOT use when:

• The task is debugging a current incident or broken observability pipeline.
• The request is only vendor-specific dashboard clicking with no reusable monitoring design.
• The work is log aggregation alone without broader observability requirements.

Response format

Always structure the final response with these top-level sections, in this order:

1. Summary — state the task, scope, and main conclusion in 1-3 sentences.
2. Decision / Approach — state the key classification, assumptions, or chosen path.
3. Artifacts — provide the primary deliverable(s) for this skill. Use clear subheadings for multiple files, commands, JSON payloads, queries, or documents.
4. Validation — state checks performed, important risks, caveats, or unresolved questions.
5. Next steps — list concrete follow-up actions, or write None if nothing remains.

Rules:

• Do not omit a section; write None when a section does not apply.
• If files are produced, list each file path under Artifacts before its contents.
• If commands, JSON, SQL, YAML, or code are produced, put each artifact in fenced code blocks with the correct language tag when possible.
• Keep section names exactly as written above so output stays predictable across skills.

Workflow

1. Add health check endpoints

Create three distinct health check endpoints. Each serves a different purpose in orchestration systems like Kubernetes:

Endpoint	Path	Purpose	What to check
Liveness	GET /healthz	"Is the process alive?"	Process is running, not deadlocked. Minimal checks only.
Readiness	GET /readyz	"Can this instance serve traffic?"	Database connected, cache warm, dependencies reachable.
Startup	GET /startupz	"Has initialization completed?"	Migrations run, config loaded, initial data seeded.

Critical distinction: Liveness should NEVER check external dependencies. If your liveness probe checks the database and the DB goes down, Kubernetes will restart your healthy pods — making an outage worse. Liveness = "is this process fundamentally broken?" Readiness = "should traffic be routed here?"

Response format:

{
  "status": "ok",
  "checks": {
    "database": { "status": "ok", "latency_ms": 2 },
    "cache": { "status": "ok", "latency_ms": 1 },
    "external_api": { "status": "degraded", "latency_ms": 450 }
  },
  "version": "1.2.3",
  "uptime_seconds": 84321
}

Return HTTP 200 for healthy, 503 for unhealthy. Include individual check statuses so operators can see which dependency is failing.

2. Instrument metrics collection

Use the RED and USE methods to ensure comprehensive coverage:

RED method (for request-driven services):

Metric	What to measure	Prometheus type	Example
Rate	Requests per second	Counter	http_requests_total{method, path, status}
Errors	Failed requests per second	Counter	http_errors_total{method, path, code}
Duration	Request latency distribution	Histogram	http_request_duration_seconds{method, path}

USE method (for resource-driven components):

Metric	What to measure	Prometheus type	Example
Utilization	% of resource capacity in use	Gauge	db_pool_utilization_ratio
Saturation	Queue depth / backpressure	Gauge	request_queue_length
Errors	Resource-level error count	Counter	db_connection_errors_total

Implementation requirements:

• Use Prometheus client library for the service's language
• Expose metrics at GET /metrics in Prometheus exposition format
• Use histogram buckets appropriate for the service: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] seconds for HTTP
• Label metrics with method, path (normalized), and status_code
• Add a metrics middleware that instruments ALL requests automatically
• Include process metrics (memory, CPU, GC, event loop lag where applicable)

3. Add distributed tracing

Implement OpenTelemetry-compatible tracing with correlation ID propagation:

Trace context propagation:

• Generate a unique trace-id (128-bit hex) for each incoming request without one
• Propagate via W3C Trace Context headers: traceparent, tracestate
• Also support X-Correlation-ID / X-Request-ID for backward compatibility
• Pass trace context to ALL downstream HTTP calls, message queue publishes, and async jobs

Span creation:

• Create a root span for each incoming request
• Create child spans for: database queries, external HTTP calls, cache operations, message queue operations
• Include span attributes: http.method, http.url, http.status_code, db.system, db.statement
• Set span status to ERROR on failures with error message

Configuration output — generate a trace config file:

// tracing.js - OpenTelemetry configuration
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require(
  "@opentelemetry/auto-instrumentations-node",
);
const { OTLPTraceExporter } = require(
  "@opentelemetry/exporter-trace-otlp-http",
);

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ||
      "http://localhost:4318/v1/traces",
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: process.env.OTEL_SERVICE_NAME || "my-service",
});

sdk.start();

4. Configure alert thresholds

Define alerts based on SLOs (Service Level Objectives), not arbitrary values. The process:

1. Define SLOs — e.g., "99.9% of requests complete in < 500ms"
2. Derive SLIs — the metric that measures the SLO (e.g., http_request_duration_seconds)
3. Set burn rate alerts — alert when you're consuming error budget too fast

Alert threshold guidelines:

SLO Target	Burn Rate 1h	Burn Rate 6h	Burn Rate 24h
99.9%	14.4x	6x	3x
99.5%	14.4x	6x	3x
99.0%	14.4x	6x	3x

Alert rule format (Prometheus alerting rules):

groups:
  - name: slo-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > (1 - 0.999) * 14.4
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Error rate burning through SLO budget at 14.4x"
          description: "Current error rate: {{ $value | humanizePercentage }}"
          runbook: "https://runbooks.example.com/high-error-rate"
          dashboard: "https://grafana.example.com/d/slo-overview"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
          > 0.5
        for: 5m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "P99 latency exceeds 500ms SLO target"
          runbook: "https://runbooks.example.com/high-latency"

Escalation policy:

Severity	Response Time	Notification Channel	Escalation After
critical	5 minutes	PagerDuty page	15 min to lead
warning	30 minutes	Slack #alerts	2 hours to team
info	Next business	Slack #monitoring	None

Every alert MUST include a runbook annotation linking to resolution steps.

5. Create runbook templates

Generate a runbook for each alert with this structure:

# Runbook: [Alert Name]

## Alert Details

- **Severity:** critical/warning/info
- **SLO:** Which SLO this protects
- **Dashboard:** Link to relevant Grafana dashboard

## Symptoms

What the operator will observe when this fires.

## Diagnosis Steps

1. Check [specific metric/dashboard]
2. Look for [specific log pattern]
3. Verify [specific dependency]

## Resolution

### If caused by [root cause A]

1. Step-by-step fix

### If caused by [root cause B]

1. Step-by-step fix

## Escalation

- If not resolved in [time]: escalate to [team/person]
- If customer-facing: notify [channel]

6. Generate dashboard configuration

Produce a Grafana dashboard JSON or config covering:

• Overview row: Request rate, error rate, latency P50/P95/P99
• Health row: Health check status, uptime, version
• Resources row: CPU, memory, DB pool utilization, queue depth
• SLO row: Error budget remaining, burn rate, SLO compliance

Checklist

• Liveness endpoint at /healthz — checks process only, NOT dependencies
• Readiness endpoint at /readyz — checks all dependencies with individual status
• Startup endpoint at /startupz — checks initialization completion
• Health responses include status, individual checks, version, uptime
• Metrics endpoint at /metrics in Prometheus exposition format
• RED metrics: request rate, error rate, duration histogram
• USE metrics: utilization, saturation, errors for resources
• Metrics middleware instruments all requests automatically
• Trace context propagation via W3C headers (traceparent)
• Correlation ID generated for requests without trace context
• Child spans for DB queries, HTTP calls, cache, message queues
• Alert thresholds derived from SLOs, not arbitrary values
• Every alert has severity, team label, runbook link, and dashboard link
• Escalation policy defined per severity level
• Runbook template generated for each alert
• Dashboard config covers request metrics, health, resources, and SLOs

Example

Input: "Add monitoring to our Express.js order service"

Output files produced:

File	Contents
health.js	Liveness, readiness, startup route handlers
metrics.js	Prometheus client setup + metrics middleware
tracing.js	OpenTelemetry SDK configuration
alert-rules.yml	Prometheus alerting rules with SLO-based thresholds
runbooks/	One markdown file per alert
dashboard.json	Grafana dashboard configuration

Example health endpoint implementation:

// health.js
const express = require("express");
const router = express.Router();
const { Pool } = require("pg");

const startTime = Date.now();
let startupComplete = false;

// Liveness - process alive, no dependency checks
router.get("/healthz", (req, res) => {
  res.status(200).json({
    status: "ok",
    uptime_seconds: Math.floor((Date.now() - startTime) / 1000),
  });
});

// Readiness - can serve traffic
router.get("/readyz", async (req, res) => {
  const checks = {};
  let healthy = true;

  // Check database
  try {
    const start = Date.now();
    await pool.query("SELECT 1");
    checks.database = { status: "ok", latency_ms: Date.now() - start };
  } catch (err) {
    checks.database = { status: "error", error: err.message };
    healthy = false;
  }

  // Check Redis
  try {
    const start = Date.now();
    await redis.ping();
    checks.cache = { status: "ok", latency_ms: Date.now() - start };
  } catch (err) {
    checks.cache = { status: "error", error: err.message };
    healthy = false;
  }

  res.status(healthy ? 200 : 503).json({
    status: healthy ? "ok" : "unhealthy",
    checks,
    version: process.env.APP_VERSION || "unknown",
    uptime_seconds: Math.floor((Date.now() - startTime) / 1000),
  });
});

// Startup - initialization complete
router.get("/startupz", (req, res) => {
  res.status(startupComplete ? 200 : 503).json({
    status: startupComplete ? "ok" : "starting",
    uptime_seconds: Math.floor((Date.now() - startTime) / 1000),
  });
});

function markStartupComplete() {
  startupComplete = true;
}

module.exports = { router, markStartupComplete };

Common mistakes

Mistake	Fix
Liveness checks database/external deps	Liveness = process health only. Move dependency checks to readiness. DB down + liveness fail = cascading restarts.
Using console.log instead of metrics	Logs are for debugging, metrics are for monitoring. Use counters/histograms for anything you'd alert on.
Arbitrary alert thresholds ("error > 10")	Derive thresholds from SLOs and burn rates. "10 errors" means nothing without knowing request volume.
No correlation ID propagation	Generate trace ID on ingress, propagate to ALL downstream calls. Without this, distributed debugging is impossible.
Missing runbook links on alerts	Every alert must link to a runbook. An alert without a runbook is just noise that trains operators to ignore alerts.
Single health endpoint for everything	Separate liveness/readiness/startup. Kubernetes uses them differently; conflating them causes incorrect pod lifecycle decisions.
Metrics without labels	Always label with method, path, status. Aggregate metrics hide the signal — you need to slice by dimension.
No histogram buckets for latency	Use histograms, not averages. P99 latency matters more than mean. Configure buckets for your expected range.

Quick reference

Component	Output file	Format
Health checks	health.{js,ts,py}	Express/Fastify/Flask routes
Metrics	metrics.{js,ts,py}	Prometheus client + middleware
Tracing	tracing.{js,ts,py}	OpenTelemetry SDK config
Alert rules	alert-rules.yml	Prometheus alerting rules
Runbooks	runbooks/*.md	Markdown per alert
Dashboard	dashboard.json	Grafana dashboard JSON

Key principles

1. Liveness is sacred — Never put dependency checks in liveness probes. A liveness failure triggers a pod restart. If your DB is down and liveness checks the DB, Kubernetes restarts all pods, making recovery harder. Liveness answers only: "is this process fundamentally broken?"

2. SLOs drive alerts — Every alert threshold must trace back to a Service Level Objective. "Error rate > 1%" is meaningless without knowing the SLO. Use burn rate alerting: alert when you're consuming error budget faster than sustainable.

3. Metrics over logs — Anything you would alert on must be a metric, not a log line. Metrics are aggregatable, queryable, and cheap. Log-based alerting is fragile, expensive, and misses patterns that counters catch naturally.

4. Trace everything cross-service — Every request entering the system gets a trace ID. Every downstream call propagates it. Without end-to-end tracing, debugging distributed systems requires correlating timestamps across log streams — which doesn't scale.

5. Alerts without runbooks are noise — Every alert must link to a runbook with diagnosis steps and resolution procedures. Operators receiving alerts without context will either ignore them or waste time investigating from scratch. Runbooks encode institutional knowledge.

Optimization Notes

• Preserve the user's requested output shape exactly and do not substitute generic advice for concrete artifacts.
• Include exact commands, code structures, protocol fields, tags, parameters, file paths, or deliverable sections when the task asks for them.
• Make safety gates explicit before irreversible, destructive, externally visible, or compliance-sensitive actions.
• For multi-step work, present steps in execution order and include validation or rollback checks where relevant.
• Avoid overfitting to a single eval example: express lessons as reusable rules, not as task-specific answers.