Skip to main content
monitoring-observability Provides monitoring and observability best practices covering the three pillars (logs, metrics, traces), OpenTelemetry instrumentation, Prometheus/Grafana dashboards, SLO-based alerting, and APM strategies. Use when setting up monitoring, observability, prometheus, grafana, opentelemetry, alerting, tracing, logging, metrics, dashboards, SLOs, or APM.
Ir para a instalação Skills Marketplace Descubra e explore skills de IA criadas pela comunidade.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Mostrar detalhes do prompt Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
npx skills add https://github.com/Tibsfox/gsd-skill-creator --skill monitoring-observabilityO comando permanece em uma só linha. Role horizontalmente para revisá-lo antes de copiar.
Prefere uma cópia local? Baixe os arquivos disponíveis atualmente no SkillsMP.
Baixar Zip Baixando... Mais deste repositório Use the moment a request is underspecified and you are about to act on an ASSUMED goal. Enumerate the candidate goals the request could mean, project the action or artifact each one implies, and check whether they diverge: if two or more plausible goals lead to materially different actions or artifacts, ask exactly ONE targeted clarifying question; otherwise proceed on the most-likely goal and state the assumption in one line. Clarification is an evidence-producing action, not a delay — but one question, never an interrogation. Distinct from gsd-spec-phase (GSD-phase-bound, emits a heavy SPEC.md) and intent-router (fetch strategy, not goal disambiguation). Backed by agent goal-state inference (arxiv 2606.16813v1). Triggers on acting under an assumed goal when the request admits more than one materially different reading.
Use at write time to vector/embedding memory — Grove content-addressed insertion, chroma/pgvector upserts, memory-consolidation promoting session traces to MEMORY.md, or embedding externally-ingested content. Scores each candidate record against a fixed panel of sentinel queries and quarantines any record that would become the nearest neighbor of too many unrelated queries — a hub — whether from adversarial poisoning or accidental over-generality. This is the memory-record-side sibling of skill-injection-guardian (file-side) and the write-side complement of memory-use-warrant (read-side). Quarantine, never silently drop; a human reviews. Backed by the admission-time hubness gate (arxiv 2606.19692v1). Triggers on inserting into vector memory, consolidating memory, and embedding stranger content.
Run this appropriateness check the moment you are about to integrate a retrieved long-term memory — a Grove content-addressed hit, a chroma/pgvector neighbour, a memory-consolidation digest, or a MEMORY.md line — into a response, especially anything touching private origins, Fox Companies IP, credentials, or Center Camp / consent-governed content. It answers a question intent-router never asks: not WHAT to fetch or HOW, but WHETHER a correctly-retrieved item should reach output. Relevance is not appropriateness — a perfect similarity match can still be a boundary violation. Default is FAIL-CLOSED: if in-context authorization is uncertain, the memory may inform behaviour but must not be surfaced. Backed by RBI-Eval (arxiv 2606.06055v1). Triggers on surfacing recalled sensitive memory into a response.
Ocupações relacionadas SOC
Baseado na classificação ocupacional SOC
name monitoring-observability description Provides monitoring and observability best practices covering the three pillars (logs, metrics, traces), OpenTelemetry instrumentation, Prometheus/Grafana dashboards, SLO-based alerting, and APM strategies. Use when setting up monitoring, observability, prometheus, grafana, opentelemetry, alerting, tracing, logging, metrics, dashboards, SLOs, or APM. type skill category ops status stable origin tibsfox modified false first_seen "2026-02-07T00:00:00.000Z" first_path examples/monitoring-observability/SKILL.md superseded_by null
Monitoring and Observability
Production systems require visibility into their behavior. Observability goes beyond simple monitoring by enabling you to ask arbitrary questions about system state using logs, metrics, and traces. This guide covers instrumentation, collection, visualization, alerting, and the operational patterns that prevent alert fatigue while keeping systems reliable.
The Three Pillars
Pillar What It Captures Best For Key Tools Logs Discrete events with context Debugging specific requests, audit trails ELK, Loki, CloudWatch Logs Metrics Numeric measurements over time Trends, thresholds, capacity planning Prometheus, Datadog, CloudWatch Metrics Traces Request flow across services Latency breakdown, dependency mapping Jaeger, Tempo, X-Ray
Question Signal "Why did this request fail?" Logs (event detail) + Traces (call chain) "Is error rate increasing?" Metrics (counters over time) "Which service is slow?" Traces (span timing) "What happened at 3:42 AM?" Logs (timestamped events) "Are we within SLO budget?" Metrics (error ratio, latency percentiles) "How do services depend on each other?" Traces (service graph)
OpenTelemetry SDK Setup
OpenTelemetry provides a vendor-neutral API for emitting all three signals. Instrument once, export anywhere.
Node.js Auto-Instrumentation
import { NodeSDK } from '@opentelemetry/sdk-node' ;
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http' ;
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'
import
PeriodicExportingMetricReader
from
'@opentelemetry/sdk-metrics'
import
from
'@opentelemetry/auto-instrumentations-node'
import
Resource
from
'@opentelemetry/resources'
import
ATTR_SERVICE_NAME
ATTR_SERVICE_VERSION
from
'@opentelemetry/semantic-conventions'
const
new
NodeSDK
resource
new
Resource
ATTR_SERVICE_NAME
'order-service'
ATTR_SERVICE_VERSION
env
APP_VERSION
'0.0.0'
traceExporter
new
OTLPTraceExporter
url
env
OTEL_EXPORTER_OTLP_ENDPOINT
'http://localhost:4318/v1/traces'
metricReader
new
PeriodicExportingMetricReader
exporter
new
OTLPMetricExporter
exportIntervalMillis
15000
instrumentations
getNodeAutoInstrumentations
'@opentelemetry/instrumentation-fs'
enabled
false
'@opentelemetry/instrumentation-http'
ignoreIncomingPaths
'/healthz'
'/readyz'
'/metrics'
start
on
'SIGTERM'
() =>
shutdown
finally
() =>
exit
0
Manual Span Creation import { trace, SpanStatusCode } from '@opentelemetry/api' ;
const tracer = trace.getTracer ('order-service' , '1.0.0' );
async function processOrder (orderId : string Promise <Order > {
return tracer.startActiveSpan ('processOrder' , async (span) => {
try {
span.setAttribute ('order.id' , orderId);
const order = await tracer.startActiveSpan ('db.fetchOrder' , async (dbSpan) => {
dbSpan.setAttribute ('db.system' , 'postgresql' );
const result = await db.orders .findById (orderId);
dbSpan.end ();
return result;
});
span.addEvent ('order.validated' , { 'order.total' : order.total });
await chargePayment (order);
span.setStatus ({ code : SpanStatusCode .OK });
return order;
} catch (error) {
span.setStatus ({ code : SpanStatusCode .ERROR , message : error.message });
span.recordException (error);
throw error;
} finally {
span.end ();
}
});
}
Prometheus Metrics Prometheus uses a pull model -- it scrapes HTTP endpoints at intervals.
Type Use Case Example Counter Cumulative count (only goes up) Total requests, errors, bytes sent Gauge Current value (goes up and down) Active connections, queue depth Histogram Distribution of values in buckets Request latency, response size Summary Quantiles calculated client-side Legacy use -- prefer histograms
Express Application Metrics import express from 'express' ;
import promClient from 'prom-client' ;
const app = express ();
promClient.collectDefaultMetrics ({ prefix : 'orderservice_' });
const httpRequestDuration = new promClient.Histogram ({
name : 'http_request_duration_seconds' ,
help : 'Duration of HTTP requests in seconds' ,
labelNames : ['method' , 'route' , 'status_code' ],
buckets : [0.01 , 0.05 , 0.1 , 0.25 , 0.5 , 1 , 2.5 , 5 , 10 ],
});
const httpRequestsTotal = new promClient.Counter ({
name : 'http_requests_total' ,
help : 'Total number of HTTP requests' ,
labelNames : ['method' , 'route' , 'status_code' ],
});
app.use ((req, res, next ) => {
const end = httpRequestDuration.startTimer ();
res.on ('finish' , () => {
const labels = { method : req.method , route : req.route ?.path || req.path , status_code : res.statusCode .toString () };
end (labels);
httpRequestsTotal.inc (labels);
});
next ();
});
app.get ('/metrics' , async (req, res) => {
res.set ('Content-Type' , promClient.register .contentType );
res.end (await promClient.register .metrics ());
});
Grafana Dashboard Define dashboards as JSON for version control. Key panels for any service overview:
{
"dashboard" : {
"title" : "Order Service Overview" ,
"refresh" : "30s" ,
"panels" : [
{
"title" : "Request Rate" ,
"type" : "timeseries" ,
"targets" : [ {
"expr" : "sum(rate(http_requests_total{service=\"order-service\"}[5m])) by (status_code)" ,
"legendFormat" : "{{status_code}}"
} ] ,
"fieldConfig" : { "defaults" : { "unit" : "reqps" } }
} ,
{
"title" : "P99 Latency" ,
"type" : "timeseries" ,
"targets" : [ {
"expr" : "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=\"order-service\"}[5m])) by (le, route))" ,
"legendFormat" : "{{route}}"
} ] ,
"fieldConfig" : { "defaults" : { "unit" : "s" } }
} ,
{
"title" : "Error Rate (SLO: 99.9%)" ,
"type" : "stat" ,
"targets" : [ {
"expr" : "1 - (sum(rate(http_requests_total{service=\"order-service\", status_code=~\"5..\"}[1h])) / sum(rate(http_requests_total{service=\"order-service\"}[1h])))" ,
"instant" : true } ] ,
"fieldConfig" : { "defaults" : { "unit" : "percentunit" } }
}
]
}
}
SLO-Based Alerting Alert on Service Level Objectives, not raw thresholds. This ties alerting to user-visible impact.
Concept Definition Example SLI Service Level Indicator -- a measured metric Proportion of requests < 300ms SLO Service Level Objective -- target for the SLI 99.9% of requests < 300ms Error Budget Allowed failures = 1 - SLO 0.1% can be slow (43 min/month) Burn Rate How fast you consume error budget 2x = burning twice as fast as sustainable
Multi-Window Burn Rate Alerts
groups:
- name: slo-burn-rate
rules:
- alert: HighErrorBudgetBurnRate_Fast
expr: |
(
sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[1h]))
/ sum(rate(http_requests_total{service="order-service"}[1h]))
) > (14.4 * 0.001)
and
(
sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[5m]))
/ sum(rate(http_requests_total{service="order-service"}[5m]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "Burning error budget 14.4x faster than sustainable"
runbook: "https://wiki.internal/runbooks/order-service-high-errors"
- alert: HighErrorBudgetBurnRate_Slow
expr: |
(
sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[6h]))
/ sum(rate(http_requests_total{service="order-service"}[6h]))
) > (1.0 * 0.001)
for: 30m
labels:
severity: warning
annotations:
summary: "Slowly burning error budget"
- alert: LatencySLOBreach
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{service="order-service"}[5m])) by (le)
) > 0.3
for: 10m
labels:
severity: warning
annotations:
summary: "P99 latency exceeds 300ms SLO"
Log Aggregation Patterns
Structured Logging with Trace Correlation import pino from 'pino' ;
import { context, trace } from '@opentelemetry/api' ;
const logger = pino ({
level : process.env .LOG_LEVEL || 'info' ,
mixin (const span = trace.getSpan (context.active ());
if (span) {
const { traceId, spanId } = span.spanContext ();
return { traceId, spanId };
}
return {};
},
redact : {
paths : ['req.headers.authorization' , 'password' , 'ssn' , 'creditCard' ],
censor : '[REDACTED]' ,
},
});
logger.info ({ orderId, userId, total : order.total }, 'Order placed successfully' );
Loki Query Examples # Find errors for a specific trace
{service="order-service"} |= "error" | json | traceId="abc123def456"
# Error rate by service
sum(rate({service=~".+"} |= "error" [5m])) by (service)
# Slow database queries
{service="order-service"} | json | db_duration_ms > 500
Distributed Tracing with Context Propagation Client -> API Gateway -> Order Service -> Payment Service -> Database
| | | |
trace_id=abc trace_id=abc trace_id=abc trace_id=abc
span_id=001 span_id=002 span_id=003 span_id=004
parent=001 parent=002 parent=003
import { context, propagation } from '@opentelemetry/api' ;
async function callPaymentService (order : Order Promise <PaymentResult > {
const headers : Record <string , string > = {};
propagation.inject (context.active (), headers);
return fetch ('https://payment-service/charge' , {
method : 'POST' ,
headers : { 'Content-Type' : 'application/json' , ...headers },
body : JSON .stringify ({ orderId : order.id , amount : order.total }),
}).then (r =>json ());
}
function processMessage (message : QueueMessage const parentContext = propagation.extract (context.active (), message.headers );
context.with (parentContext, () => {
tracer.startActiveSpan ('processMessage' , (span ) => {
span.end ();
});
});
}
APM Business Metrics import { metrics } from '@opentelemetry/api' ;
const meter = metrics.getMeter ('order-service' , '1.0.0' );
const orderCounter = meter.createCounter ('orders_total' , {
description : 'Total orders processed' ,
});
const orderValue = meter.createHistogram ('order_value_dollars' , {
description : 'Distribution of order values' ,
});
async function completeOrder (order : Order add (1 , { payment_method : order.paymentMethod , region : order.region });
orderValue.record (order.total , { currency : order.currency });
}
Alert Fatigue Prevention Strategy Implementation Impact Alert on symptoms, not causes Alert on error rate, not CPU Fewer alerts, user-facing focus Multi-window burn rates Short + long window must both fire Eliminates transient spikes Severity routing Critical -> page, Warning -> ticket Right urgency for right signal Alert deduplication Group related alerts by service One alert per incident Regular alert review Monthly: delete or tune noisy alerts Continuous improvement Require runbook links Every alert links to a runbook Responders know what to do Error budget based Alert when burning budget, not on every error Tolerates expected failure rate Inhibition rules Critical suppresses warning for same service Reduces duplicate noise
Anti-Patterns Anti-Pattern Problem Fix Alerting on raw thresholds (CPU > 80%) Alerts on non-issues, misses real problems Alert on SLOs and user-facing symptoms No structured logging Logs are unsearchable at scale Use JSON structured logging with consistent fields Missing trace context in logs Cannot correlate logs with traces Inject traceId/spanId via logger mixin High-cardinality metric labels Prometheus memory explosion, slow queries Never use userId, requestId, or IP as label values Alerting without runbooks Responders waste time figuring out what to do Require runbook URL in every alert annotation Sampling 100% of traces Storage costs explode, collectors overloaded Use head-based or tail-based sampling (1-10%) Logging PII in plaintext Compliance violations (GDPR, HIPAA) Redact sensitive fields, use structured redaction Dashboard sprawl without ownership Stale dashboards with broken queries Assign team owners, review quarterly Monitoring only infrastructure Misses application-level failures Add business KPI metrics (orders/sec, revenue) Ignoring metric staleness Stale metrics give false "all clear" Alert on absent metrics with absent() function No log retention policy Storage costs grow indefinitely Set TTLs: 7d hot, 30d warm, 90d cold, archive Synchronous log shipping Log pipeline failure blocks application Use async buffered shipping with local fallback No baseline for normal Cannot detect anomalies Record baselines during stable periods
Observability Maturity Checklist
Level 1: Foundations
Level 2: Instrumentation
Level 3: Alerting
Level 4: Advanced 