Production-grade observability engineering for AI agents. Covers the full observability lifecycle: OpenTelemetry instrumentation, metrics collection, structured logging, distributed tracing, SLI/SLO management, alert design, and incident response workflows.
When to Use This Skill
Invoke this skill when the user asks to:
Instrument a service, application, or library with OpenTelemetry
Set up monitoring dashboards, alerts, or metrics pipelines (Prometheus, Grafana, Datadog)
Design SLOs/SLIs with error budgets and burn-rate alerts
Configure distributed tracing with sampling strategies and context propagation
Aggregate logs with structured JSON logging, trace correlation, and PII redaction
Build incident response runbooks, communication templates, and postmortems
Manage observability-as-code via Terraform/Pulumi for dashboards and alerts
Optimize observability costs through cardinality management and retention policies
Do NOT use this skill for: general bug fixes (use code-review), Kubernetes deployment configuration (use a k8s skill), or generic DevOps questions without an observability intent.
1. OpenTelemetry Instrumentation
1.1 Quick-Start Patterns by Language
Node.js / TypeScript
// packages: @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
// @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http
// @opentelemetry/sdk-logs @opentelemetry/exporter-logs-otlp-http
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces`,
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/metrics`,
}),
exportIntervalMillis: 15000,
}),
instrumentations: [getNodeAutoInstrumentations()],
serviceName: process.env.OTEL_SERVICE_NAME || 'my-service',
});
sdk.start();
process.on('SIGTERM', () => sdk.shutdown().then(() => process.exit(0)));
Python
# packages: opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
# opentelemetry-instrumentation-flask opentelemetry-instrumentation-requests
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
resource = Resource(attributes={SERVICE_NAME: "my-service"})
# Traces
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
# Metrics
metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter())
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
// dependencies: opentelemetry-bom, opentelemetry-exporter-otlp
// Run with: java -javaagent:opentelemetry-javaagent.jar -jar app.jar
// Auto-instrumentation is the recommended approach for Java.
// Manual configuration (Spring Boot example):
@Configuration
public class OpenTelemetryConfig {
@Bean
public OpenTelemetry openTelemetry() {
Resource resource = Resource.getDefault()
.merge(Resource.create(Attributes.of(
ResourceAttributes.SERVICE_NAME, "my-service")));
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(
OtlpHttpSpanExporter.builder().build()).build())
.setResource(resource)
.build();
SdkMeterProvider meterProvider = SdkMeterProvider.builder()
.registerMetricReader(PeriodicMetricReader.builder(
OtlpHttpMetricExporter.builder().build()).build())
.setResource(resource)
.build();
return OpenTelemetrySdk.builder()
.setTracerProvider(tracerProvider)
.setMeterProvider(meterProvider)
.build();
}
}
.NET
// packages: OpenTelemetry, OpenTelemetry.Exporter.OpenTelemetryProtocol
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
var resourceBuilder = ResourceBuilder.CreateDefault()
.AddService("my-service");
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
.SetResourceBuilder(resourceBuilder)
.AddOtlpExporter()
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.Build();
using var meterProvider = Sdk.CreateMeterProviderBuilder()
.SetResourceBuilder(resourceBuilder)
.AddOtlpExporter()
.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.Build();
Ruby
# gems: opentelemetry-sdk opentelemetry-exporter-otlp
# opentelemetry-instrumentation-all
require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
OpenTelemetry::SDK.configure do |c|
c.service_name = 'my-service'
c.use_all # auto-instrument all registered libraries
c.add_span_processor(
OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
OpenTelemetry::Exporter::OTLP::Exporter.new
)
)
end
1.2 Auto-Instrumentation vs Manual Instrumentation
Approach
When to Use
Pros
Cons
Auto-instrumentation
HTTP frameworks, DB clients, gRPC, messaging
Zero code changes, fast coverage
Less semantic depth, some noise
Manual spans
Business logic, custom operations, critical paths
Full semantic control, business context
Requires code changes, risk of gaps
Hybrid (recommended)
Production services
Best coverage + business context
Requires planning
Auto-instrumentation agents:
Language
Agent/Approach
Node.js
@opentelemetry/auto-instrumentations-node or --require @opentelemetry/auto-instrumentations-node/register
Python
opentelemetry-instrument CLI wrapper
Java
opentelemetry-javaagent.jar (JVM agent)
.NET
OpenTelemetry.AutoInstrumentation NuGet + env vars
Go
eBPF-based auto-instrumentation (experimental)
Ruby
opentelemetry-instrumentation-all gem
1.3 Manual Span Creation Pattern
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def process_order(order_id: str):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("order.source", "api")
# Nested span for a sub-operation
with tracer.start_as_current_span("validate_inventory"):
check_inventory(order_id)
with tracer.start_as_current_span("charge_payment"):
charge(order_id)
span.set_status(trace.Status(trace.StatusCode.OK))
1.4 Context Propagation (W3C TraceContext)
All OpenTelemetry SDKs propagate trace context via W3C TraceContext headers by default:
Pre-aggregate in the Collector: batch + memory_limiter processors
DON'T:
❌ Put user IDs, session IDs, or request IDs as labels
❌ Use unbounded dynamic values (timestamps, IPs, full URLs)
❌ Let GraphQL query names explode cardinality
Relabel example to drop high-cardinality labels:
relabel_configs:
- source_labels: [__name__]
regex: 'http_request_duration_seconds_bucket'
action: drop
# Drop if url label is set (too many unique values)
- source_labels: [url]
regex: '.+'
action: labeldrop
# Errors with trace correlation
{service="order-service", level="error"} | json | line_format "{{.message}}"
# Errors in the last hour, grouped by endpoint
sum by (http_path) (count_over_time({service="api-gateway"} | json | level="error" [1h]))
from opentelemetry.trace import Status, StatusCode
try:
result = process_order(order_id)
span.set_status(Status(StatusCode.OK))
except Exception as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e, attributes={"order_id": order_id})
raise
4.5 Service Maps
Service maps are auto-generated by OTel backends (Grafana Tempo, Jaeger, Datadog) when trace context is consistently propagated across all services. Key requirements:
Every service MUST propagate trace context to downstream calls
Every service MUST export spans to the same collector/backend
Span names should follow semantic conventions for proper grouping
Good: data processed within freshness window (e.g., <5min stale)
Bad: data older than freshness window
SLI = fresh_data_points / total_data_points
Coverage SLI
Good: data that passed validation/filtering
Bad: data dropped/ignored
SLI = processed_data / total_ingested_data
6.3 Error Budget
Error Budget = 1 - SLO_target
For 99.9% SLO over 30 days:
Total minutes: 43,200
Allowed downtime: 43.2 minutes/month
Error budget: 0.1%
Burn rate = actual_error_rate / budgeted_error_rate
A burn rate of 1: consuming budget at exactly the SLO pace
A burn rate of 10: consuming budget 10x faster than allowed
# PagerDuty / Opsgenie escalation policy pattern
# Level 1: Primary on-call (5 min ack)
# Level 2: Secondary on-call (10 min ack, auto-escalate if L1 doesn't ack)
# Level 3: Engineering manager (30 min ack)
# Key practices:
# - Rotations should be at least 1 week (not daily)
# - Never have a single point of failure in the rotation
# - Shadow rotations for new on-call engineers
# - Post-on-call writeup within 24h of rotation end
7.5 Alert Fatigue Prevention
Remove flapping alerts immediately — If it fires and resolves 5x in an hour, it's broken
Aggregate during incidents — Group related alerts, don't page for every instance
📊 INCIDENT UPDATE #{N}: {title}
Time elapsed: {duration}
Status: {investigating/mitigating/resolved}
Current understanding:
- {bullet point findings}
Actions taken:
- {bullet point actions}
Next steps:
- {bullet point next actions}
ETA to resolution: {estimate}
Incident Resolution
✅ INCIDENT RESOLVED: {title}
Duration: {start_time} to {end_time} ({total_duration})
Severity: {SEV0/SEV1/SEV2}
Root Cause: {brief description}
Fix: {what was done to resolve}
Customer Impact: {final impact summary}
Postmortem: scheduled for {date} — {link}
Ticket: {ticket link}
# Postmortem: {incident title}
**Date:** YYYY-MM-DD
**Authors:** {names}
**Severity:** {SEV0/SEV1/SEV2}
**Duration:** {start → end, total duration}
## Summary
{2-3 sentence summary of what happened and impact}
## Customer Impact
- Who was affected and for how long
- What functionality was degraded/unavailable
- Error budget consumed: X% of monthly budget
## Timeline
{Same format as Section 8.4 — copy from incident channel}
## Root Cause Analysis
### Direct Cause
{The technical thing that broke}
### Contributing Factors
- {Why the direct cause was possible}
- {What allowed it to propagate}
- {What delayed detection}
## Detection
- How was it detected? (Alert, user report, social media)
- How long from start to detection? (TTD)
- How long from detection to resolution? (TTR)
- Could detection have been faster? How?
## Resolution
- What action resolved the incident?
- Was any data lost or corrupted?
## Action Items
| Priority | Action | Owner | Due |
|----------|--------|-------|-----|
| P0 | {critical fix to prevent recurrence} | @owner | YYYY-MM-DD |
| P1 | {improvement} | @owner | YYYY-MM-DD |
| P2 | {nice-to-have} | @owner | YYYY-MM-DD |
## Lessons Learned
- What went well
- What went poorly
- Where we got lucky (near-misses)
