| license | Apache-2.0 |
| name | observability-apm-expert |
| description | OpenTelemetry, distributed tracing, Grafana, and Datadog for full-stack observability. Activate on: observability, tracing, OpenTelemetry, Grafana, Datadog, metrics, logging, APM, SLO, alerting. NOT for: application error handling (use relevant language skill), security monitoring (use relevant security skill). |
| allowed-tools | Read,Write,Edit,Bash(npm:*,npx:*,docker:*) |
| category | DevOps & Infrastructure |
| tags | ["observability","opentelemetry","tracing","grafana","metrics"] |
| pairs-with | [{"skill":"service-mesh-microservices-expert","reason":"Service meshes provide automatic telemetry collection"},{"skill":"event-driven-architecture-expert","reason":"Tracing across async event flows"},{"skill":"distributed-transaction-manager","reason":"Saga observability across distributed transactions"}] |
Observability & APM Expert
Implement comprehensive observability with distributed tracing, metrics, structured logging, and SLO-based alerting using OpenTelemetry and modern backends.
Decision Points
1. Sampling Strategy Selection
Error Rate Analysis:
├── Error rate < 0.1%
│ ├── Low cardinality service (< 10k spans/min) → 100% sampling
│ └── High cardinality service (> 10k spans/min) → Tail-based sampling
│ ├── Keep all error traces (100%)
│ ├── Keep slow traces > P95 latency (100%)
│ └── Sample successful traces (1-10%)
└── Error rate > 0.1%
├── Critical service → Keep all errors + 50% successful
└── Non-critical service → Keep all errors + 10% successful
2. Backend Selection Strategy
If self-hosted tolerance = high AND cost sensitivity = high:
├── Use Grafana stack (Tempo + Mimir + Loki)
└── Export via OTLP to unified collector
If operational overhead tolerance = low OR compliance = strict:
├── Cloud vendors (Datadog, New Relic, Honeycomb)
└── Direct SDK exports + OTLP fallback
If hybrid requirements:
├── Critical services → SaaS backend
└── Development/staging → Self-hosted stack
3. Alert Configuration Logic
For each SLO:
├── Define error budget (e.g., 99.9% = 43.2min downtime/month)
├── Calculate burn rates:
│ ├── Fast burn (14.4x) over 1h → Critical alert (2min delay)
│ ├── Medium burn (6x) over 6h → Warning alert (15min delay)
│ └── Slow burn (3x) over 24h → Info alert (1h delay)
└── Link each alert to specific runbook action
Failure Modes
Schema Bloat
Symptom: Metrics cardinality > 10M series, query timeouts, high storage costs
Detection: prometheus_tsdb_head_cardinality growing exponentially
Fix: Add label cardinality limits, aggregate high-cardinality labels, use recording rules
Trace Orphaning
Symptom: Spans appearing disconnected, missing parent-child relationships
Detection: Spans with same trace_id but no parent reference in service map
Fix: Verify context propagation headers (traceparent/tracestate), check async context handling
Alert Fatigue Storm
Symptom: > 10 alerts per incident, team ignoring notifications
Detection: Alert:incident ratio > 5:1, MTTA (time to acknowledge) > 30min
Fix: Implement alert dependencies, use SLO burn rate instead of threshold alerts
Sampling Blind Spots
Symptom: Critical errors not captured in traces, debugging impossible
Detection: Error logs present but corresponding traces missing
Switch to tail-based sampling, increase error trace retention to 100%
