| name | qa-observability |
| description | Implement OpenTelemetry logs/metrics/traces, SLI/SLO gates, burn-rate alerts, and APM integrations. Use when adding or validating observability. |
| compatibility | Portable core. Works on Claude Code and Codex. |
| version | 1.1 |
| last_validated | 2026-07-11T00:00:00.000Z |
QA Observability
Use telemetry as a QA signal and a debugging substrate. Treat logs, metrics, traces, and profiles as evidence for test outcomes, release readiness, and production regressions.
Core references live in data/sources.json. Prefer primary docs and re-check volatile external facts before recommending versions, pricing, or vendor features.
Quick Start (Default)
If key context is missing, ask for: critical user journeys, service/dependency inventory, environments (local/staging/prod), current telemetry stack, and current SLO/SLA commitments.
- Establish the minimum bar: correlation IDs, structured logs, traces, and golden metrics (latency, traffic, errors, saturation).
- Verify propagation: confirm
traceparent and your request ID flow across boundaries end-to-end.
- Make failures diagnosable: every integration or E2E failure should capture a trace link or trace ID plus correlated logs, and critical degraded paths should expose structured error metadata such as rate-limit codes, retry hints, and state-transition markers.
- Define SLIs/SLOs and an error budget policy; wire multi-window burn-rate alerts.
- Produce artifacts: a readiness checklist, an SLO definition, and alert rules using
assets/checklists/template-observability-readiness-checklist.md, assets/monitoring/slo/slo-definition.yaml, and assets/monitoring/slo/prometheus-alert-rules.yaml.
Default QA stance
- Treat telemetry as acceptance criteria, especially for integration and E2E flows.
- Require correlation: request ID plus trace ID across service boundaries.
- For critical journeys, make auth redirects, rate limits, and state-sync lag diagnosable with structured codes or attributes instead of opaque text-only errors.
- Prefer SLO-based release gates and burn-rate alerts over raw infrastructure thresholds.
- Treat sampling, cardinality, retention, and cost as quality constraints.
- Redact PII and secrets by default in logs, spans, and attributes.
- Treat logs and profiles as ecosystem-dependent in OpenTelemetry: confirm language and backend support before promising a vendor-neutral implementation.
- The OTel Span Events API (
Span.AddEvent, Span.RecordException) is being deprecated in favour of log-based events (announced March 2026). Write new event instrumentation via the Logs API; existing span event data remains functional during the gradual transition.
Expert Judgment (what a checklist misses)
- Zero observability, first hour: Do not start with a dashboard. Instrument, in order: (1) one structured log line per request with
request_id, trace_id, route, status, duration_ms; (2) the four golden metrics (latency histogram, traffic counter, error counter, saturation gauge) on the single busiest entry point; (3) one end-to-end trace for the single highest-revenue or highest-incident-rate user journey. Skip dashboards, SLOs, and alerting until these three exist — they are the substrate everything else reads from, and building alerting on top of nothing produces false confidence, not safety.
- Logs vs. traces, the actual decision rule: Reach for a trace when the question is "where in this one request did time or an error go" — traces are cheap to read only when propagation already works end-to-end. Reach for a log when the question is "did this business event happen, and what were its exact values" — logs are the durable record of state transitions (payment captured, order cancelled, flag evaluated) that a trace's short retention window will not have next month. When both would answer the question, prefer the trace for latency/causality debugging and the log for audit, compliance, or anything a support engineer needs six weeks from now. Metrics answer "how often" and "how much" cheaply at scale; never reconstruct a rate or percentile by scanning logs or traces if a metric could have carried it.
- Sampling, the trade a checklist glosses over: Head sampling is a bet that the sample is representative — it is not, for errors, once error rate is below the sampling ratio (see
references/sampling-strategies.md for the survivor-bias math). Tail sampling fixes that at the cost of collector-side buffering, memory, and the operational complexity of routing all spans of one trace to the same collector instance. Default recommendation: head-sample low-value traffic at a low, stable ratio; tail-sample (or always keep) errors and traces above a latency threshold; never derive SLIs from sampled trace data — derive them from metrics, which are not sampled.
- Alert fatigue is a design failure, not a tuning problem: if a service has more than 3-5 paging alerts, the fix is consolidation onto SLO burn rate (see
references/alerting-strategies.md), not better thresholds on the existing alerts. An on-call engineer who receives more than ~2 pages per 12-hour shift will start ignoring pages before the quarter ends, regardless of how correct any individual alert is.
- Cost control is a first-class design constraint, not a later optimization pass: cardinality (unique label/attribute combinations) is the dominant cost driver in metrics and the dominant query-latency driver in high-cardinality log/trace backends — a single unbounded label (user ID, order ID, raw URL path) can turn a $200/month Prometheus instance into a $20k/month one. Budget cardinality and retention per signal before instrumenting, not after the bill arrives; treat a proposed new label as a design review item, not a one-line PR.
Workflow
- Establish the baseline: logs, metrics, traces, correlation, and at least one diagnosable critical journey.
- Instrument with OpenTelemetry: auto-instrument first, then add manual spans for business workflow boundaries.
- Verify context propagation across HTTP, queues, and RPC boundaries.
- Define SLIs/SLOs, error budgets, and burn-rate alerts. For low-traffic services, prefer event-count or window sizing guidance from SRE workbook material.
- Make failures diagnosable: attach trace links, key logs, and relevant metrics to failed tests.
- Add performance evidence only after telemetry is trustworthy: profiling, load tests, exemplars, and baselines.
Quick reference
| Task | Recommended default | Notes |
|---|
| Tracing | OpenTelemetry + Collector + OTLP-compatible backend | Jaeger and Tempo are both fine backends; keep Collector as the default routing layer |
| Metrics | Prometheus + Grafana | Use latency histograms; native histograms are GA (Grafana, Oct 2025) — treat rollout as an infrastructure change, test dashboards and alerts first |
| Logging | Structured JSON to stdout/stderr + Collector/filelog pipeline | Never log secrets or high-cardinality IDs as labels; OTel Logs SDK maturity varies by language — check status page before adopting direct SDK path |
| Reliability gates | SLIs/SLOs + error budgets + burn-rate alerts | Gate releases on sustained burn and material regressions |
| Performance | Continuous profiling + load tests + budgets | For stable production profiling use Pyroscope or Parca today; OTel Profiles signal is Alpha (target GA Q3 2026) |
| Zero-code visibility | eBPF-based instrumentation where feasible | Beyla is production-ready and donated to CNCF as opentelemetry-ebpf-instrumentation (OBI); versioning moves fast (3.x line by mid-2026) — check the current release before citing a version; validate kernel 5.8+, runtime, and backend compatibility |
| LLM / AI agent visibility | OTel GenAI semconv + cost metrics + eval events | Stability tier of gen_ai.client/gen_ai.agent spans is unverified and sources conflict as of 2026-07-11 — check github.com/open-telemetry/semantic-conventions-genai before shipping; see references/tools-ebpf-apm.md |
ASCII Flow
Observability QA request
-> Identify critical journeys, services, dependencies, and environments
-> Establish correlation IDs, structured logs, traces, and golden metrics
-> Verify propagation across HTTP, queues, jobs, and RPC boundaries
-> Define SLIs, SLOs, error budgets, and burn-rate alerts
-> Attach trace/log/metric evidence to test and release failures
-> Use profiles and load evidence only after telemetry is trustworthy
Navigation
Open these guides when needed:
| If the user needs... | Read | Also use |
|---|
| A minimal production baseline | references/core-observability-patterns.md | assets/checklists/template-observability-readiness-checklist.md |
| Current Node or Python instrumentation | references/opentelemetry-best-practices.md | assets/opentelemetry/nodejs/opentelemetry-nodejs-setup.md, assets/opentelemetry/python/opentelemetry-python-setup.md |
| Working trace propagation across services | references/distributed-tracing-patterns.md | assets/checklists/template-observability-readiness-checklist.md |
| SLOs, burn-rate alerts, and release gates | references/slo-design-guide.md | assets/monitoring/slo/slo-definition.yaml, assets/monitoring/slo/prometheus-alert-rules.yaml |
| Profiling and load testing evidence | references/performance-profiling-guide.md | assets/load-testing/load-testing-k6.js, assets/load-testing/template-load-test-artillery.yaml |
| A maturity model and roadmap | references/observability-maturity-model.md | assets/checklists/template-observability-readiness-checklist.md |
| What to avoid and how to fix it | references/anti-patterns-best-practices.md | assets/checklists/template-observability-readiness-checklist.md |
| Alert design and fatigue reduction | references/alerting-strategies.md | assets/monitoring/slo/prometheus-alert-rules.yaml |
| Dashboard hierarchy and layout | references/dashboard-design-patterns.md | assets/monitoring/grafana/template-grafana-dashboard-observability.json |
| Structured logging and cost control | references/log-aggregation-patterns.md | assets/observability/template-logging-setup.md |
| RED vs USE vs Golden Signals — choosing a metrics framework |
Implementation guides:
references/core-observability-patterns.mdreferences/opentelemetry-best-practices.mdreferences/distributed-tracing-patterns.mdreferences/slo-design-guide.mdreferences/performance-profiling-guide.mdreferences/observability-maturity-model.mdreferences/anti-patterns-best-practices.mdreferences/alerting-strategies.mdreferences/dashboard-design-patterns.mdreferences/log-aggregation-patterns.mdreferences/methods-red-use-golden.mdreferences/sampling-strategies.mdreferences/tools-ebpf-apm.md- references/information-theory-applied.md — Information-theory applied recipes for observability: alert-noise audit, sampling budget, KL drift detection.
- references/control-theory-applied.md — Control-theory applied recipes for observability: anti-flap thresholds, burn-rate damping, Kalman health score.
- references/queueing-theory-applied.md — Queueing applied recipes for observability: Little's-Law saturation SLOs, USL release-regression, end-to-end latency budgets.
- references/reliability-theory-applied.md — Reliability primitives (MTBF/MTTR, availability, FMEA, error budgets) applied to observability and SLO design.
Templates:
assets/checklists/template-observability-readiness-checklist.mdassets/observability/template-logging-setup.mdassets/opentelemetry/nodejs/opentelemetry-nodejs-setup.mdassets/opentelemetry/python/opentelemetry-python-setup.mdassets/monitoring/slo/slo-definition.yamlassets/monitoring/slo/prometheus-alert-rules.yamlassets/monitoring/grafana/grafana-dashboard-slo.jsonassets/monitoring/grafana/template-grafana-dashboard-observability.jsonassets/load-testing/load-testing-k6.jsassets/load-testing/template-load-test-artillery.yamlassets/performance/frontend/template-lighthouse-ci.jsonassets/performance/backend/template-nodejs-profiling-config.js
Curated sources:
Do / Avoid
Do
- Start with correlation IDs, structured logs, and traces as the minimum bar
- Use SLO-based burn-rate alerts over raw infrastructure thresholds
- Redact PII and secrets by default in logs, spans, and attributes
- Use OpenTelemetry semantic conventions for standard protocol attributes
- Add manual spans around business workflow boundaries, not route-level duplicates
Avoid
- Logging secrets or high-cardinality IDs as metric labels
- Alerting on raw infrastructure metrics without SLO context
- Inventing custom attribute names when semantic conventions exist
- Adding instrumentation without a sampling and cardinality strategy
- Trusting auto-instrumentation alone for business workflow visibility
Scripts
Stdlib-only Python CLI tools. No pip dependencies — run with Python 3.9+.
| Script | Purpose |
|---|
scripts/observability_scorer.py | Maturity scoring, SLO error budget analysis, and readiness report generation |
Quick start:
python scripts/observability_scorer.py maturity \
--input data/sample-observability-profile.json
python scripts/observability_scorer.py slo \
--input data/sample-slo-data.json
python scripts/observability_scorer.py report \
--input data/sample-observability-profile.json \
--slos data/sample-slo-data.json \
--output report.md
See scripts/README.md for full CLI reference and input schema.
Data
Sample input files for the scripts.
| File | Description |
|---|
data/sample-observability-profile.json | Realistic B2B SaaS service observability assessment (checkout-service, Node.js) |
data/sample-slo-data.json | Five SLO definitions with current availability and event counts |
data/sources.json | Curated reference sources for the skill |
Related Skills
Tool selection notes
- Default to OpenTelemetry + OTLP + Collector where possible.
- Prefer SLO-based burn-rate alerting over alerting on raw infrastructure metrics.
- Use semantic conventions for standard protocol attributes; avoid inventing parallel names for HTTP, DB, messaging, or RPC spans.
- When auto-instrumentation already creates server spans, add manual spans around business workflow boundaries instead of duplicating route spans.
- When asked to pick vendors or tools, start from
data/sources.json and validate time-sensitive claims with current docs or releases.
Fact-Checking
- Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
- Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
- Prefer primary sources; report source links and dates for volatile information.
- If web access is unavailable, state the limitation and mark guidance as unverified.
Learnings Loop
Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).
After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.
