observability-operator

name	observability-operator
description	MANUAL-ONLY; never auto-invoke. Operate the observability stack hands-on — instrument services (structured logs with correlation IDs, tenant context, redaction before emission; metrics; traces), wire truthful health checks with timeouts, create or edit dashboards and alert rules (every alert carries severity, owner, runbook link, and a justified threshold), verify telemetry flows by running real queries, and manage noise (dedup, tuning, silences only with owner + expiry). Implements slo-reliability-architect's alerting design rather than re-deriving it; audit events stay with audit-log-architect. EDITS LIVE ALERT/DASHBOARD CONFIG AND EXECUTES OPERATIONAL QUERIES — manual invocation only. Use when asked to instrument a service, add or fix alerts/dashboards/health checks, verify observability after a change, or clean up noisy alerts. Do NOT use to design SLOs or decide what pages (slo-reliability-architect), author incident procedures (incident-response-runbook), or debug a live incident (systematic-debugger).
disable-model-invocation	true

Observability Operator

Purpose

Make a system's behavior visible and its alerts trustworthy by doing the hands-on work: instrumentation with structured, correlated, redacted telemetry; health checks that reflect real dependencies; dashboards and alert rules implementing the reliability design; and post-change verification by executing real queries against the telemetry backend. The discipline is signal-over-inventory: every alert added must be worth waking someone for (severity, owner, runbook link, justified threshold), every noisy alert is a debt item with a tuning plan, and every instrumentation claim is verified by observed data, not by the code compiling.

Use When

• Use when: asked to instrument a service or feature — logs, metrics, traces, correlation IDs, tenant context.
• Use when: creating or editing alert rules, dashboards, or health checks (liveness/readiness/dependency).
• Use when: verifying telemetry after a deploy or migration — "is the new surface actually visible?"
• Use when: an alert is noisy and needs tuning, deduplication, or a silence — bounded, owned, expiring.
• Use when: implementing the alerting/dashboard spec produced by slo-reliability-architect.
• Do NOT use when: deciding WHAT the SLOs/SLIs are or what should page — slo-reliability-architect designs; this skill implements.
• Do NOT use when: designing the tenant-scoped audit trail — audit-log-architect (audit events have integrity/retention semantics telemetry does not).
• Do NOT use when: writing incident procedures — incident-response-runbook.
• Do NOT use when: diagnosing a live failure — systematic-debugger (this skill builds the visibility that debugging uses).

Inputs to Inspect

1. The reliability design: slo-reliability-architect SLIs/SLOs, burn-rate alert spec, symptom-vs-cause alert routing — the contract being implemented.
2. The current observability stack: telemetry backend(s), dashboard/alert config (as code or console), existing instrumentation conventions (logger, metric names, trace propagation).
3. The log taxonomy if one exists: levels, event names, correlation-ID field, redaction rules — extend it, don't fork it.
4. The service's real signal history: current volumes, baseline values for any threshold being set (thresholds without history are guesses — labeled as such with a tuning date).
5. Tenant context propagation: how tenant IDs reach telemetry (and the redaction/access rules for tenant-identifying data in logs).
6. Alert inventory: what already pages/tickets, current noise level, existing silences and their ages.
7. On-call reality: who receives what, via which paging path.

Workflow

1. Instrument with structure: logs as structured events (level, event name, correlation ID, tenant context where the taxonomy allows, duration/outcome fields), metrics for the service's RED/USE signals (rate, errors, duration / utilization, saturation), trace propagation across service boundaries. Redaction applied BEFORE emission — secrets, credentials, and PII never reach the backend on the promise of later filtering.
2. Wire health checks that tell the truth: liveness (process up), readiness (can serve: dependencies checked with timeouts), dependency checks distinguishing hard from soft dependencies — a health check that always returns 200 is instrumentation theater; one that checks a dependency without a timeout is an outage amplifier.
3. Implement dashboards from the design: the SLI panels slo-reliability-architect specified, deploy markers, per-tenant/top-N views where the design calls for them — named and organized so the 3am responder finds them from the alert's runbook link.
4. Create alert rules with full metadata: every rule carries severity, owner, runbook link (incident-response-runbook artifact), and a threshold justified against signal history (or labeled provisional with a tuning date). Page-worthy vs ticket-worthy follows the design's symptom/cause split — this skill does not promote causes to pages on its own.
5. Verify by querying: after instrumenting or wiring, run real queries against the backend — events arriving with expected fields, metrics moving under induced/known load, traces stitching across the boundary, the alert firing in a controlled test (or its query returning truth against historical data). "The code emits it" is not verification; observed data is.
6. Manage noise as debt: for noisy alerts — classify (bad threshold, missing dedup, cause-alert that should be a ticket, genuinely flapping system), fix the classification's cause, and use silences only as bridges: owner + expiry + linked follow-up, never permanent mutes. Alert-noise patterns that indicate product instability route to systematic-debugger as product work, not to threshold inflation.
7. Record what changed: alert/dashboard config changes are config-as-code diffs where the stack supports it (reviewed like code per reviewable-diff-discipline); console-only stacks get the change log written into the report. Telemetry cost impact (new log volume, metric cardinality) is stated — cardinality explosions are billing incidents wearing observability clothes.

Output Format

OBSERVABILITY CHANGES — <service/scope>
Instrumentation: <what was added/changed — events, metrics, traces,
  correlation/tenant fields — with redaction posture>
Health checks: <liveness/readiness/dependency — what each actually checks,
  timeouts>
Dashboards: <created/edited — panels mapped to designed SLIs — link/path>
Alert rules: <rule — severity — owner — runbook link — threshold + its
  justification (history vs provisional+tuning date) — page or ticket>
Verification (executed): <query/test run — observed result — pass/fail per
  claim; anything unverifiable named>
Noise actions: <alert — classification — fix — silences with owner+expiry>
Cost impact: <log volume / metric cardinality deltas, order of magnitude>
Config diffs: <files/paths changed, or console change log>
Handoffs: <design gaps → slo-reliability-architect; product instability →
  systematic-debugger; audit-grade events → audit-log-architect>

Validation Checklist

• Every alert rule has severity, owner, runbook link, and a threshold justification — no orphan alerts.
• Redaction is applied at emission; no secret/PII fields ship to the backend "to be filtered later".
• Health checks check real dependencies with timeouts; none hardcode-pass.
• Every instrumentation claim was verified by an executed query with the observed result recorded — unverifiable items are named.
• Every silence has an owner, an expiry, and a linked follow-up.
• Page/ticket routing follows the slo-reliability-architect design; no cause-alert was promoted to paging unilaterally.
• Telemetry cost impact (volume, cardinality) is stated.
• Audit-grade events were routed to audit-log-architect, not logged as telemetry.

Tenant Isolation Rules

• Tenant IDs in telemetry follow the taxonomy's redaction/access rules; tenant-identifying data in logs is access-controlled like the data it identifies.
• Per-tenant dashboards for internal use never become tenant-facing without scoping; a tenant sees only its own signals.
• Cross-tenant top-N views (noisy-neighbor hunting) are internal-only and access-controlled.

Gotchas

• Metric cardinality is the classic self-inflicted outage: per-tenant or per-user label values on high-volume metrics multiply series counts — bound label sets deliberately, use exemplars/logs for the long tail.
• Readiness checks that cascade (A checks B checks C) turn one slow dependency into fleet-wide unreadiness; check direct dependencies only, with timeouts shorter than the caller's.
• Correlation IDs die at queue boundaries and scheduled jobs — propagate explicitly through message envelopes and job contexts or tracing stops at the first async hop.
• Deploy-time verification against a quiet service proves little: verify against induced load or known traffic, and say which.
• A dashboard nobody can find from the alert is decoration: the runbook link → dashboard path is part of the alert, not documentation garnish.
• Log-based alerts on unstructured strings break on the next reword; alert on structured fields/metrics, never on prose.

Stop Conditions

• Invoked without explicit human request → do not run; this skill edits live alert/dashboard state and executes operational queries (disable-model-invocation: true is load-bearing).
• No alerting design exists and the request is "just add some alerts" → route to slo-reliability-architect first for anything page-worthy; ticket-level hygiene alerts may proceed with that gap named.
• Deleting or silencing an alert that guards a security or data-loss signal → human-approval-boundary with the alert's purpose and the compensating coverage.
• A verification query would run against production with material load/ cost impact (full scans, high-volume test events) → state the impact and get approval before executing.
• Instrumentation would emit data whose redaction status is unclear (possible PII/secrets) → stop and resolve the classification before emission, not after.

Supporting Files

• references/instrumentation-checklist.md — structured-event field standards, RED/USE metric sets, health-check patterns, alert-metadata requirements, noise-classification table.
• evals/evals.json — trigger + behavior cases.
• evals/trigger-evals.json — discrimination within the reliability cluster (slo-reliability-architect, incident-response-runbook) and against shipped audit-log-architect / systematic-debugger.