Use when wiring metrics, logs, traces, and operational readiness for a Kubernetes workload or namespace after operations and reliability have decided the observability substrate, SLOs, and runbook hooks. Produces ServiceMonitor/PodMonitor scrape wiring, kube-state-metrics and cAdvisor signal coverage, log-shipping configuration (Fluent Bit / Vector / Loki / cloud-native), OpenTelemetry collector wiring for traces, SLO alert rules, audit-log collection, and runbook inputs for pod-eviction storms, ImagePullBackOff sprees, and node-pressure incidents. Do not use for base manifest authoring, network/identity policy, autoscaler tuning, image hardening, or the observability backend/substrate decision and cluster provisioning; use the other Family G archetype skills (substrate selection is upstream; cluster provisioning is out of family).
Use when wiring metrics, logs, traces, and operational readiness for a Kubernetes workload or namespace after operations and reliability have decided the observability substrate, SLOs, and runbook hooks. Produces ServiceMonitor/PodMonitor scrape wiring, kube-state-metrics and cAdvisor signal coverage, log-shipping configuration (Fluent Bit / Vector / Loki / cloud-native), OpenTelemetry collector wiring for traces, SLO alert rules, audit-log collection, and runbook inputs for pod-eviction storms, ImagePullBackOff sprees, and node-pressure incidents. Do not use for base manifest authoring, network/identity policy, autoscaler tuning, image hardening, or the observability backend/substrate decision and cluster provisioning; use the other Family G archetype skills (substrate selection is upstream; cluster provisioning is out of family).
Kubernetes Observability and Operations Readiness
When to use
Invoke when making a Kubernetes workload or namespace observable and operable for production — scrape wiring, log shipping, tracing, SLO alerts, and runbook inputs — or when auditing an inherited workload that is a black box in incidents.
Do not use for: base manifest authoring (use k8s-workload-packaging-and-manifest); NetworkPolicy/RBAC (use k8s-network-and-identity-policy); autoscaler tuning that consumes these metrics (use k8s-scaling-and-resilience-topology); image hardening and admission (use k8s-supply-chain-and-image-hardening); choosing the observability backend/substrate (decided upstream in infrastructure-platform.md / architecture/operations); cluster provisioning and control-plane audit enablement (out of Family G — owned by the cloud platform stack and Terraform; this skill wires workload/namespace-level collection).
Inputs
Required:
A workload manifest set from k8s-workload-packaging-and-manifest (the Service/pod this skill makes observable; it exposes a metrics port and stdout logs).
Approved architecture/operations decisions on the observability substrate, alerting destinations, and runbook hooks, or explicit confirmation they are intentionally deferred.
Optional:
Approved architecture/reliability SLOs and error-budget policy (drive alert thresholds).
The workload tier from architecture-schema (drives alert severity, retention, sampling).
The observability stack in use (Prometheus/Grafana/Loki/Tempo or cloud-native) — consumed, not chosen here.
The trace propagation format (W3C tracecontext / B3) and sampling target.
Known operational failure modes for this workload (informs runbook inputs).
Operating rules
Never generate vanity telemetry. A metric nobody alerts on, a log nobody can query, and a trace with no propagation are not observability — they are cost.
Consume architecture/operations and architecture/reliability; do not invent decisions. The observability substrate, alert destinations, SLOs, and runbook hooks are decisions made upstream. If a needed decision is missing, pause and raise an ADR candidate rather than guessing.
The observability backend is consumed, not chosen. ServiceMonitor/log-shipper/collector targets the substrate named upstream. Selecting Prometheus-vs-Datadog here is out of scope — a missing substrate is an ADR candidate, not a default.
Every workload exposes the three signals. Metrics via a scraped endpoint (ServiceMonitor/PodMonitor, not just bare annotations where the stack supports CRDs), logs to stdout shipped structured, traces emitted with context propagation. A workload missing one signal is not operations-ready.
Alerts are tied to SLOs and are actionable. Each alert rule maps to an SLO or a known failure mode, has a severity from the tier, and names the runbook. A threshold with no runbook and no SLO is rejected.
Signal coverage includes the platform, not just the app. kube-state-metrics (object state), cAdvisor (container resource), and node-pressure signals are wired so eviction storms and node pressure are visible — not only app latency.
Logs are structured and PII-safe by default. Shipped logs are structured (JSON), correlate to traces (trace/span IDs), and do not carry secrets or PII; redaction is in the pipeline, not assumed downstream.
Audit-log collection is scoped to what this layer owns. Workload/namespace-relevant audit events (RBAC denials, policy-deny from the network archetype, admission rejections) are collected; enabling the API-server audit policy is a control-plane concern handed off, not done here.
Runbook inputs are concrete, not prose. For pod-eviction storms, ImagePullBackOff sprees, and node-pressure incidents: the signal that fires, the query to confirm, the first diagnostic step, and the escalation — structured for the operations runbook, not a paragraph.
This skill owns observability + operations readiness wiring. Base manifests, network/identity, autoscaler tuning, image hardening, and the substrate decision are named handoffs, not implemented here.
A workload whose alerts have not been test-fired and whose runbook inputs have not been dry-run is not operations-ready.
Output contract
The generated observability and operations wiring MUST conform to:
observability-standards — metrics scraped via ServiceMonitor/PodMonitor, structured stdout logs shipped, traces with context propagation, SLO-tied alerts, trace/log correlation.
deployment-standards — observability wiring shipped as reproducible manifests, not click-ops dashboards; env-agnostic.
naming-conventions — ServiceMonitor/alert/dashboard object names kebab-case, suffixed by kind when ambiguous.
Upstream contract: architecture/operations is the source of truth for the observability substrate, alert destinations, and runbook hooks; architecture/reliability is the source of truth for SLOs and error budgets. If a needed decision is missing, pause and raise an ADR candidate. Base manifests, network/identity, autoscaler tuning, image hardening, and substrate selection are named handoffs.
Progressive references
Read references/k8s-observability-operations-playbook.md when wiring any owned signal or checking the anti-pattern list.
Read references/k8s-observability-operations-quality-rubric.md before declaring the workload operations-ready.
Use assets/k8s-observability-operations.template.md as the ServiceMonitor/log/trace/alert/runbook pattern reference.
Process
Gather context: load architecture/operations (substrate, alert destinations, runbook hooks) and architecture/reliability (SLOs, error budgets). Resolve the workload tier from architecture-schema. Confirm the workload exposes a metrics port and stdout logs. If a needed decision is missing, raise an ADR candidate before proceeding.
Wire metrics: a ServiceMonitor/PodMonitor (or the substrate's equivalent) scraping the workload endpoint; confirm kube-state-metrics and cAdvisor coverage so object state and container resource are visible.
Wire log shipping: the upstream-named shipper (Fluent Bit / Vector / cloud-native) collecting stdout, parsing to structured JSON, injecting trace/span IDs, redacting PII/secrets in-pipeline, to the named log backend.
Wire tracing: the OpenTelemetry collector (or substrate equivalent), context propagation format, and the tier-correct sampling rate; confirm trace↔log correlation IDs line up.
Author SLO alert rules: each tied to an SLO or a known failure mode, severity from the tier, destination from the upstream alert config, and a named runbook — no orphan thresholds.
Wire platform-signal coverage: node-pressure, eviction, and ImagePullBackOff signals so cluster-level incidents are visible, not only app metrics.
Scope audit-log collection: collect workload/namespace RBAC-denial, network-policy-deny, and admission-rejection events; hand off API-server audit-policy enablement to the control-plane owner.
Produce runbook inputs: for pod-eviction storms, ImagePullBackOff sprees, and node-pressure — the firing signal, the confirming query, the first diagnostic step, the escalation — structured for the operations runbook.
Verify: test-fire each alert (synthetic breach) and confirm it routes to the destination; dry-run each runbook input against the live signals; document any check that cannot run in the environment.
Emit the wiring under k8s/observability/ plus operations-readiness.md (signal inventory, alert→SLO→runbook map, sampling/retention) and the named handoff list. Validate against observability-, deployment-standards, naming-conventions, and architecture-schema. Revise until all pass or the gap is documented.
Outputs
Required:
ServiceMonitor/PodMonitor scrape wiring with kube-state-metrics and cAdvisor coverage.
Log-shipping configuration (structured, trace-correlated, PII/secret-redacted) to the named backend.
OpenTelemetry collector wiring with context propagation and tier-correct sampling.
SLO alert rules, each tied to an SLO/failure mode, severity, destination, and a named runbook.
Related Family G archetype skills: k8s-network-and-identity-policy (produces policy-deny audit events collected here), k8s-scaling-and-resilience-topology (consumes the metrics wired here), k8s-supply-chain-and-image-hardening.
