Integrate an anton workload with kube-prometheus-stack. Use to add a ServiceMonitor, add a PodMonitor, expose metrics to Prometheus, author a Grafana dashboard (sidecar ConfigMap), write a PrometheusRule (alerts or recording rules), verify scrape targets, or debug why a metric or series is missing. Covers the observability stack installed per ADR 0007 / plan 0002.
Integrate an anton workload with kube-prometheus-stack. Use to add a ServiceMonitor, add a PodMonitor, expose metrics to Prometheus, author a Grafana dashboard (sidecar ConfigMap), write a PrometheusRule (alerts or recording rules), verify scrape targets, or debug why a metric or series is missing. Covers the observability stack installed per ADR 0007 / plan 0002.
allowed-tools
Read, Write, Edit, Glob, Grep, Bash
Integrate with kube-prometheus-stack
Task skill for wiring anton workloads into the metrics stack. The stack itself was installed under plan 0002 (closed 2026-04-16); this skill is what you use every time afterward. For the why behind the install shape (retention, replica counts, deferred pillars) read context/adrs/0007-adopt-kube-prometheus-stack.md. This skill assumes you already know the Flux 3-file pattern — if not, load add-flux-app or anton-repo-conventions first.
Namespace: observability. All kps workloads live here.
Chart: kube-prometheus-stack83.5.0 via HelmRepository (the OCI mirror at ghcr.io/prometheus-community/charts is stale at 41.7.4 — do not switch sources without checking Artifact Hub).
Discovery scope: the HelmRelease sets every *MonitorSelector, probeSelector, and ruleSelector to {} (match-all) with nil *NamespaceSelector. You can put a ServiceMonitor / PodMonitor / PrometheusRule in any namespace and Prometheus will pick it up. Co-locate with the app.
Grafana sidecar: looks for ConfigMaps with label grafana_dashboard: "1" in every namespace (searchNamespace: ALL). Hardened: allowUiUpdates: false, disableDeletion: true — UI edits don't persist, which is intentional.
Grafana admin creds: sourced via ExternalSecret from 1Password vault anton, item grafana-admin. Rotating the item does not hot-reload — restart the Grafana pod (helm-charts #3679).
Exposure: Grafana on envoy-internal at grafana.<cluster-domain>. Prometheus + Alertmanager are ClusterIP only — no HTTPRoute.
Alertmanager: zero routes by design (ADR 0007). Don't add a receiver until Trigger 4 (paging need) fires.
Talos-absent scrape targets: kubeControllerManager, kubeScheduler, kubeProxy are disabled in values. Do not re-enable — the endpoints don't exist on Talos (Cilium replaces kube-proxy).
Flux controllers scrape via a PodMonitor, not a ServiceMonitor, because the FluxInstance chart's Services only expose port 80, not the http-prom metrics port. Path: kubernetes/apps/flux-system/flux-instance/app/podmonitor.yaml.
Flux ready/not-ready rollup: use flux_resource_info{ready="True"|"False", suspended, kind} from the flux-operator job. The commonly-cited gotk_reconcile_condition metric is not exported by the controllers on this chart line — queries using it return empty. See flux-metrics.
Prometheus probe recipe (port-forwards in the background are unreliable in this repo — use this instead):
Standard workflow — add metrics for a new (or existing) app
Does the app already have a Service on the metrics port?
Yes → ServiceMonitor. Confirm the Service's spec.ports[].name matches what you'll put in spec.endpoints[].port.
No, but pods expose /metrics on a named container port → PodMonitor.
No /metrics at all → check the chart values; most charts ship a metrics.serviceMonitor.enabled flag. Flipping it is almost always simpler than hand-authoring an SM (this is what we did for Cilium, Longhorn).
Co-locate the manifest with the app under kubernetes/apps/<ns>/<app>/app/. Add it to kustomization.yaml. Labels don't matter to discovery — kps selectors are match-all — but use app.kubernetes.io/name: <app> for readability.
Commit, push, let Flux reconcile. Do not kubectl apply manually.
Verify with the probe recipe: {__name__=~"<something>",job="<ns>/<sm-or-pm-name>"} or check /api/v1/targets?state=active for the scrape pool. Full recipe in verify-scrapes.
If the scrape pool exists but every target is DOWN → it's almost always a port name mismatch, a cert mismatch (HTTPS endpoints need tlsConfig.insecureSkipVerify: true or real TLS), or NetworkPolicy. See debug-missing-series.
Standard workflow — add a dashboard
Author the dashboard JSON. Either export from Grafana UI (dev-only; don't commit UI-generated JSON that won't pin datasource UIDs) or hand-write referencing grafana-dashboards.
Wrap it in a ConfigMap with exactlylabels.grafana_dashboard: "1".
Put it next to the HelmRelease that owns the subject (e.g. kubernetes/apps/storage/longhorn/app/dashboard-longhorn.yaml). Add to the app/kustomization.yaml.
Commit, push, reconcile. The Grafana sidecar watches all namespaces, so the location is purely for repo hygiene.
Verify: open Grafana → Dashboards → Browse. Dashboard appears within ~60s of the ConfigMap existing. If it doesn't, check the sidecar logs: kubectl -n observability logs deploy/kube-prometheus-stack-grafana -c grafana-sc-dashboard.
Validate dashboard queries before committing. Every panel's expr should resolve against live Prometheus — use the probe recipe. A dashboard with broken queries is worse than no dashboard (it hides the fact that no one is watching).
Anton dashboard shape (established 2026-04-17, ADR 0013 / plan 0003)
The cluster-health-glance dashboard is the canonical exemplar — new dashboards extend this shape rather than inventing their own:
Header stat strip at y: 0, h: 4 — five always-visible stat panels covering subsystem headline numbers (Nodes Ready, Pods non-Running, Flux ready %, API 5xx rate, cert expiries). First impression on page load.
Subsystem rows (type: row) below the header, all collapsed: false — one row per conceptual subsystem (Capacity, Workload Health, Control Plane RED, Nodes USE, Pod Events, Storage, Network & CNI). Operator preference: show content on load, don't hide behind a click.
Debug surfaces pair a summary stat strip with a detail table. When the healthy state is an empty table ("no OOMKills", "no restart hotspots"), the table alone reads as "is this broken?" Add a stat tile above with the "0 — healthy" pattern (see grafana-dashboards.md). The dashboard's Workload Health row does this for Unhealthy Workloads / Bad Waiting Pods / Restart Hotspots.
Preserve uid across rewrites so existing bookmarks don't break.
One dashboard, one scroll up to ~30 panels. Split into sibling boards only when a single row outgrows ~6 panels.
Standard workflow — add a PrometheusRule
Decide: alerting (fires a notification) or recording (pre-computes an expensive series). See promql-and-rules for when to use which.
For alerts: follow the "symptom-based, page-on-action" doctrine — alert on what a human can do, not on what's metrically interesting. The plan 0002 revisit (2026-07-15) is the moment to build out the first wave; until then, one-offs are fine.
Put the PrometheusRule next to the thing it alerts on. Anton convention: under the owning app's app/ directory.
Validate before committing: promtool check rules <rendered-file>. You can pipe through yq:
Commit, push, reconcile. Alertmanager has no routes, so firing alerts go to the void until someone wires a receiver — this is fine for silent-alert authoring, not for paging.
Verification one-liners
# List all scrape pool jobs and how many targets each has
kubectl -n observability exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
wget -qO- 'http://localhost:9090/api/v1/targets?state=active' > /tmp/t.json
mise exec -- jq -r '.data.activeTargets | group_by(.scrapePool) | .[] | "\(length)\t\(.[0].scrapePool)"' /tmp/t.json
# List every job label Prometheus knows about
kubectl -n observability exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
wget -qO- 'http://localhost:9090/api/v1/label/job/values' | mise exec -- jq
# Check whether a specific metric name exists at all
kubectl -n observability exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
wget -qO- --post-data='query=<metric_name>' http://localhost:9090/api/v1/query | mise exec -- jq '.data.result | length'
Related skills
add-flux-app — the 3-file pattern every new manifest rides on
debug-flux-reconciliation — when the manifest is committed but the ConfigMap / SM / PM hasn't applied
