mz-release-signoff

Verify a release candidate by comparing metrics across the release boundary, then report the outcome. The release bot posts a request in the `#release` Slack channel naming the version and linking one dashboard per area. Each area is verified by that area's team. This skill covers the shared method plus per-area metric references. ## Prerequisites This workflow needs the Grafana MCP server against `grafana.dev.materialize.com`. Verify it answers before starting: ``` mcp__grafana__list_datasources with type "prometheus" ``` Reading the bot request also needs the Slack MCP server, but a user-pasted dashboard link or version string is enough to proceed without it. ## Read the panel instructions, then apply this skill Two dashboards carry their own `Signing off on Releases` text panel, and they say different things. Read both at the start of every run, because they are maintained separately from this skill: ``` mcp__grafana__get_dashboard_property uid: f248986d-81c6-42a6-817b-00cd7759d808 # compute jsonPath: $.panels[?(@.type=="text")].options.content mcp__grafana__get_dashboard_property uid: e6dc7745-7d35-4968-a23d-689883a984bb # storage-overview jsonPath: $.panels[?(@.type=="text")].options.content ``` If a panel and this skill disagree, the panel wins on *what* to inspect and this skill wins on *how* to measure it. Report the disagreement so one of them gets fixed. ## Every dashboard names its variables differently This is the single most common source of a wasted query. There is no shared convention. | Dashboard | Environment variable | Others | |---|---|---| | `compute-overview` | `namespace` | `version`, `organization`, `cluster_id`, `replica_id`, `worker_id`, `collection_id` | | `storage-overview` | `env` | `cluster`, `replica`, `object_id`, `pod` | | `storage-upsert-sources` | `namespace` | `pod`, `source` | | `persist` | `env` | `pod` | | `environmentd-health` | `env` | `pod` | | `release-health` | `organization`, an organization id, not a namespace | `mz_cluster`, `version` | | `networking` | `namespace` | `pod`, `tenant` | The release bot sets both `var-namespace` and `var-env` on every link for this reason. ## Step 1: Establish the version and the boundaries Every judgement in this workflow is a before/after comparison, so the upgrade times must be pinned first. Ask Prometheus which version each environment runs over the past 10 days: ```promql count by (mz_version) (group by (namespace, mz_version) (max_over_time(v2_mz_compute_cluster_status{mz_version!~".*-dev.*"}[6h]))) ``` Run it as a range query with a 6h step. The `-dev` exclusion drops personal development environments, which run arbitrary old builds and only add noise. The `max_over_time` window matches the step for the reason given under *Step 2*, and it also changes how a transition reads: an environment that upgrades mid-bucket is counted under both versions, so the counts in that one bucket sum to more than the fleet. That is the boundary bucket announcing itself, not an error. Two facts come out of this. The release under test is the highest `vX.Y.0-rc.N` present, and the boundary per stack is the step where the previous version disappears and it appears. Expect two boundaries for the same release, because a later `rc` usually supersedes an earlier one, and both are the new release for sign-off purposes. A second, sharper boundary marker is the process count doubling. Zero-downtime upgrades run the old and new generation side by side, so any per-pod series count roughly doubles for one bucket: ```promql count(avg_over_time(container_memory_working_set_bytes{pod=~".*cluster-.*-replica-.*", container="clusterd"}[6h])) ``` Use those doubling buckets as the boundary, and exclude them from both the before and after samples. They contain two full fleets and will corrupt any sum. ## Step 2: Choose the namespace set The dashboards' `version` variable does not filter panels directly. It narrows the `organization` variable, which narrows `namespace`, and the panels filter on `namespace`. Setting `version` to the new release therefore means "the environments that run the new release now", and the panels then show those same environments on both sides of the boundary. Reproduce that selection explicitly rather than relying on the variable chain. Production and staging need different selectors. In production only the canary environments run a release candidate, and they share the plain released version with every customer environment for part of the week, so a version-based filter loses them. Derive their namespaces once and pin them by name: ```promql group by (namespace, mz_context_org_name) (max_over_time(v2_mz_compute_cluster_status{mz_version=~"<new release>.*"}[<analysis window>])) ``` This derivation is a one-shot whose output is pinned for the rest of the run, so there is no step to match and the lookback spans the analysis window instead, which makes the list independent of any single 6h stretch. Widening it cannot pull in a customer environment, because in production only the canaries run an rc version and `<new release>` is an rc version by *Step 1*. Count the rows before pinning them. This list is the entire production selector for the rest of the run, and a canary region holds one or two environments, so a namespace lost to one missed scrape puts the whole prod sweep on half the fleet with nothing downstream to catch it. Expect one row per canary organization per region. The canary organizations are `Materialize Production Sandbox` and `Materialize Production Analytics`. Which regions carry which has changed over time, so always derive rather than assume, and note that the bot's links and Prometheus have disagreed on this. Sources and Sinks historically inspected only the sandbox environment. The canonical canary list lives in `MaterializeInc/release`, in `templates/issue.md`, as the `--environment` arguments to `bin/deploy upgrade production`. The `storage-overview` sign-off panel points instead at `MaterializeInc/cloud/.github/ISSUE_TEMPLATE/03-release.md`, which no longer exists. In staging every environment runs a release candidate, so a version join both selects the right set and excludes development environments. The join has to span the sample window rather than the evaluation instant, which is what `max_over_time` is doing here: ```promql sum(rate(<metric>[6h]) * on(namespace) group_left() group by (namespace) (max_over_time(v2_mz_compute_cluster_status{mz_version=~".*-rc[.].*"}[6h]))) ``` Escaping note: write `-rc[.]` rather than `-rc\\.` so the expression survives JSON encoding unchanged. **Never read `v2_mz_compute_cluster_status` at an instant. Wrap every use of it in `max_over_time` over the window it is being read across.** For the two range queries that is the step; for Step 2's one-shot derivation it is the analysis window. It comes from the promsql exporter, so one missed scrape leaves a namespace with no series at that timestamp, and each of the three places this workflow reads it then fails silently and in the direction of health: the staging join drops every pod of that namespace from the aggregate, Step 2's derivation drops it from the pinned production selector, and Step 1's boundary query reports its version as having disappeared. On the v26.40.0-rc.3 run the staging working-set sum read 161 GB instead of 229 GB and swap read 0 instead of 57 GB at one bucket, because a single customer environment had no status sample at that instant. Widening the lookback closes all three. Dropping the staging join instead is not an option, since development environments run at roughly a quarter of staging clusterd CPU. ## Step 3: Choose the time window The bot's links default to roughly `now-2d`, resolved to an absolute timestamp at post time. That is too short. The window must contain a clean stretch of the previous release, the boundary, and a clean stretch of the new release, which in practice means seven days or more. Look back far enough to include the previous upgrade, so that the previous boundary is available as a calibration reference. A window of `Sun 12:00 UTC` through the current day at `12:00 UTC` with a 6h step has worked well. The 6h step averages away diurnal structure while leaving enough points to see a step. Sample the three phases separately, and never straddle a boundary or a doubling bucket: * Previous release, steady state. * New release under the earlier `rc`, steady state. * New release under the current `rc`. ## Step 4: Derive the metric roster Take the roster from the dashboard itself so it cannot drift out of date: ``` mcp__grafana__get_dashboard_property uid: <dashboard uid> jsonPath: $.panels[*].title mcp__grafana__get_dashboard_panel_queries uid: <dashboard uid> ``` The panel titles are cheap and give the row structure. The panel queries are not. On the compute dashboard that call returns about 62 KB across 165 panels, which overflows the tool result and is written to a file instead. Slice that file with a script and extract only metric names and label selectors. Never read it whole, and never paste it into the conversation. Row panels appear in the title list but not in the query list, so the two are offset. Match them by title, not by index. `scripts/panel-metrics.py` does the slicing. Take the roster from the dashboard, but resolve every name against the metrics catalog for the release under test before trusting it, as described in *When a metric has gone missing* below. A dashboard panel outlives the metric it plots, so the panel list is a superset of what the build actually exports. ## Step 5: Measure Batch many metrics into one range query by tagging each aggregate with a synthetic label and combining with `or`. This turns twenty tool calls into one: ```promql label_replace(sum(rate(<counter>[6h])), "m", "1_name", "", "") or label_replace(sum(avg_over_time(<gauge>[6h])), "m", "2_name", "", "") ``` Prefix the tags so the result order is stable and readable. Use `rate(x[6h])` for counters and `avg_over_time(x[6h])` for gauges, both matching the step so buckets do not overlap. `scripts/build-range-query.py` writes these expressions from a metric list, because twenty of them by hand is where typos live. It also encodes the one thing that differs between stacks, namely the staging release-candidate join against the pinned production canary namespaces, so the same roster can be run against either without rewriting the selector. Run each area twice. Once across all clusters, and once restricted to the system clusters, which the panel instructions call out because a system-cluster regression is easy to lose in the noise of user clusters. System clusters are `instance_id=~"s[0-9]+"` for the controller and replica metrics, and `pod=~".*cluster-s[0-9]+-replica-.*"` for the container metrics. ## Step 6: Judge There are no thresholds, so the discipline is in ruling out the confounders before believing a signal. **Calibrate before calling anything a regression.** Fleet composition and workload drift produce steps of the same size as most real regressions, so pull three weeks of the metric and look at its natural spread before believing a step. Do this live rather than against a recorded figure, because the spread itself changes as the fleet does. When this was first measured, staging fleet clusterd CPU ranged from 2.83 to 5.55 cores over three weeks, which put a 3% step across the boundary far inside the noise; the useful part of that observation is its size, roughly a factor of two, not the numbers. **Compare at equal post-restart age.** Every upgrade restarts `clusterd`, and a fresh process holds less memory than one that has been running for days. Comparing the pre-upgrade level against the post-upgrade level therefore flatters the new release, and comparing a post-upgrade level against a mid-week pre-upgrade level exaggerates a regression. Sample both sides at a similar age since restart, and treat a monotonic climb within one release as more informative than any level difference across the boundary. **Read the base level of bimodal metrics.** Arrangement record counts and sizes swing by a factor of three or more as periodic dataflows rebuild. Compare the low state against the low state; spike heights are not comparable. **Discount pre-existing noise.** Some staging environments crashloop or carry permanently erroring dataflows. In August 2026 staging us-east-1 sustained roughly 90 `clusterd` restarts per 6h and staging eu-west-1 carried 50 to 400 dataflow errors continuously, both flat across the boundary. Flat means not release-related. The panel instructions suggest filtering such environments out with the dashboard variables, which is worth doing when they mask everything else. **Separate a signal from its location.** A fleet-wide step and one environment moving the fleet total are different findings. Split them with a per-namespace ratio of the two windows: ```promql sort_desc(sum by (namespace) (rate(<metric>[15h])) / sum by (namespace) (rate(<metric>[42h] offset 126h))) ``` Set the instant query's end time to the end of the new window, and pick the offset so the second window lands in the previous release. ## Step 7: Drill deeper When a signal survives Step 6, tighten the aggregation one level at a time. Each level costs one query and narrows the search: 1. Fleet total, which establishes that something moved. 2. Per namespace, which finds the environments involved. 3. Per cluster, via `instance_id`, which separates system from user clusters. 4. Per replica, via `replica_id`, which distinguishes a replica-local effect from a cluster-wide one. 5. Per worker, via `worker_id`, which exposes skew across workers of one replica. 6. Per collection, via `collection_id`, which names the dataflow. Below the metric layer, hand off rather than guess. Use `mz-profile` for CPU and memory attribution inside a process, `mz-query-tracing` for the latency breakdown of a statement, and the Polar Signals MCP server for on-CPU profiles of a running environment. If the signal implicates a specific change, `mz-debug-ci` covers finding the responsible build. ## Step 8: Report State the verdict first, then the method, then the findings. The method matters because the reader has to judge whether the comparison was fair: name the version, the boundaries, the window, the namespace selection, and which regions and rows were covered. Say explicitly what was skipped and why. Separate release-blocking findings from notes. A note is something worth a second look next release that does not survive Step 6 as a regression. Give each note its magnitude and the reason it is not conclusive, so the next person can compare against it rather than rediscover it. Sign-off happens in the bot's thread. As of August 2026 there is a proposal to sign off by reacting to each team's message rather than replying, which was not yet confirmed; check the thread's convention before posting, and never post to Slack without the user asking. ## Areas The bot links one dashboard per area. All are on `grafana.dev.materialize.com`. | Area | Dashboard | UID | Reference | |---|---|---|---| | Compute | `compute-overview` | `f248986d-81c6-42a6-817b-00cd7759d808` | `references/compute.md` | | Sources and Sinks | `storage-overview` | `e6dc7745-7d35-4968-a23d-689883a984bb` | `references/sources-and-sinks.md` | | Sources and Sinks | `storage-upsert-sources` | `ac2de0ab-4a35-48b5-93aa-7e645569debb` | `references/sources-and-sinks.md` | | Persist | `persist` | `m3U1U6ZVk` | `references/persist.md` | | Adapter | `environmentd-health` | `mR1Kg1d4z` | `references/adapter.md` | | Reference | `release-health` | `zKe0K0N4z` | `references/reference-dashboards.md` | | Reference | `networking` | `bHQE8bN4k` | `references/reference-dashboards.md` | Each reference names the metrics, their types and labels, the invariants that hold at any fleet size, and the hazards specific to that area. Read the one for the area you are verifying before running a single query. The `Storage` label the bot still uses refers to work now split between Sources and Sinks and Persist. Sweep sizes differ by an order of magnitude. Compute has about 60 sweep-relevant panels, adapter about 45, storage-overview about 55, and persist 375 panel targets over roughly 230 metrics. For persist, the dashboard's own `should be small` panel defines the sweep; see its reference. The bot links production only, but staging is a larger and earlier sample of the same release, and Adapter verification has covered staging in practice. Prefer running both. Datasource UIDs, which the dashboards take as the `datasource` variable: | Stack | Region | UID | |---|---|---| | Production | us-east-1 | `2K85O21Vz` | | Production | eu-west-1 | `E0J0O2J4k` | | Production | us-west-2 | `ee2e6227-dc2d-4ca5-bb58-ca826fd6d614` | | Staging | us-east-1 | `Ks85Oh14z` | | Staging | eu-west-1 | `JKT0Oh1Vk` | | Staging | us-west-2 | `c979f3ee-16d0-44f1-9f24-9e208e0326d9` | Staging us-west-2 held no compute environments in August 2026, so `v2_mz_compute_cluster_status` returns nothing there. Confirm it is still empty rather than reporting a region as clean. ## Characterizing an area All seven dashboards have a reference, written from a sweep of v26.38.0-rc.3 against v26.37.0 in August 2026 across production canary and staging in us-east-1, with compute additionally covering production eu-west-1, production us-west-2, and staging eu-west-1. To characterize a new dashboard, or to refresh one, run Steps 1 through 6 against it for one release and record in `references/<area>.md` what the run taught you: * Each metric with its type, the labels that select cluster and replica, and what it means. * Which metrics are bimodal, restart-sensitive, or absent when zero. * Hazards and invariants, in one section rather than two. Each entry leads with the property that holds at any fleet size, such as one counter equalling the difference of two others, a gauge whose only meaningful aggregate is a series count, or a metric that is structurally absent in one stack, and then gives the measurement it came from. Splitting these across two sections produced near-verbatim duplication in every reference, because almost every hazard is an invariant with a number attached. * Label naming inconsistencies, duplicate-series hazards, and any panel expression whose filters are not what they appear to be. * Known noise classes, meaning the environments that are unhealthy independently of any release and whose flat contribution can dominate a fleet aggregate. Record invariants, not levels. A recorded level is stale the week after it is written, because environments are created, deleted, and resized continuously, and a stale reference value is worse than none: it invites a comparison the reader should not make. The comparison that matters is always derived in-run, since the before-window of your own query is the only baseline guaranteed to describe the same fleet as the after-window. Coarse order-of-magnitude figures are worth keeping for one narrow purpose: catching a mis-scoped selector, for example a missing `container="clusterd"` that inflates a result tenfold. Keep them dated, keep them to one significant figure, and say plainly that they are not for comparison. ## When a metric has gone missing A reference that names a metric the build no longer exports is worse than no reference, because the sweep reads the empty result as a healthy zero. Two things make this checkable, and both have to be applied to the right version. `doc/user/data/metrics.yml` is the generated catalog of `metric!` invocations in the Rust tree. It is checked in, so it is tagged along with each release, and the working tree's copy describes `main`, which is ahead of whatever you are verifying. Reading that copy during a sign-off answers a question you did not ask. Read the catalog at the two release tags instead and diff them: ``` scripts/catalog-diff.sh v26.38.0 v26.39.0-rc.3 ``` That turns "is this empty panel a regression or a rename?" into a lookup. A name in the removed list explains an after-window that went empty, and it is a documentation fix rather than a finding. A name in the added list explains an empty before-window, and comparing across the boundary on it is meaningless. **Catalog membership is not a string match.** A `metric!` whose name is built with `format!` is catalogued with its placeholders globbed, so `mz_persist_user_bytes` and `mz_persist_compaction_goodbytes` are both covered by the single entry `mz_persist_*_bytes`, and histograms are catalogued as their expanded `_bucket`, `_count`, and `_sum` families rather than under the base name. Grepping the catalog for a literal name will therefore report a live metric as missing. Match against the patterns, as `scripts/lint_metrics.py` does. **The catalog covers only this repository's Rust tree.** It holds no `v2_mz_*`, which the promsql exporter derives from SQL, no `container_*`, `kube_*`, or `kubelet_*` from cAdvisor and kube-state-metrics, and no cloud-side names such as `mz_envd_up` or `mz_external_*`. Those families are confirmed against Prometheus with `list_prometheus_metric_names`, and they are where the sweep has actually been bitten: the compute dashboard's scratch-disk panels plot `kubelet_volume_stats_used_bytes`, which resolves to five series in a single namespace in both production and staging us-east-1 and none of them a `clusterd` volume, so those panels render blank in both stacks. The diff survives that blind spot, because both sides share it and a dynamically named metric is missing from both catalogs and so never appears in either list. The diff therefore never reports a rename that did not happen. It can still miss one, so an empty result that the diff does not explain is not yet cleared. Confirm it against Prometheus with `list_prometheus_metric_names` before reporting the metric as zero. When a name resolves in neither the catalog nor Prometheus, the panel that plots it is dead, and the reference should record that rather than the metric. `git log -S<name> --all -- src/` settles which kind of dead it is, and the two kinds read differently in a report. `mz_query_latency` was real, added in #22049 and deleted in #26647 along with the stash, so the `environmentd-health` panel that still plots its `_bucket` family has been empty since that deletion. `mz_persist_columnar_validation_count` and `mz_txn_placeholder_schema_apply` have never appeared in this repository at all, yet both are live arms of the persist dashboard's `should be small` panel, and the working spelling of the first sits beside it on the same panel as `mz_persist_columnar_op_count` with `op="validation"` and `result="invalid"`. `ci/test/lint-skill-metrics.sh` guards the other direction, so the references cannot rot silently between releases. It resolves every `mz_*` name in this skill against the working tree's catalog, patterns included, and fails on any that neither resolves nor appears in `scripts/metrics-allowlist.txt`. Roster rows that abbreviate a family as `` `mz_foo_bar`, `_baz` `` are expanded against the preceding name, and fenced blocks are extracted before backticks are paired, since pairing across a fence desynchronises every span after it. An abbreviation that resolves against more than one prefix of its base also fails, because the row is then not pinned to one metric: deleting the intended one leaves a shorter cut resolving and the row green while it points at nothing. Spell that member out in full. Two blind spots bound what a green run proves. A name covered by a glob entry cannot be falsified, since the pattern keeps matching after the metric is gone, which currently affects ten names including the whole `mz_persist_*_bytes` family. A bare `` `_count` ``, `` `_sum` ``, or `` `_bucket` `` is read as prose about a histogram rather than as an abbreviation, because that is what it usually is, so a family member has to be written out in full to be checked. The allowlist is the point: it carries one line per name the catalog does not cover, with the reason, so adding to it is a deliberate act and a genuinely renamed metric still fails. It also fails on an allowlist entry that has started resolving, so the exemptions cannot outlive their reason. A failure is an instruction to update the skill, never to suppress the lint. ## Traps **A gauge divided by a limit can fail on duplicate series.** `container_spec_memory_limit_bytes` is exported once per node-label set, and labels such as `karpenter_sh_initialized` flip during node lifecycle, which yields two series for one pod and a `many-to-many matching not allowed` error. Collapse both sides first: ```promql