release-monitoring-report

# Release Monitoring Report Produce a **version-over-version** release-health report covering the Android **Broker** and/or the **Authenticator** app in one self-contained HTML file at `$env:USERPROFILE\android-release-reports\release-report-broker-<bv>-auth-<av>-<YYYY-MM-DD>.html` (home folder, **outside the workspace** — reports never get committed). Omitted apps drop out of the filename, so broker-only or authenticator-only runs are first-class. The report answers two questions per app: **what changed** this release (KPIs + tables vs the baseline) and **why** (error-code movers / per-scenario deltas). It ends in a clear **verdict** per app: SAFE to proceed / WATCH / HOLD. The output mirrors the canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html) — a real filled example kept as the structural + visual reference. The Step 1 bootstrap script copies it into `~/android-release-reports/…` and **you edit it in place** (versions, dates, KPI values, table rows, verdict prose). Do **not** redesign the layout. **Before writing any KQL, read the three reference files:** - [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md) — how to run queries (auth, `run-kql.ps1` invocation, output JSON shape, token substitution, the end-to-end loop, version-resolution recipes, hard gotchas). - [`assets/queries/README.md`](assets/queries/README.md) — the query catalog: what each `.kql` computes, the token convention, and the Authenticator scenario→MV→column map. - [`assets/docs/investigation-patterns.md`](assets/docs/investigation-patterns.md) — **the diagnostic methodology**: the count-vs-rate / version-attribution / code-frozen-control / rollout-cohort / benign-vs-real / dimensional-decomposition / drill-to-sub-code / release-tag-diff / MV-introspection / crash-version-attribution (P10) patterns, plus the decision flow. A KPI delta — or a crash that looks new — is a question, not a verdict; run these before any table row becomes "regression" prose or a HOLD. **For the Authenticator crash/stability layer, also read:** - [`assets/docs/crash-sources.md`](assets/docs/crash-sources.md) — App Center crash pull (auth/token, `errorGroups` endpoint, the `nextLink` quirk), the **share-vs-rate trap**, the Kusto rate denominator, secret handling, and the deferred Play Console (Phase 2). Authenticator only — the Broker is not a store app and has no crash section. ## Inputs to confirm Ask only for what's missing; infer the rest. 1. **Which app(s)** — Broker version rolling out, Authenticator version rolling out, or both. At least one is required. 2. **Baseline per app** — the previous release version to compare against, OR "auto-resolve the previous version" (pick the next-highest-volume recent version), OR "all-versions baseline". 3. **Window** — default last **30 days** (Authenticator MVs) / **14 days** (Broker). The new version is usually young, so a longer window mostly grows the baseline cohort. 4. **Report date** — defaults to today (used in the filename + the "Generated" banner). 5. **Authenticator crash layer (optional)** — if an Authenticator version is given, add the crash/stability section. It needs an App Center read-only token (`~/.android-release-reports/appcenter.token`, `$APPCENTER_API_TOKEN`, or `--token-file`). **Step 3b step 0 auto-runs [`preflight-appcenter.ps1`](assets/scripts/preflight-appcenter.ps1) `-Check`** and, if the token is missing/expired, posts an **unmissable action block** for the engineer to run `-Setup` once, then **blocks on an auto-poll** and won't finalize the report until a token validates or the engineer explicitly skips (never a silent skip). The token is a **secret** — never echo it or write it into the report. If the user gives versions but not the baseline, run the resolution queries first and propose `<FIRST>` (rolling out) / `<SECOND>` (previous) from volume, then continue. ## Data sources (summary — full detail in the cheatsheet) | App | Cluster | Database | Version dim | Time col | |-----|---------|----------|-------------|----------| | Broker | `https://idsharedeus2.kusto.windows.net` | `ad-accounts-android-otel` | `broker_version` (`16.1.0`) | `EventInfo_Time` | | Authenticator | `https://idsharedeus2.eastus2.kusto.windows.net` | `d496be22d62a46b0a3cf67ea2e736fd8` | `AppVersion` (`6.2606.3817`) | `EventDate` (MVs) | `run-kql.ps1` defaults to Broker; pass `-Cluster`/`-Database` for Authenticator. Requires `az login` (Android Auth Client SDK security group). ## Bundled assets | File | Purpose | |---|---| | [`templates/report-template.html`](assets/templates/report-template.html) | Canonical layout — a real filled example. **Edit in place**; do not restyle. The CSS in `<head>` is canonical. | | [`docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md) | Operational runbook: run-kql usage, JSON shape, token substitution, end-to-end loop, gotchas. | | [`docs/investigation-patterns.md`](assets/docs/investigation-patterns.md) | **Diagnostic methodology** — count-vs-rate, version-attribution-vs-substitution, code-frozen control, rollout-cohort, benign-vs-real classification, dimensional decomposition, drill-to-sub-code, release-tag diff, MV introspection, **crash version-attribution (P10 — is a crash new/release-caused?)**, plus the Authenticator drill ladder (outcomes MV → `*_Errors_MV_V1` → raw `passkeyoperations`) and a decision flow. Apply before calling any delta a regression. | | [`docs/crash-sources.md`](assets/docs/crash-sources.md) | Authenticator crash layer: App Center auth/endpoint, the three gotchas (`errorGroupId`-is-version-scoped, share-vs-rate, **`firstOccurrence`-is-rollout-date**), the **`newcrashes`/`signature` new-crash detection** flow, Kusto rate denominator, secret handling, Play Console Phase 2. | | [`queries/README.md`](assets/queries/README.md) | Query catalog + Authenticator scenario→MV→column map. | | [`queries/*.kql`](assets/queries/) | 6 Broker + 6 Authenticator scenario templates + `authenticator-crash-denominator.kql` (crash-rate denominator), all validated live. Substitute `<TOKENS>` before running. Includes `broker-errors-by-host-app-span.kql` — the per-`span_name` request-rate drill-down that complements the host-app device-share movers. | | [`scripts/run-kql.ps1`](assets/scripts/run-kql.ps1) | Direct-REST Kusto helper. `-Query`/`-Out` mandatory; `-Cluster`/`-Database` for Authenticator. | | [`scripts/find-suspect-prs.ps1`](assets/scripts/find-suspect-prs.ps1) | Release PR correlation. **Auth-code** (eSTS) attribution: defaults to `broker/`+`common/` over a broker tag range (`-Range v16.1.0..v16.2.0`; broker uses its own tags, common maps via the broker submodule pointer). **Crash** attribution: `-Repos authenticator` over the app tag range (`-Range 6.2606.3817..6.2606.4029`) — resolves the app's own tags and parses ADO `Merged PR NNNNNNNN:` → pullrequest URLs. Three search streams: `-S` pickaxe + `-DiffGrep` (`git log -G` over diff text) + `--grep` (subject). **For a crash, set `-Symbol` to the exception/API token from the stack (e.g. `EntryPoints.get`), not the crashing class, and always pass `-DiffGrep`** — the crashing class is the victim, the culprit is a caller whose subject rarely names the subsystem. Prints PR ids + URLs for attribution cards. | | [`scripts/preflight-appcenter.ps1`](assets/scripts/preflight-appcenter.ps1) | App Center token gate for the crash layer. **`-Check`** (non-interactive, the skill auto-runs it) resolves the token (`--token-file` → `$APPCENTER_API_TOKEN` → `~/.android-release-reports/appcenter.token`) and validates it with one `GET /apps/{owner}/{app}`, emitting `STATUS: ok\|missing\|invalid\|no-access\|network` + exit `0/2/3/4/5` (+ `-Json`). Add **`-Wait <sec>` `-IntervalSec <sec>`** to block-poll until a valid token appears (auto-resume, no handshake). **`-Setup`** (interactive — the engineer runs it once in a real terminal) opens the token page, reads the token via `Read-Host -AsSecureString` (never echoed), validates, then saves with a user-only ACL. Never prints the token. | | [`scripts/fetch-appcenter-crashes.js`](assets/scripts/fetch-appcenter-crashes.js) | Pull Authenticator crash clusters from App Center → run-kql array-form JSON. `groups` (one version) + `diff` (two versions, signature-joined, per-1k rate when given Kusto denominators) + `enrich` (top signatures' daily **trend** + instance-sampled **OS-major/device-model** concentration) + **`newcrashes`** (genuinely-new java-frame signatures via anti-join against a **union of priors**, native/hex frames split out as `new-native?`) + **`signature`** (cross-version presence of one signature + trend — "is crash X version-specific?"). Captures `exceptionMessage`/`appCodeFrame`/`firstOccurrence`, drops `hidden`/`Ignored` groups, and `--page-cap 0` exhausts paging for an accurate total. | | [`scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) | Copy the template to a version-named file, create `_data/<slug>-<date>/`, stamp the Generated date, prune old `_data`, detect unfilled-stub vs real-report collisions. | | [`scripts/compare-versions.js`](assets/scripts/compare-versions.js) | Delta + classification engine over run-kql JSON. `rows` mode (version-per-row metrics) and `movers` mode (paired error-share rows). Thresholds + volume guard. | | [`scripts/validate-report.ps1`](assets/scripts/validate-report.ps1) | Pre-publish validator: stale tokens, mojibake, raw-count leaks, version-string presence, both app sections, verdict callouts. | ## Workflow ### Step 1 — Bootstrap ```powershell $S = ".github/skills/release-monitoring-report/assets/scripts" $out = & "$S\bootstrap-report.ps1" -BrokerVersion 16.1.0 -AuthVersion 6.2606.3817 ``` Omit either `-…Version` for a single-app run. The script prints the report path and creates the `_data/<slug>-<date>/` folder (`$DATA`) for raw query payloads. Re-running on an unfilled stub is silent; a populated report halts unless you pass `-Force`. ### Step 2 — Resolve versions If the baseline wasn't given, run `broker-adoption.kql` and the cheap Authenticator resolver (cheatsheet § "Version resolution recipes") to pick `<FIRST>`/`<SECOND>` by volume. For an "all-versions baseline", drop the version filter / list every version in `<VERSIONS>`. ### Step 3 — Pull queries (both apps) For each needed `.kql`: read it, substitute tokens (cheatsheet § "Filling tokens"), run via `run-kql.ps1 -Out $DATA\<name>.json`. Run independent queries in parallel (PowerShell jobs). Minimum useful set: - **Broker:** `broker-adoption`, `broker-reliability-by-version`, `broker-error-rate-by-version`, `broker-top-errors-by-version` (the "why"), `broker-latency-by-version`. - **Broker via Authenticator (when an Authenticator version is given):** also run `broker-by-host-app` and `broker-top-errors-by-host-app` with `<PACKAGE>=com.azure.authenticator` and `<FIRST>`/`<SECOND>` = the **Authenticator app versions** (these MVs key the host's broker on `AppInfo_Version`, which for that package is the app version). This isolates whether the broker regresses *because of this app rollout*, separate from the fleet-wide `broker_version` view. - **Authenticator:** `auth-version-resolve` (or cheap fallback), then per top scenario: `auth-scenario-success-rate` (Registration/Auth) or `auth-pn-checkforauth-completion` (PN+CFA), always alongside `auth-scenario-initiates` for the volume guard. Use `auth-stats` for adoption. ### Step 3b — Authenticator crash layer (optional) If an Authenticator version is given, **read [`assets/docs/crash-sources.md`](assets/docs/crash-sources.md) first**, then gate on the token: 0. **Preflight the App Center token (do this first — don't silently skip).** Auto-run: ```powershell pwsh -NoProfile -File "$S\preflight-appcenter.ps1" -Check -Json ``` Branch on the `STATUS:` line / exit code: - **`ok`** (exit 0) → proceed to step 1 (fully automatic, no user action). - **`missing`** (2) / **`invalid`** (3) → the token is absent or expired. **Don't write the placeholder yet, and DO NOT just tell the engineer and move on** — set up a seamless capture + auto-resume. Split this across the run so the engineer's paste window overlaps the Kusto work: **prompt EARLY (step a — the moment `-Check` first returns not-ok, even before the main queries), then block LATE (step b — the foreground poll, at the crash layer, which is the report's last section).** Both are hard requirements; skipping either is a bug: **a. Surface an UNMISSABLE action block.** Post the setup ask as its own dedicated, visually prominent message — a `⚠️ ACTION REQUIRED` header, one line stating *"the report's crash/stability section is blocked until you do this,"* and the exact command. Do **not** bury it mid-paragraph, trail it with "meanwhile I'll continue," or imply the report will finish without it — that both makes engineers miss it and tempts you to drop the gate in step b. Open the best capture surface for the host and launch `-Setup` there: - **Desktop app** (Terminal canvas available) → open a **Terminal canvas** and run `pwsh -File "<ABSOLUTE>\preflight-appcenter.ps1" -Setup` in it, so the engineer pastes without leaving the app. - **VS Code Copilot Chat** → tell the engineer to run that same command in the **integrated terminal** (they already have one open). - **Copilot CLI / anything else** → surface the command for the engineer's **own terminal**. In every case relay the **ABSOLUTE** script path that `preflight -Check` just printed (its message uses `$PSCommandPath`). **Never a repo-relative path** like `.github\…\preflight-appcenter.ps1` — the engineer's terminal is usually at their home directory, where a relative path fails with *"not recognized as the name of a script file"*. **b. Run the blocking poll in the FOREGROUND as a single step and wait on its result — do NOT background it and "check later."** This is the enforcement mechanism, and the most important rule in this branch. **Observed failure:** when the poll is launched detached (or merely described as running "in parallel"), the agent posts the message, moves on, and the poll *never actually runs* — so pasting the token resumes nothing and the crash layer silently never happens. The crash layer is