release-monitoring-report

name

release-monitoring-report

description

Generate a version-over-version release-health report for the Android Broker and/or the Authenticator app as one polished self-contained HTML file. Use this skill when monitoring a rollout — triggers include "monitor this release", "release health report", "broker rollout health", "authenticator rollout", "is this release safe to widen", "what's changing this release and why", "crash regression", "stability report", or "compare a new version vs a previous version". Accepts a Broker version and/or an Authenticator version being rolled out, plus a previous/baseline version per app (or an all-versions baseline), runs Kusto queries for each app to quantify what changed and why, adds an Authenticator crash/stability layer from App Center (crashes per 1k active devices), and writes the HTML to the android-release-reports folder under the user's home directory (outside the workspace so reports are never committed).

Release Monitoring Report

Produce a version-over-version release-health report covering the Android Broker and/or the Authenticator app in one self-contained HTML file at $env:USERPROFILE\android-release-reports\release-report-broker-<bv>-auth-<av>-<YYYY-MM-DD>.html (home folder, outside the workspace — reports never get committed). Omitted apps drop out of the filename, so broker-only or authenticator-only runs are first-class.

The report answers two questions per app: what changed this release (KPIs + tables vs the baseline) and why (error-code movers / per-scenario deltas). It ends in a clear verdict per app: SAFE to proceed / WATCH / HOLD.

The output mirrors the canonical template at assets/templates/report-template.html — a real filled example kept as the structural + visual reference. The Step 1 bootstrap script copies it into ~/android-release-reports/… and you edit it in place (versions, dates, KPI values, table rows, verdict prose). Do not redesign the layout.

Before writing any KQL, read the three reference files:

• assets/docs/kusto-cheatsheet.md — how to run queries (auth, run-kql.ps1 invocation, output JSON shape, token substitution, the end-to-end loop, version-resolution recipes, hard gotchas).
• assets/queries/README.md — the query catalog: what each .kql computes, the token convention, and the Authenticator scenario→MV→column map.
• assets/docs/investigation-patterns.md — the diagnostic methodology: the count-vs-rate / version-attribution / code-frozen-control / rollout-cohort / benign-vs-real / dimensional-decomposition / drill-to-sub-code / release-tag-diff / MV-introspection / crash-version-attribution (P10) patterns, plus the decision flow. A KPI delta — or a crash that looks new — is a question, not a verdict; run these before any table row becomes "regression" prose or a HOLD.

For the Authenticator crash/stability layer, also read:

• assets/docs/crash-sources.md — App Center crash pull (auth/token, errorGroups endpoint, the nextLink quirk), the share-vs-rate trap, the Kusto rate denominator, secret handling, and the deferred Play Console (Phase 2). Authenticator only — the Broker is not a store app and has no crash section.

Inputs to confirm

Ask only for what's missing; infer the rest.

1. Which app(s) — Broker version rolling out, Authenticator version rolling out, or both. At least one is required.
2. Baseline per app — the previous release version to compare against, OR "auto-resolve the previous version" (pick the next-highest-volume recent version), OR "all-versions baseline".
3. Window — default last 30 days (Authenticator MVs) / 14 days (Broker). The new version is usually young, so a longer window mostly grows the baseline cohort.
4. Report date — defaults to today (used in the filename + the "Generated" banner).
5. Authenticator crash layer (optional) — if an Authenticator version is given, add the crash/stability section. It needs an App Center read-only token (~/.android-release-reports/appcenter.token, $APPCENTER_API_TOKEN, or --token-file). Step 3b step 0 auto-runs preflight-appcenter.ps1 -Check and, if the token is missing/expired, posts an unmissable action block for the engineer to run -Setup once, then blocks on an auto-poll and won't finalize the report until a token validates or the engineer explicitly skips (never a silent skip). The token is a secret — never echo it or write it into the report.

If the user gives versions but not the baseline, run the resolution queries first and propose <FIRST> (rolling out) / <SECOND> (previous) from volume, then continue.

Data sources (summary — full detail in the cheatsheet)

App	Cluster	Database	Version dim	Time col
Broker	https://idsharedeus2.kusto.windows.net	ad-accounts-android-otel	broker_version (16.1.0)	EventInfo_Time
Authenticator	https://idsharedeus2.eastus2.kusto.windows.net	d496be22d62a46b0a3cf67ea2e736fd8	AppVersion (6.2606.3817)	EventDate (MVs)

run-kql.ps1 defaults to Broker; pass -Cluster/-Database for Authenticator. Requires az login (Android Auth Client SDK security group).

Bundled assets

File	Purpose
templates/report-template.html	Canonical layout — a real filled example. Edit in place; do not restyle. The CSS in <head> is canonical.
docs/kusto-cheatsheet.md	Operational runbook: run-kql usage, JSON shape, token substitution, end-to-end loop, gotchas.
docs/investigation-patterns.md	Diagnostic methodology — count-vs-rate, version-attribution-vs-substitution, code-frozen control, rollout-cohort, benign-vs-real classification, dimensional decomposition, drill-to-sub-code, release-tag diff, MV introspection, crash version-attribution (P10 — is a crash new/release-caused?), plus the Authenticator drill ladder (outcomes MV → *_Errors_MV_V1 → raw passkeyoperations) and a decision flow. Apply before calling any delta a regression.
docs/crash-sources.md	Authenticator crash layer: App Center auth/endpoint, the three gotchas (errorGroupId-is-version-scoped, share-vs-rate, firstOccurrence-is-rollout-date), the newcrashes/signature new-crash detection flow, Kusto rate denominator, secret handling, Play Console Phase 2.
queries/README.md	Query catalog + Authenticator scenario→MV→column map.
queries/*.kql	6 Broker + 6 Authenticator scenario templates + authenticator-crash-denominator.kql (crash-rate denominator), all validated live. Substitute <TOKENS> before running. Includes broker-errors-by-host-app-span.kql — the per-span_name request-rate drill-down that complements the host-app device-share movers.
scripts/run-kql.ps1	Direct-REST Kusto helper. -Query/-Out mandatory; -Cluster/-Database for Authenticator.
scripts/find-suspect-prs.ps1	Release PR correlation. Auth-code (eSTS) attribution: defaults to broker/+common/ over a broker tag range (-Range v16.1.0..v16.2.0; broker uses its own tags, common maps via the broker submodule pointer). Crash attribution: -Repos authenticator over the app tag range (-Range 6.2606.3817..6.2606.4029) — resolves the app's own tags and parses ADO Merged PR NNNNNNNN: → pullrequest URLs. Three search streams: -S pickaxe + -DiffGrep (git log -G over diff text) + --grep (subject). For a crash, set -Symbol to the exception/API token from the stack (e.g. EntryPoints.get), not the crashing class, and always pass -DiffGrep — the crashing class is the victim, the culprit is a caller whose subject rarely names the subsystem. Prints PR ids + URLs for attribution cards.
scripts/preflight-appcenter.ps1	App Center token gate for the crash layer. -Check (non-interactive, the skill auto-runs it) resolves the token (--token-file → $APPCENTER_API_TOKEN → ~/.android-release-reports/appcenter.token) and validates it with one GET /apps/{owner}/{app}, emitting STATUS: ok|missing|invalid|no-access|network + exit 0/2/3/4/5 (+ -Json). Add -Wait <sec> -IntervalSec <sec> to block-poll until a valid token appears (auto-resume, no handshake). -Setup (interactive — the engineer runs it once in a real terminal) opens the token page, reads the token via Read-Host -AsSecureString (never echoed), validates, then saves with a user-only ACL. Never prints the token.
scripts/fetch-appcenter-crashes.js	Pull Authenticator crash clusters from App Center → run-kql array-form JSON. groups (one version) + diff (two versions, signature-joined, per-1k rate when given Kusto denominators) + enrich (top signatures' daily trend + instance-sampled OS-major/device-model concentration) + newcrashes (genuinely-new java-frame signatures via anti-join against a union of priors, native/hex frames split out as new-native?) + signature (cross-version presence of one signature + trend — "is crash X version-specific?"). Captures exceptionMessage/appCodeFrame/firstOccurrence, drops hidden/Ignored groups, and --page-cap 0 exhausts paging for an accurate total.
scripts/bootstrap-report.ps1	Copy the template to a version-named file, create _data/<slug>-<date>/, stamp the Generated date, prune old _data, detect unfilled-stub vs real-report collisions.
scripts/compare-versions.js	Delta + classification engine over run-kql JSON. rows mode (version-per-row metrics) and movers mode (paired error-share rows). Thresholds + volume guard.
scripts/validate-report.ps1	Pre-publish validator: stale tokens, mojibake, raw-count leaks, version-string presence, both app sections, verdict callouts.

Workflow

Step 1 — Bootstrap

$S = ".github/skills/release-monitoring-report/assets/scripts"
$out = & "$S\bootstrap-report.ps1" -BrokerVersion 16.1.0 -AuthVersion 6.2606.3817

Omit either -…Version for a single-app run. The script prints the report path and creates the _data/<slug>-<date>/ folder ($DATA) for raw query payloads. Re-running on an unfilled stub is silent; a populated report halts unless you pass -Force.

Step 2 — Resolve versions

If the baseline wasn't given, run broker-adoption.kql and the cheap Authenticator resolver (cheatsheet § "Version resolution recipes") to pick <FIRST>/<SECOND> by volume. For an "all-versions baseline", drop the version filter / list every version in <VERSIONS>.

Step 3 — Pull queries (both apps)

For each needed .kql: read it, substitute tokens (cheatsheet § "Filling tokens"), run via run-kql.ps1 -Out $DATA\<name>.json. Run independent queries in parallel (PowerShell jobs). Minimum useful set:

• Broker: broker-adoption, broker-reliability-by-version, broker-error-rate-by-version, broker-top-errors-by-version (the "why"), broker-latency-by-version.
• Broker via Authenticator (when an Authenticator version is given): also run broker-by-host-app and broker-top-errors-by-host-app with <PACKAGE>=com.azure.authenticator and <FIRST>/<SECOND> = the Authenticator app versions (these MVs key the host's broker on AppInfo_Version, which for that package is the app version). This isolates whether the broker regresses because of this app rollout, separate from the fleet-wide broker_version view.
• Authenticator: auth-version-resolve (or cheap fallback), then per top scenario: auth-scenario-success-rate (Registration/Auth) or auth-pn-checkforauth-completion (PN+CFA), always alongside auth-scenario-initiates for the volume guard. Use auth-stats for adoption.

Step 3b — Authenticator crash layer (optional)

If an Authenticator version is given, read assets/docs/crash-sources.md first, then gate on the token:

0. Preflight the App Center token (do this first — don't silently skip). Auto-run:

pwsh -NoProfile -File "$S\preflight-appcenter.ps1" -Check -Json

Branch on the STATUS: line / exit code:

• ok (exit 0) → proceed to step 1 (fully automatic, no user action).

• missing (2) / invalid (3) → the token is absent or expired. Don't write the placeholder yet, and DO NOT just tell the engineer and move on — set up a seamless capture + auto-resume. Split this across the run so the engineer's paste window overlaps the Kusto work: prompt EARLY (step a — the moment -Check first returns not-ok, even before the main queries), then block LATE (step b — the foreground poll, at the crash layer, which is the report's last section). Both are hard requirements; skipping either is a bug:

  a. Surface an UNMISSABLE action block. Post the setup ask as its own dedicated, visually prominent message — a ⚠️ ACTION REQUIRED header, one line stating "the report's crash/stability section is blocked until you do this," and the exact command. Do not bury it mid-paragraph, trail it with "meanwhile I'll continue," or imply the report will finish without it — that both makes engineers miss it and tempts you to drop the gate in step b. Open the best capture surface for the host and launch -Setup there:

  • Desktop app (Terminal canvas available) → open a Terminal canvas and run pwsh -File "<ABSOLUTE>\preflight-appcenter.ps1" -Setup in it, so the engineer pastes without leaving the app.
  • VS Code Copilot Chat → tell the engineer to run that same command in the integrated terminal (they already have one open).
  • Copilot CLI / anything else → surface the command for the engineer's own terminal.

  In every case relay the ABSOLUTE script path that preflight -Check just printed (its message uses $PSCommandPath). Never a repo-relative path like .github\…\preflight-appcenter.ps1 — the engineer's terminal is usually at their home directory, where a relative path fails with "not recognized as the name of a script file".

  b. Run the blocking poll in the FOREGROUND as a single step and wait on its result — do NOT background it and "check later." This is the enforcement mechanism, and the most important rule in this branch. Observed failure: when the poll is launched detached (or merely described as running "in parallel"), the agent posts the message, moves on, and the poll never actually runs — so pasting the token resumes nothing and the crash layer silently never happens. The crash layer is