| name | qa-monitoring |
| description | [QA Method] Online bug monitoring from Application Insights: query both layers, dedup by fingerprint, triage new/spiking signatures, reproduce HIGH-confidence bugs live, report. Detect-and-report only — never files JIRA or auto-fixes. |
| argument-hint | [frontend|backend|both] [--since=MIN] [--dry-run] |
/qa-monitoring — Online Bug Monitoring (methodology)
Methodology + reference for the /qa-monitoring command and its headless twin
ci/run-monitor.ts. The command file (commands/qa-monitoring.md) is the
terminal entry; this skill holds the data flow, the KQL probe library, the triage
taxonomy, and the dedup model. Detect-and-report only: the flow stops at a
report + Teams alert and a human decides whether to /qa-bug or /qa-fix.
Why this exists
Real production-like defects in the QA environment (storefront JS exceptions, failed
xAPI/REST calls, backend 5xx, failed dependencies) sit in Application Insights unread
until someone happens to query the portal. This flow polls App Insights on a cadence,
keeps a memory of what it has already seen, and surfaces only genuinely new or
surging problems — confirmed live before anyone is asked to act.
Resolve the environment FIRST (no hardcoded resources)
Like /qa-investigate, determine TEST_ENV and resolve the active App Insights
resources from the environment — never assume a fixed resource name:
APPINSIGHTS_APP_ID_BACKEND / APPINSIGHTS_API_KEY_BACKEND — platform/backend.APPINSIGHTS_APP_ID_STOREFRONT / APPINSIGHTS_API_KEY_STOREFRONT — storefront.APPINSIGHTS_RESOURCE_BACKEND / _STOREFRONT — display names, only for portal links.AZURE_SUBSCRIPTION_ID / AZURE_RESOURCE_GROUP — for portal links.
A layer with no App ID/key is skipped with a clear message; some envs have none.
Data flow (one run)
- Query — run the probe queries (below) against each enabled layer over the
look-back window. The window is applied by the REST API
timespan param.
- Fingerprint + dedup — group rows into signatures, classify NEW / SPIKING /
SEEN-stable against
reports/monitoring/.seen-fingerprints.json. Only NEW/SPIKING
proceed; cap at MONITOR_MAX_SIGNALS and log deferrals.
- Triage — classify each candidate (taxonomy below) with severity, repo route,
confidence — consulting the oracle knowledge files.
- Repro gate — only HIGH-confidence
REAL_BUG with a code-fixable REPRO_LAYER
is reproduced live by the layer's QA expert. Reproduced → draft bug report with a
## Fix Routing block; not reproduced → NEEDS_REVIEW (listed, not drafted).
- Report + notify + STOP — write the monitoring report + summary, persist the
store, send the Teams card. No JIRA, no fix.
KQL probe library (ci/monitoring/queries/)
| Probe | Layer | Surfaces |
|---|
backend-exceptions.kql | backend | server-side exceptions by problemId |
backend-failed-requests.kql | backend | 5xx requests by route template |
backend-failed-dependencies.kql | backend | failed SQL/Redis/HTTP/payment deps |
frontend-exceptions.kql | frontend | browser JS exceptions by problemId |
frontend-browser-failures.kql | frontend | failed browser AJAX (path normalized) |
Each query summarizes by a stable signature with count_, first/last seen, and a
sample operation_Id for a portal deep-link. Signatures use route/path templates,
not concrete URLs, so they don't drift with test data (feedback_env_resilience).
Triage taxonomy (ci/agents/monitor-triage-agent.md)
REAL_BUG · KNOWN_ISSUE · NOISE · CONFIG_GATED · THIRD_PARTY · TRANSIENT.
Oracles: vc-bug-catalog.md (familiar failures), business-logic.md (BL-* violations
are always high severity), debugging-signals.md (benign-noise filter),
platform-patterns.md (expected desync/cache behavior). When ambiguous, prefer
NEEDS_REVIEW over REAL_BUG — a log line alone is not a defect.
Dedup model (ci/lib/fingerprint-store.ts)
- Fingerprint = md5(
env|layer|probe|normalized-signature). The store is shared across
envs (vcst, vcptcore, …); the env (= TEST_ENV) keeps each env's signatures separate
inside the one file, so a vcst decision never masks the same signature in vcptcore.
- Per signature: first/last seen, cumulative count, run count, EMA baseline, status
(
new|triaged|confirmed|noise|filed).
- SPIKING =
count ≥ max(baseline×factor, baseline+minDelta) (factor/minDelta env-tunable).
classify() reads the previous baseline; recordRun() updates it — order matters.- The store is local working state (gitignored). In CI it is carried across scheduled
runs via
actions/cache (see .github/workflows/monitor.yml).
Knobs (env)
MONITOR_LAYERS (both|frontend|backend) · MONITOR_SINCE_MIN (35) · MONITOR_MAX_SIGNALS
(15) · MONITOR_SPIKE_FACTOR (3) · MONITOR_SPIKE_MIN_DELTA (20) · MAX_BUDGET_USD (15)
· MODEL · DRY_RUN (triage only — no repro/drafts).
Cross-references
- Command:
commands/qa-monitoring.md · Twin: ci/run-monitor.ts
- Bug handoff contract (
## Fix Routing): commands/qa-bug.md Step 4 → /qa-fix
- Reports policy + the
reports/monitoring/ category: .claude/rules/reports.md
- Bug lifecycle after a confirmed draft:
/qa-bug → /qa-fix → /qa-verify-fix
