| name | e2e-coverage |
| description | Use when running a broad manual/AI-driven end-to-end verification of Breeze RMM across many merged PRs or commits — "test everything since the last release", release-readiness sweeps, or resuming a partial sweep. Tracks coverage across runs so each sweep only tests the delta. NOT for single-feature post-implementation checks (use feature-testing for that). |
E2E Coverage Sweep
Overview
A coverage sweep verifies everything that changed since a baseline (a release tag or the last sweep), live, across UI/API/agent layers — and records what passed, failed, or was deferred so the next sweep is cheaper.
Core principle: every sweep starts from the last sweep. You test the delta (new commits + previously-deferred items), reuse a known-good environment recipe, and write results back to a durable ledger. Efficiency compounds: the env setup is recorded once, and covered ground is never re-tested.
For single-feature post-implementation verification, use the feature-testing skill instead — this skill is for broad multi-PR passes and reuses feature-testing's per-layer mechanics.
The trap that costs the most time
The running containers may NOT be running the code you think they are. A checked-out branch does not mean the live stack serves it — the containers can be an old image, a detached build, or a stale dist/. Testing against the wrong build produces false PASSes and phantom 404s.
Always verify the live stack before testing anything. See runbook.md → "Verify the stack is running the code under test". This is step 0 of every sweep.
The efficiency loop
digraph sweep {
rankdir=TB;
baseline [label="1. Baseline = max(last release tag, last swept commit)", shape=box];
delta [label="2. Delta = git log baseline..HEAD + carry-forward unresolved items", shape=box];
env [label="3. Verify/bring up env (runbook.md) — confirm it serves HEAD", shape=box];
classify [label="4. Classify each item: P0/P1/P2 x API/UI/AGENT/SKIP, creds-free vs creds-gated", shape=box];
test [label="5. Test the delta (reuse runbook recipes); record as you go", shape=box];
record [label="6. Write plan + results files; update the index pointer", shape=box];
baseline -> delta -> env -> classify -> test -> record;
}
- Baseline.
git tag --sort=-creatordate | head → last released tag. Find the most recent docs/testing/*-test-results.md and its "last swept commit". Baseline = whichever is newer. If no prior sweep, baseline = the last release tag.
- Delta + carry-forward.
git log <baseline>..HEAD --oneline. Open the previous results file and carry forward every item still marked DEFERRED / NEEDS-CREDS / open FAIL — recheck whether creds/agents are now available. Items marked PASS below the baseline are not re-tested.
- Verify/bring up env. Step 0 above — confirm the live stack serves HEAD, then bring up only what's missing. All breeze-specific recipes are in
runbook.md. Don't rediscover them.
- Classify each delta commit/PR using the legend below. Map each to the layer(s) that can prove it, and mark whether it needs external credentials or a live agent (defer those explicitly — never silently drop them).
- Test the delta. Reuse
runbook.md recipes (login/JWT, RBAC negative tests, device lookups) and feature-testing's per-layer mechanics. Record each result immediately.
- Record. Write/update the two ledger files and the index pointer (below).
The coverage ledger
Two files per sweep in docs/testing/, plus one living index. Keep the existing format (see docs/testing/v0.66.1-to-HEAD-*.md as the reference shape).
| File | Purpose |
|---|
docs/testing/<baseline>-to-HEAD-test-plan.md | The classified checklist of delta items (one per PR/commit), P0/P1/P2 + layer tags |
docs/testing/<baseline>-to-HEAD-test-results.md | Results: per-item status + evidence, findings, bugs filed. Records the "last swept commit". |
docs/testing/e2e-coverage-index.md | One line per sweep: baseline → results file → last-swept commit. The pointer the next sweep reads first. |
Status legend (use these exact tokens so the next sweep can grep them):
PASS — verified live, with evidence.FAIL — broken; file/track a bug, link it. Stays carried-forward until fixed+reverified.DEFERRED — testable but not exercised this run (time/data); say why.NEEDS-CREDS — blocked on external credentials or a live agent (Google/M365 tenant, Win/macOS agent). Carried forward.SKIP — covered by unit/CI; no manual E2E warranted (dep bumps, internal refactors). Note it; don't silently omit.
Priority legend: P0 = security / auth / multi-tenant isolation / data corruption. P1 = user-visible new feature or behavior change. P2 = secondary surface or pre-existing area touched.
Quick reference
| Need | Command / pointer |
|---|
| Last release tag | git tag --sort=-creatordate | head |
| Delta since baseline | git log <baseline>..HEAD --oneline |
| Is the live stack serving HEAD? | runbook.md → stale-stack verification |
| Bring up local env (caddy + dev mode) | runbook.md → bring-up |
| Local login → JWT | runbook.md → JWT helper (admin@breeze.local / BreezeAdmin123!) |
| RBAC negative tests | runbook.md → site-scoped e2e-sitea user |
| Seed topology (orgs/sites/devices) | runbook.md → seed topology |
| curl fails inside a loop? | runbook.md → sandbox gotcha (use /usr/bin/curl, no loops) |
Common mistakes
- Testing against a stale stack. The #1 false-PASS source. Verify the live build serves HEAD before anything (runbook step 0).
- Re-testing covered ground. Read the previous results file first; only test the delta + carried-forward items.
- Silently dropping creds-gated items. Mark
NEEDS-CREDS and carry forward — don't let them vanish from coverage.
- Using the prod
E2E_* env creds locally. Those target 2breeze.app (often down). Local DB uses admin@breeze.local / BreezeAdmin123! — see runbook.
- Trusting a single-org admin to catch tenant bugs. The seed partner admin is multi-org on purpose; org-resolution bugs only surface with multi-org/multi-site users (runbook → seed topology).
- Forgetting to write results back. The ledger is the efficiency engine. No results file = next sweep redoes your work.
