Menu
Run app smoke recipes against a local AVD and capture per-step screenshots for the training deck. Per-opp content only.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Produce a domain-expert-facing before/after (pre-post) Google Doc for a DDD narrative iteration, and read the expert's edits back. Pull the verbatim CURRENT narration from canopy-web, apply reviewer feedback into a PROPOSED next version, publish a scannable side-by-side doc a non-engineer can review, then read their suggestion-mode edits via the structured Doc. Use when iterating a Connect DDD narrative on expert feedback (e.g. RF Surveys / Sophie), when asked to "make a before/after of the narration", "iterate the narrative on <expert>'s feedback", or "let <expert> sign off on the language".
Apply the two HQ-layer standing-instruction settings Nova can't set at build time — camera-only photo capture (appearance="acquire" on Deliver image uploads) and grid menu display on every module — to the deployed draft apps, then resolve the matching Phase-3 residuals. Runs between app-deploy and app-release.
Build the CommCare Learn (training) app from the PDD via Nova's /nova:autobuild. Captures nova_app_id and writes a structure summary.
Render a training deck spec.yaml into a Google Slides deck via the 14-stencil ACE template. Produces a presentable Slides URL.
Phase 3 § Step 2.8 — structural + install-time QA on the released Learn + Deliver CCZs. Downloads each CCZ via commcare_download_ccz, parses the zip + suite.xml + form XMLs, verifies form counts + Connect-marker presence match the Nova blueprint, then runs commcare-cli `validate` + `play` as install-time runtime gates. AVD-free, Connect-free — purely CCHQ-side. Halts loud on mismatch.
Generate a training deck spec.yaml from PDD, app summaries, screenshot manifests, and a template bundle. The spec is the source of truth for the training deck — training-deck-render produces Google Slides from it.
| name | app-screenshot-capture |
| description | Run app smoke recipes against a local AVD and capture per-step screenshots for the training deck. Per-opp content only. |
| disable-model-invocation | true |
Run the smoke recipes from app-test-cases.yaml against a local AVD,
capture PNGs at every takeScreenshot step, and ship a thin per-app UX
smoke judge so Phase 6 has a meaningful (but cheap) signal that the
built apps are usable end-to-end. Deep, per-journey UX grading lives in
/ace:qa-deep → app-ux-eval — this skill is intentionally shallow.
| Source | Artifact | Used for |
|---|---|---|
Phase 1 (pdd-to-app-journeys) | ACE/<opp>/runs/<run-id>/2-scenarios/pdd-to-app-journeys.md | persona summary for the UX judge prompt; archetype context |
Phase 3 (app-test-cases) | ACE/<opp>/runs/<run-id>/3-commcare/app-test-cases.yaml | smoke-recipe selection (is_smoke: true) + recipe paths |
| Phase 1 | ACE/<opp>/inputs/pdd.md | persona-summary fallback if not embedded in pdd-to-app-journeys |
| Phase 3 | ACE/<opp>/runs/<run-id>/3-commcare/app-deploy_summary.md | HQ domain for ${HQ_DOMAIN} env var |
| Phase 4 (run_state.yaml) | phases.connect-setup.products.connect.opportunity.{id, name} + top-level run_id + ACE test user invite | ${OPP_RUN_ID} (verbatim from run_state.yaml.run_id, the deterministic tile matcher), ${OPP_NAME} (verbatim from opportunity.name, logging/screenshot context only), ${ACE_E2E_PHONE_LOCAL}, etc. |
Recipes are read by path from the entries in app-test-cases.yaml
(recipe_path field). They were composed and validated by Phase 3's
app-test-cases skill — this skill does NOT compose or validate
recipes itself. If a smoke recipe is missing or malformed, halt and
point at app-test-cases.
6-qa-and-training/screenshots/<recipe-base>/<step-name>.png — per-step PNGs (anyone-with-link permission set at upload for Slides ingest)6-qa-and-training/app-screenshot-capture_manifest.yaml — fileId/alias index consumed by training-flw-guide and training-deck-generate6-qa-and-training/app-screenshot-capture_verdict-shallow.yaml — thin per-app UX smoke verdictPer-opp content only. Common Connect navigation screenshots come from
the standalone connect-baseline-screenshots skill (NOT a Phase 6
dispatch).
Read 2-scenarios/pdd-to-app-journeys.md and
3-commcare/app-test-cases.yaml from Drive. If either is missing or
empty, halt with a structured error pointing at the upstream phase:
2-scenarios/pdd-to-app-journeys.md → Phase 1 (pdd-to-app-journeys)3-commcare/app-test-cases.yaml → Phase 3 (app-test-cases)Do NOT generate recipes or test cases independently — Phase 6 is an executor, not a synthesizer.
Before booting the AVD, verify the upstream Phase 3 outputs are
structurally complete. Phase 3's app-test-cases SKILL contracts
BOTH the master yaml AND a per-journey Maestro recipe (see
skills/app-test-cases/SKILL.md § Outputs). Half-emitted state — the
master yaml present but recipes missing — is the canonical "upstream
incomplete output" failure mode. Catch it here before AVD wall-clock
burns.
Read app-test-cases.yaml. Filter journeys[] to entries with
is_smoke: true (this is the only set Phase 6 walks). Deep
(non-smoke) journeys carry recipe: deferred — Phase 3 does NOT author
their recipe files (they are generated lazily by /ace:qa-deep; see
skills/app-test-cases/SKILL.md § Products + jjackson/ace#605). This
skill never reads or runs deep journeys, so a recipe: deferred entry
is expected and must NOT be treated as a missing-recipe failure. Scope
every check below to the is_smoke: true subset. Then group the smokes
by the app: field. Run these checks in order. Halt at the first
failure with a structured PLATFORM-tag auto_surfaced entry naming the
exact remediation command:
| Failure mode | PLATFORM message | Remediation |
|---|---|---|
Master yaml has zero is_smoke: true journeys | app-test-cases.yaml has no is_smoke:true journeys; upstream Phase 3 (app-test-cases) emitted no smoke set | /ace:step app-test-cases <opp>/<run-id> |
app: learn smoke count != 1 OR app: deliver smoke count != 1 | app-test-cases.yaml smoke set malformed: expected exactly one is_smoke:true journey per app, got learn=N deliver=M | /ace:step app-test-cases <opp>/<run-id> |
3-commcare/recipes/ subfolder does not exist on Drive | app-test-cases.yaml declares is_smoke:true journeys but 3-commcare/recipes/ subfolder is missing — upstream Phase 3 produced incomplete output (master yaml without per-journey recipes) | /ace:step app-test-cases <opp>/<run-id> BEFORE retrying this skill |
One or more smoke journeys' recipe_path doesn't resolve to a real file | recipe_path journey-<app>.yaml referenced by app-test-cases.yaml does not resolve under 3-commcare/recipes/ — upstream Phase 3 produced an incomplete output set | /ace:step app-test-cases <opp>/<run-id> BEFORE retrying |
(A non-smoke journey with recipe: deferred and no recipe file is NOT a
failure — those are generated on demand by /ace:qa-deep. Only the
is_smoke: true journeys are checked here.)
Each of these halts writes the incomplete-mode verdict shape (see
Step 9 below) with verdict: incomplete and the matching PLATFORM
auto_surfaced entry. Do not write verdict: fail for these — fail is
reserved for cases where the recipes ran but a smoke recipe broke.
Upstream gaps are incomplete, not fail.
This pre-flight is not optional, even when journeys + recipes are
on disk. The count check is the load-bearing gate, not the recipe-
presence check. Caught in vivo on malaria-itn-app run 20260517-1829:
Phase 3 emitted smoke_journeys_per_app: {learn: 0, deliver: 1} (a
known-incomplete state) and Phase 6 ran the Deliver recipe anyway —
producing a recipe-vs-app-state mismatch (post-claim handoff landed in
Learn, not Deliver, because Connect gates Deliver behind Learn-
assessment completion; see
docs/learnings/2026-05-18-connect-gates-deliver-on-learn-completion.md).
The agent SHOULD have halted at row 2 of the failure-mode table the
moment it read learn=0. If the agent finds itself rationalizing past
this check ("the recipe is on disk, the AVD is healthy, I'll just run
the Deliver smoke and skip the Learn one") — that's the exact anti-
pattern this table prevents. Halt instead.
The agent-level pre-flight (agents/qa-and-training.md § Pre-flight checklist) catches the same class of gap before the skill is
dispatched. This skill-level check is the second line of defense for
direct /ace:step app-screenshot-capture invocations.
Recipes assume the AVD is at the Connect "New Opportunities" home —
the app is configured, the test user is signed in to PersonalID, the
opp tile is reachable. This is a precondition of running the
smoke recipes, not a state we adapt to. Per the "phase preconditions
are restored, not adapted" pattern (see CLAUDE.md § Phase preconditions), the restore happens in mobile_ensure_avd_running,
unconditionally:
loadSnapshot('registered-test-user')
every Phase 6 dispatch. ~3s, deterministic. If the snapshot doesn't
exist (fresh machine, deleted, etc.), tier-2 auto-bootstrap fires:
install CommCare APK if missing → registerTestUser → saveSnapshot.
~3-5 min on fresh-machine first dispatch; ~3s thereafter via the
freshly-saved snapshot. Phase 4's connect-opp-setup already
invited ${ACE_E2E_PHONE} to the run's opp, so the CONNECT-ID-3F
server-side invite-check precondition is satisfied automatically.
/ace:mobile-bootstrap is no longer required as an operator
pre-step inside /ace:run — useful only for ad-hoc workstation
setup outside the lifecycle./api/mobile/ensure-running cold-boots the
AVD and runs the registration recipes against it (see
backends/cloud.ts header). Same precondition contract, different
mechanism — and no explicit snapshot-load is needed because the AMI
ships the registration recipes built-in.After mobile_ensure_avd_running returns, AvdInfo.heal.deviceUserState
carries the outcome: { classified_as, attempted, healed_via, verified_as, ui_dump_signal }. Local always shows attempted: true, healed_via: 'snapshot-load'. Cloud shows attempted: false (cold-boot
already handled it).
Recovery escalation. If mobile_ensure_avd_running threw
DeviceUserStateError, two recoverable error codes:
snapshot-load-failed — the snapshot doesn't exist (fresh machine,
snapshot was deleted) OR the emulator console couldn't load it. The
remediation is /ace:mobile-bootstrap — it'll register the test
user, configure the app, and save a fresh snapshot.needs-app-config / needs-personal-id / commcare-not-installed
(as the verify: segment in the error attempts) — the snapshot
loaded but post-restore probe still showed a wiped state. This is
snapshot corruption or post-snapshot APK upgrade drift. Same
remediation: /ace:mobile-bootstrap re-snapshots.This step is now ~zero code in the skill. The state-classifier
lives in mcp/mobile/client.ts (post-restore verification only); no
duplicate probe at the skill level. Earlier versions of this skill
documented a local probe table here — that was the "tolerate any
starting state" anti-pattern. Removed 0.13.202.
The recipe-error → failure-mode table further down (Step 5) is the second-line classifier for things the upstream restore couldn't catch (e.g. CCZ content issues, Maestro recipe drift, OPP_NAME collisions in long invite lists). It stays.
Before booting the AVD, run a static comparison between recipe expectations and live app + Connect state. This catches the failure classes the turmeric/20260515-0536 cycle surfaced one-at-a-time over 8 attempts (~80 min of wall-clock burn) — module-name == form-name, expected-module-not-in-app, opp-name-mismatch, tile-name-collision.
Helper: mcp/mobile/recipe-sanity-probe.ts (pure function — same
inputs always produce the same verdict; no MCP calls inside the
probe itself).
Inputs the caller assembles:
3-commcare/recipes/journey-*.yaml on
Drivenova_get_app({app_id}) per app in
app-test-cases.yaml (typically learn + deliver)connect_get_opportunity({org_slug, opportunity_id}) from run_state.yaml.phases.connect-setup.productsenv.OPP_NAME or from app-test-cases.yamlmobile_capture_ui_dump after a
quick login. Skipping this skips only the tile-name-collision
check; everything else still runs.Failure classes the probe surfaces (each with a canonical remediation):
| Class | Remediation |
|---|---|
module-name-equals-form-name | Verify plugin >= 0.13.255 (handled by learn-tap-module). If older, re-author via /ace:step app-test-cases. |
expected-module-not-in-app | Recipe needs re-author via /ace:step app-test-cases — live app structure has drifted. |
expected-form-not-in-module | Same as above — module/form structure has drifted. |
opp-name-mismatch | Pass OPP_NAME verbatim from run_state.yaml.phases.connect-setup.products.connect.opportunity.name (NOT slug-reassembled). Fallback only if missing: connect_get_opportunity({org_slug, opportunity_id}).name. |
tile-name-collision | Clean up prior-run invites OR use Resume-branch (exact-match claim). |
form-advance-without-answer-tap | Recipe chains ≥2 consecutive form-advance steps with no answer step between them — required-input questions will stall on warning_root. Re-author via /ace:step app-test-cases: for each required field, read its label/options via Nova get_form and emit a tapOn:text:"<literal>" (or inputText / photo-capture sequence) BEFORE the form-advance. |
brief-label-drift | Recipe has a tapOn:text:"X" matcher where X matches a PDD-brief naming pattern (^[LFM]\d+ — , ^Stage \d+ — ). Nova rewrites these during autobuild and the matcher won't resolve live. Re-author via /ace:step app-test-cases: read the live label from Nova get_form/get_module and use it verbatim. |
deliver-smoke-rewalks-learn | Re-author the Deliver smoke as resume-only (connect-resume-opp → deliver-launch.yaml) via /ace:step app-test-cases. The Learn leg already completes Learn. |
On any failure, halt with the incomplete-mode verdict shape (see
Step 9), verdict: incomplete, and a per-class
PLATFORM auto_surfaced entry naming the remediation. Do NOT proceed
to Step 3 — every one of these classes is structurally guaranteed to
produce a recipe-level failure 5-10 min later. Fail fast at the
boundary.
Learn completion is permanent per (test user, opportunity). Once
the test user has completed Learn on an opp, Connect routes "Continue
Learning" to the Deliver download gate, not the Learn home — the
Learn flow cannot be walked again on that opp+user. This is documented
in docs/learnings/2026-05-18-connect-gates-deliver-on-learn-completion.md;
this step makes it operational so the Learn-walk smoke does not halt on a
consumed opp.
Two hard rules:
A recipe-defect diagnosis must NOT mutate one-way live state. If the Learn smoke recipe is broken (e.g. a missing FINISH press), fix the recipe — do NOT manually complete Learn on the live opp to "confirm the fix." Completing Learn consumes the Learn-not-complete precondition permanently, and the only restore is a fresh opportunity (run independence). The bednet-spot-check/20260529-1124 cycle burned two Phase-6 attempts + a failed opp re-mint exactly this way.
An already-Learn-complete opp is NOT a re-walkable state and NOT a blocker to "recover" by re-walking. The Learn leg (Step 5) must branch on it:
connect-claim-opp.yaml surfaces
this via its already-Learn-complete branch, #570), record the Learn
sub-verdict as satisfied-by-prior-completion (NOT fail, NOT
incomplete — Learn genuinely completed, just on a prior pass) and
proceed directly to the Deliver leg, which is unlocked.nsv_home_screen) — i.e. Learn is genuinely
not yet complete.If a fresh Learn-walk screenshot set is specifically required (e.g. the
prior completion produced no usable captures), that needs a fresh
opportunity — start a new /ace:run (new opp → fresh
OpportunityAccess). Do NOT re-mint a Phase-4 opp on the same released
Deliver app to get there: Connect shares one DeliverUnit across opps
on the same cc_app_id, so the fresh opp can't get a payment unit
(#573) — a fresh run (new apps → new cc_app_id) is the clean path.
This branch is the structural fix; it does not require a live pre-AVD probe (Connect exposes no clean per-user Learn-completion atom today) — the claim-prefix landing screen IS the signal.
Boot the AVD via mobile_ensure_avd_running and install the Connect
APK via mobile_install_apk (no-op if cached).
mobile_ensure_avd_running is the single source of truth for "AVD is
ready for Maestro" — since 0.13.165 it also probes the on-device
Maestro driver's gRPC channel and auto-heals (force-stop + uninstall +
reinstall of dev.mobile.maestro) if the driver isn't responding.
Same code path is what mobile-bootstrap calls and what ace-doctor
probes (read-only via mobile_probe_maestro_driver). One healing
implementation, three callers.
If mobile_ensure_avd_running throws MaestroDriverError (the heal
exhausted), halt with verdict: fail + severity: BLOCKER naming
/ace:mobile-bootstrap as the operator recovery. Do NOT downgrade
to verdict: incomplete — Maestro driver health is solvable on the
workstation, not via a placeholder ship. Pre-0.13.165 the skill wrote
incomplete here and Phase 6 shipped without screenshots; that escape
valve hid real capability gaps behind a yellow verdict.
These set up the AVD to the post-claim state the smoke recipes assume:
connect-login.yaml with ${ACE_E2E_PHONE_LOCAL}, ${ACE_E2E_PIN}.connect-claim-opp.yaml with ${OPP_RUN_ID} (the deterministic
tile matcher) read verbatim from run_state.yaml.run_id, AND
${OPP_NAME} read verbatim from
run_state.yaml.phases.connect-setup.products.connect.opportunity.name
(logging / screenshot context only).OPP_RUN_ID matcher — the deterministic tile finder (#618). Phase 4
(connect-opp-setup) front-prefixes every opp name with the run-id:
"<run_id> · <PDD display name>" (e.g. "20260601-0739 · Malaria RDT Performance Sampling"). The run-id (format YYYYMMDD-HHMM, the
top-level run_state.yaml.run_id) is unique per run and, as a FRONT
prefix, lands on the tile's first line — short and never clipped by the
tile's name-wrap. The claim/resume recipes anchor their tile match on it
as text: ".*${OPP_RUN_ID}.*" (Maestro text: is a full-match regex;
the .* wrappers substring-match the run-id within the longer tile
label). Pass ${OPP_RUN_ID} verbatim from run_state.yaml.run_id.
This is what makes finding this run's opp among the test user's
accumulated invites deterministic.
OPP_NAME — context only (no longer the matcher). Phase 4 also writes
the exact tile name Connect renders into
phases.connect-setup.products.connect.opportunity.name. Pass that
string verbatim as ${OPP_NAME} for logging and screenshot context.
Do NOT reassemble from slug pieces. It is no longer the recipe
matcher — ${OPP_RUN_ID} (above) is.
Fallback (legacy runs only). If opportunity.name is missing from
run_state.yaml (rare — only pre-0.13.x runs predating the name-field
write), fall back to connect_get_opportunity({org_slug, opportunity_id}).name and log [WARN] OPP_NAME read from connect_get_opportunity fallback — current run_state.yaml lacks phases.connect-setup.products.connect.opportunity.name.
Never compose from slug pieces.
This insulates Phase 6 from Connect's name-formatting changes — we
read what Phase 4 wrote when it created the opp, not what the Connect
UI happens to render today. Caught live on malaria-itn-fgd run
20260515-1645 Phase 6 attempt 9, where slug-based composition produced
"ai-demo-space - malaria-itn-fgd (run 20260515-1645)" but the live
tile text was "Malaria ITN FGD — malaria-itn-fgd (run 20260515-1645)";
the subagent self-corrected mid-dispatch by calling
connect_get_opportunity, wasting a recipe cycle.
OPP_NAME uniqueness assumption (durable note) — resolved by the
run-id FRONT prefix (#618). Because every /ace:run Phase 3 invites
the same ACE test user (${ACE_E2E_PHONE}) to a fresh Connect opp, the
test user's in-app invite list grows unboundedly across runs. Every
prior opp the test user was ever invited to is still listed, and there
is no per-run cleanup atom yet (TBD) — the list still grows.
What changed (#618): connect-opp-setup now front-prefixes the opp name
with the run-id — "<run_id> · <display name>" (e.g.
"20260601-0739 · Malaria RDT Performance Sampling") — and the
claim/resume recipes anchor their match on the run-id token via
text: ".*${OPP_RUN_ID}.*". This resolves the two fragilities the old
run-id-SUFFIX form carried:
tapOn:text matcher
depended on Connect ordering by invited_at descending so the right
tile sorted near the top. The recipes now scrollUntilVisible on the
run-id regex and assert it, so the match is positional-order-
independent: it finds this run's tile wherever it sits in the list.Remaining honest caveat. The invite list itself is still unbounded —
there is no per-run cleanup atom, so the test user accumulates dozens of
tiles over time. The run-id match is what makes finding the right one
deterministic despite that accumulation (a short, unique, line-1
token), but the underlying list growth is unchanged. A periodic
test-user invite-list-cleanup atom is still desirable (Connect doesn't
expose this in the public API today; would need a connect_* Playwright
atom).
Palette-composition footguns (jjackson/ace#592) — read before composing recipes:
mobile_run_recipe does NOT resolve ${SELECTOR:...} placeholders
inside runFlow'd palette files. It resolves selectors in the
top-level recipe body it's handed, but a palette piece pulled in via
runFlow: runs verbatim — Maestro receives the literal
${SELECTOR:foo} token and (because ${...} is also Maestro's env-var
syntax) resolves it to NaN (e.g. Assert that "NaN" is visible).
When composing static palette pieces, either (a) run
mobile_resolve_selectors on the composed body first, or (b) inline
literal selectors. Do NOT assume nested runFlow inherits the parent's
selector resolution.runFlow relative paths resolve against Maestro's temp chunk dir,
not the recipe-file dir. runFlow: ../../../../mcp/... fails with
Flow file does not exist: /var/folders/.../mcp/.... Always use
absolute paths when composing palette pieces.COUNTRY_CODE is 7
(the dialing code), PHONE_LOCAL is 4260000101, and the PIN is the
full ${ACE_E2E_PIN} (111111, 6 digits). The +7426 demo prefix in
the docs is CC 7 + local starting 4260000101 — do NOT mis-split it
as CC 7426.screen_login_main with
welcome_msg "Welcome ACE Test!" and a login_button
"LOGIN WITH PERSONALID". connect-login.yaml today only branches on
first-start (str_setup_message) and already-signed-in
(connect_fragment_jobs_list); it halts on this intermediate screen.
The branch (tap login_button → lockPassword → PIN) is documented
but not yet live-validated — calibrate it the next time this
surface is reached on-device, then add the branch + a
static-recipe-invariants assertion. Until then, a re-login drop
between legs is a known cause of a Deliver-leg halt.HARD RULE — screenshots come ONLY from a status: pass execution in
THIS run (jjackson/ace#756). If mobile_run_recipe returns
status != pass for a required journey, that leg's screenshots DO NOT
EXIST: record the leg fail with the recipe failure (failureClass,
stderr, failed step) inline in the verdict, and NEVER source PNGs from
the screenshot dir, a prior run, or any imagined "replay cache" — no
such mechanism exists, and any file not in the returned screenshots[]
of this dispatch is untraceable. The shallow verdict MUST NOT be pass
if any required journey's recipe failed (per the Step 9 table). Halt
loud per CLAUDE.md § Phase preconditions — don't ship placeholders.
(Since #756 mobile_run_recipe also wipes the screenshot dir at
execution start, so stale PNGs are structurally absent — but the rule
stands regardless of what's on disk.)
Capture is split into a Learn leg and a Deliver leg. The legs are graded independently; a Deliver failure never suppresses Learn capture.
Learn leg (always runs first). Run journey-learn.yaml against the
AVD. Upload every captured screenshot to
6-qa-and-training/screenshots/journey-learn/<step-name>.png
(shareAnyoneWithLink: true, mimeType: image/png; upload any sibling
<step-name>.xml ui-dump with mimeType: application/xml). Record the
Learn leg outcome (pass iff the recipe status is pass AND every
screenshot is non-zero bytes). A Learn failure records the Learn
sub-verdict and does NOT abort the dispatch — but the Deliver leg then
cannot run (Connect gates Deliver behind Learn completion), so it is
recorded blocked-by-learn.
Deliver leg (runs second; depends on the Learn leg). Only attempt
if the Learn leg reached completion. journey-deliver.yaml resumes
from the now-unlocked state in the same device session (no re-login).
Upload to 6-qa-and-training/screenshots/journey-deliver/<step-name>.png.
Record the Deliver leg outcome independently.
FIRST, before any Deliver recipe — capture the post-Learn landing
(unconditional, #618 ground truth). The instant the Learn leg completes,
mobile_capture_ui_dump and upload it to
journey-deliver/00-postlearn-landing.xml (drive_upload_binary,
mimeType: "application/xml"). This is captured BEFORE the Deliver leg
runs (and BEFORE any failure remediation) precisely because the device's
post-Learn state is fragile — see the next rule. The dump tells you which
post-Learn pathway you're on: nsv_home_screen → Path A (device on the
CommCare Learn home, inside CommCare — the
#618 landing, now HEALED in
connect-resume-opp.yaml via the #711 nav-drawer return branch);
connect_fragment_jobs_list → Path B (Connect jobs list). Both paths are
handled by connect-resume-opp.yaml — the first recipe in the Deliver leg:
its notVisible: home-jobs-list guard opens the nav drawer → taps
"Opportunities" to return from Path A to the jobs list, then proceeds to the
target tile's CTA; on Path B the guard no-ops and it goes straight to the CTA.
So just run the Deliver leg normally — do NOT special-case Path A or divert to
a calibration probe. The dump above is still captured unconditionally as #618
ground truth (and feeds atlas drift detection).
NEVER cold-boot / re-bootstrap to "recover" a Deliver-leg failure. A
Deliver landing on the CommCare home (nsv_home_screen) is the #618 nav
gap — an expected, characterized condition, NOT a device-infra failure.
mobile_ensure_avd_running (cold-boot) and /ace:mobile-bootstrap
destroy the post-Learn state and reset the device PIN, which makes
recovery impossible and wedges the rest of the run on a system auth prompt
(observed live, run 20260605-2138: a cold-boot after the Deliver handoff
failed reset the PIN → every subsequent recipe failed on com.android.systemui
→ the leg gave up after re-register attempts). On a Deliver-leg failure: do
NOT escalate to a boot/bootstrap. Capture the dump (above), let
connect-resume-opp.yaml run (it heals Path A via the nav-drawer return
branch), and only if the recipe still genuinely fails, record the Deliver leg
incomplete with the auto-captured *-FAILURE.{png,xml} forensics and STOP.
Re-bootstrap is ONLY for a Learn-leg CommCareSetupActivity/PersonalID-wipe
failure (the table row that names it), never for a post-Learn Deliver landing.
Do NOT halt the dispatch on a single leg failure. Run both legs (or record why the Deliver leg couldn't run), then write the per-app verdict in Step 9. The recipe-error → failure-mode table below is the per-leg classifier — apply it to whichever leg failed.
shareAnyoneWithLink: true flag is required.
Slides' createImage (used by training-deck-render downstream)
fetches PNGs via Google's image-import service, which doesn't carry
the SA's auth — so an SA-only file gets "image cannot be reached"
and the deck slide comes out blank. Setting anyone-with-link at
upload time avoids a class of "deck builds without errors but
slides are empty" bugs. The standalone drive_set_anyone_with_link
atom exists for retroactively sharing a file that was uploaded
without the flag. Verified live 2026-05-02 via
scripts/test-screenshot-to-slides-e2e.ts.ScreenshotEntry may now carry a sibling
uiDumpPath pointing at a <step-name>.xml file containing the
Android uiautomator dump output captured at the same moment as the
PNG. When present, upload it to
ACE/<opp>/runs/<run-id>/6-qa-and-training/screenshots/<recipe-base>/<step-name>.xml
via drive_upload_binary with mimeType: "application/xml"
(no shareAnyoneWithLink needed — XMLs aren't consumed by Slides).
The dumps are produced automatically by MaestroBackend.runRecipeWithDumps
whenever the caller passes a serial to mobile_run_recipe (the
default path for local-AVD invocations since 0.13.229). Absence is
fine — pre-0.13.229 recipes have no XMLs and the upload step
becomes a no-op. The dumps unlock atlas auto-maintenance (every
Phase 6 dispatch leaves a complete record of resource-IDs at every
surface the recipe visited) and selector drift detection (diff
current dump vs prior passing run's dump). See
docs/learnings/2026-05-14-atlas-side-channel-capture.md for the
underlying problem and mcp/mobile/recipe-splitter.ts for the
splitting logic.#618 Path-A heal lives in the recipe — the calibration probe is retired.
The Path-A return-nav (CommCare Learn home → Connect jobs list) was captured
live and encoded as the guarded notVisible: home-jobs-list branch in
connect-resume-opp.yaml (#711: open the nav drawer → tap "Opportunities"),
followed by a pre-scroll jobs-list sync (#765: the just-completed opp's card
can be absent from a stale list). So Path A no longer needs a separate
probe-and-mark-incomplete step — running the Deliver leg exercises the heal
directly, and the unconditional 00-postlearn-landing.xml dump above is the
ground-truth capture. The old connect-learn-home-deliver-probe.yaml
calibration recipe was removed (jjackson/ace#618).
If the recipe's Path-A branch ever fails on a fresh run, the auto-captured
connect-resume-opp-FAILURE.{png,xml} forensics (plus
00-postlearn-landing.xml) are the characterization signal — re-calibrate the
nav-drawer route against that live dump rather than re-introducing a probe
(the "close the loop to the source of truth — one live dump beats a guess"
rule). Because Learn completion is one-way per (test user, opp), the recipe's
nav-drawer heal + #765 sync can only be validated end-to-end on a fresh
/ace:run Phase 6 Deliver leg.
Before consulting this table, READ the failure screenshot. Two sources, in priority order:
result.failureForensics (auto-captured, cross-backend — since
0.13.538). On every recipe status: 'fail', mobile_run_recipe
captures the device state at the moment of failure — uiDumpPath
(an <recipe-id>-FAILURE.xml element tree: resource-ids/text/bounds,
the highest-signal artifact for selector + nav debugging) plus
screenshotPath (<recipe-id>-FAILURE.png of the offending screen)
— and returns them on the result. Both land in the run's
screenshotDir, so they get uploaded + provenance-stamped alongside
the smoke PNGs. This works on the cloud backend too (Maestro's own
debug-bundle screenshot is local-only). Read failureForensics
first — the ui-dump usually shows the exact screen + the
resource-ids present, which resolves "wrong selector" vs "wrong
screen" immediately.maestro.log Failed: line as
screenshot-❌-<timestamp>-(<recipe>.yaml).png, under
~/.maestro/tests/<timestamp>/.The image/dump often names the failure mode literally — "Logged out of
PersonalID", "Enter Code", "Failed to start learning", "App not
found" — making package / process / driver probing redundant.
Image-read first, infer second.
Skipping this step produced an inverted-conclusion bug live in
2026-05-13 (turmeric run 20260513-0616): the agent saw org.commcare.dalvik
absent of a connect-named sibling package and concluded "Connect not
installed" — when the failure screenshot showed "Logged out of
PersonalID — Lost PersonalID configuration with server, please recover
your PersonalID account and retry" with a Reconfigure → CTA in the
nav drawer. Reading the screenshot would have produced the right
answer in one step.
Recognized failure modes (write a [BLOCKER] to the gate brief
naming the specific failure + remediation rather than a generic
"smoke recipe failed" message):
| Recipe error contains | Failure mode + root cause | Remediation |
|---|---|---|
Failed to start learning | Released Learn CCZ has Nova <module xmlns="…connect…"> + <assessment xmlns="…connect…"> wrappers that the AVD's CommCare runtime can't launch. Confirmed live 2026-05-07 against leep-paint-collection: turmeric Learn (working) has 0 wrapper refs; LEEP Learn (broken) has 16. Tracking: voidcraft-labs/nova-plugin#7, jjackson/ace#115 finding 1. | As of 0.13.66 Phase 3's Step 2.8 invokes commcare-form-patch automatically — re-run /ace:run <opp> and Phase 6 should pick up the patched Learn release. For an in-flight opp that already shipped Phase 3 without the patch: /ace:step commcare-form-patch <opp> then re-run Phase 6. Diagnostic probe: npx tsx scripts/probe-connect-learn-handoff.ts <opp_uuid> + adb logcat. |
deviceInfo … UNAVAILABLE / MaestroDriverError | Maestro driver app on the AVD is installed but its gRPC server isn't responding. Symptom of a wedged driver process or a stale install whose runtime state diverged from the CLI's expectations. Reproduced live 2026-05-11 against leep run 20260511-0507 Phase 6 (port 7001 refusing connections, every recipe stalling on deviceInfo). | Since 0.13.165 mobile_ensure_avd_running auto-heals (force-stop + uninstall both halves of dev.mobile.maestro, then re-probe to trigger the CLI's auto-reinstall). If this error still surfaces, the heal exhausted — run /ace:mobile-bootstrap to re-baseline the AVD + driver, then /ace:step app-screenshot-capture <opp>/<run-id>. Logcat probe: `adb -s logcat |
extendedWaitUntil timeout on connect_fragment_jobs_list | Claim flow didn't reach jobs list. LLO program-application not ACCEPTED, or Connect session expired on the AVD. | Re-run connect-login.yaml and verify connect_get_opportunity returns the expected opp. |
assertVisible(text: ${OPP_NAME}) failure AND focused activity is CommCareSetupActivity AND/OR failure screenshot shows "Logged out of PersonalID" / "Lost PersonalID configuration" / "Reconfigure" / "Enter Code" / "Welcome to CommCare" | AVD's per-user state was wiped — no ApplicationDocument configured (CCHQ app never pulled OR app db was wiped) and/or PersonalID account de-registered from the device. Important: org.commcare.dalvik IS the Connect-enabled CommCare client (no separate Connect package); presence of org.commcare.dalvik in pm list packages does NOT imply a usable Connect home. Should have been caught by Step 2.5; if it surfaces here, Step 2.5's probe missed a signal worth adding. Live in 2026-05-13 (turmeric run 20260513-0616) with both states stacked: setup activity foregrounded + PersonalID drawer banner. | /ace:mobile-bootstrap — re-installs APK if needed, registers the ACE test user (${ACE_E2E_PHONE}) via the Connect registration API, pulls the CCHQ app for this run's HQ ids, saves a clean snapshot. After bootstrap returns clean: /ace:step app-screenshot-capture <opp>/<run-id>. If this state recurs after a successful bootstrap, the issue is state-loss between sessions (snapshot revert, AVD cold-boot, server-side PersonalID de-registration); file a class-level issue on mobile_ensure_avd_running's state-persistence contract rather than re-bootstrapping. |
assertVisible(text: ${OPP_NAME}) failure AND failure screenshot shows the Connect "New Opportunities" / opp list home (the user IS logged in, app IS configured, just no matching tile) | Right opp card not on screen for the OPP_NAME being matched. Wrong OPP_NAME env var (typically slug-reassembled instead of read verbatim from run_state.yaml), OR opp not yet claimed by the test user, OR ${OPP_NAME} collides ambiguously with another invite further up the test user's accumulated invite list (see Step 4 "OPP_NAME uniqueness assumption"). | Confirm OPP_NAME was read verbatim from run_state.yaml.phases.connect-setup.products.connect.opportunity.name (NOT composed from slug pieces — see Step 4 "OPP_NAME source" and #115 finding 4). If the field is present and still mismatches the tile, scroll the invite list or use the connect_get_opportunity fallback. |
assertVisible(nsv_home_screen) failure AND a claim-START-HANDOFF-WEDGED-issue629 screenshot is present (Start was tapped on the opp-detail Job Card; neither the Learn home nor the Deliver download gate ever rendered within 180s) | Inert btn_start handoff — Connect's POST /users/start_learn_app/ never fired or 500-ed, so the Learn CCZ download never began and nsv_home_screen never appeared. This is the Connect-platform half of #629, downstream of ACE — the released CCZ already passed Phase 3's commcare-cli play install gate, so it is NOT a Nova/recipe/marker defect. connect-claim-opp.yaml now captures the labeled artifact + fails loud at this exact point instead of a generic 180s timeout. | Run connect_preflight_learn_app_user({org_slug, opportunity_id}) to classify: an auth/domain/conflict failure there confirms the server-side start_learn_app block (fix Connect-side / re-invite the test user); a clean preflight points at a device-side wedge (re-run after mobile_ensure_avd_running cold-boot). Do NOT mark the Learn leg pass on placeholder screenshots; record the wedge verdict and surface #629. |
Deliver leg: deliver-home-job-card wait/assert fails AND a deliver-launch-download-server-error screenshot is present (the recipe reached + tapped the Deliver download gate, then the failure surface is the PersonalID re-login screen — connect_login_button "LOGIN WITH PERSONALID" / "Couldn't connect to the server" / welcome_msg "Welcome <user>!") | Transient server-connectivity Deliver-CCZ download failure — the async CCZ download could not reach the server, so the app de-authed to PersonalID re-login. Downstream of ACE: the released CCZ already passed Phase 3's commcare-cli play install gate, so it is NOT a Nova/recipe/marker/selector defect — the recipe navigated correctly to the gate. deliver-launch.yaml now captures this labeled forensic (#772) instead of a bare 180s timeout. Surfaced live on bednet-spot-check run 20260612-2347. | Transient — re-attempt, do NOT cold-boot. Record the Deliver leg incomplete (transient), not fail. Re-run the Deliver smoke via /ace:qa-deep <opp> once backend connectivity is stable — that (app-ux-eval), not Phase 6, is the actual Phase-9 llo-launch quality gate. Do NOT mobile_ensure_avd_running cold-boot / /ace:mobile-bootstrap — the post-Learn state is fragile and a boot resets the device PIN (#618/#629). If the drop persists across retries, the backend (CommCare HQ / Connect CCZ-serve) is genuinely unreachable — escalate that, not the recipe. |
| Two capture steps in one journey hash byte-identical (surfaced by the Step 5.5 distinctness check, not a recipe error string) | Recipe screenshots before/after a no-op transition, or the transition screen doesn't exist on this opp (e.g. Connect hands off directly into an installed app, so the intermediate screen never renders). Live instances: hh-poverty-targeting/20260702-1456 connect-resume-opp-tile ≡ connect-resume-opp-list-synced, journey-deliver-submitted ≡ journey-deliver-final (dimagi-internal/ace#866). | Adjust the recipe to capture a genuinely distinct surface (e.g. the job-detail/View-Info screen) or drop the step; downstream consumers must treat duplicate_of steps as one moment. |
Manual-debug fallback — when the table doesn't resolve it. If a
recipe error matches NO row above (a novel nav defect, an
uncharacterized inter-leg handoff like #618,
a selector that resolved on a sibling APK but not this one), do NOT
halt blind. Open failureForensics.screenshotPath and read
failureForensics.uiDumpPath before writing the verdict:
.png tells you which screen the device died on (Learn home?
Connect job card? a dialog?).-FAILURE.xml ui-dump lists the resource-ids/text actually
present on that screen — diff that against the selector the recipe
reached for. "Selector absent from the dump" → wrong screen or APK
selector drift; "selector present but tap was a no-op" → a
childOf/scoping or timing issue.gh issue create
against jjackson/ace with the dump excerpt + the screen it
characterizes (the "close the loop to the source of truth" rule —
one live dump beats another plausible guess), then encode the fix
only after a live mobile_capture_ui_dump → candidate-tap → re-dump
confirms it navigates. Until confirmed live, halt loud with the
characterization in the verdict — never ship an unvalidated nav
guess (the #618 / #591 selector-drift class).After each journey's captures land (and BEFORE assembling the Step 6 manifest), compute a content hash over every PNG in the journey:
md5 -q <screenshotDir>/<journey>/*.png # or: shasum <...>/*.png
Any byte-identical pair within a journey means the two "steps" observed the SAME frame — the recipe screenshotted before/after a no-op transition, or the transition screen doesn't exist on this opp (e.g. Connect hands off directly into an installed app, so the intermediate Connect screen never renders). Handling:
duplicate_of: <first-step-name> instead of listing it as a distinct
capture.auto_surfaced
list (severity: WARN, naming both step names + the shared hash).Never present two identical frames as different moments. Downstream
consumers (training-deck-generate, training-flw-guide) treat a
duplicate_of step as the same moment as its canonical step — never as
a distinct panel or slide. Surfaced live on
hh-poverty-targeting/20260702-1456, where connect-resume-opp-tile ≡
connect-resume-opp-list-synced and journey-deliver-submitted ≡
journey-deliver-final shipped as distinct steps and the training deck
then claimed distinct moments over duplicate frames
(dimagi-internal/ace#866).
6-qa-and-training/app-screenshot-capture_manifest.yamlLink each captured PNG back to (a) its journey id (a meaningful slug
like journey-learn-pass / journey-deliver-submit from app-test-cases.yaml), (b) its
takeScreenshot: step label, (c) its Drive path. This is the input
shape the per-artifact training skills (training-flw-guide,
training-deck-generate) consume. Steps flagged by the Step 5.5
distinctness check carry duplicate_of: <first-step-name> instead of
being listed as distinct captures.
Run the drift harvester over this run's screenshot dir (the same
screenshotDir you passed to mobile_run_recipe in Step 5, where the
*-FAILURE.xml / *-FAILURE.png forensics and the per-step runRecipeWithDumps
XMLs accumulated). This closes the consume-loop on selector drift —
otherwise the dumps just pile up unread.
npx tsx scripts/probe-atlas-drift.ts <screenshotDir> \
--apk "${ACE_CONNECT_APK_VERSION:-2.63.0}" \
--out <screenshotDir>/atlas-drift-report.md
Best-effort: a probe error never fails the phase — log it and continue.
When the report has a "⚠️ Drift suspects on FAILURE screens" section,
treat those resource-ids as the priority signal: each is a candidate
root cause for a recipe failure in this run (the recipe reached for a
logical name whose mapped id: no longer matches what this APK renders —
the #591/#618 selector-drift class). Do NOT auto-edit the selector map —
surface the candidates in the verdict and, when a candidate is
confirmed live (mobile_capture_ui_dump → candidate-tap → re-dump
navigates), gh issue create against jjackson/ace proposing the new
selectors.<logical-name> row. This is the "close the loop to the
source of truth" rule — one live dump beats another plausible guess.
For each smoke recipe (Learn + Deliver), assemble the captured screenshot set into a single LLM-as-Judge call:
Prompt: "These screenshots are from a smoke run of the {{app}} app. The target FLW persona (from PDD) is: {{persona_summary}}. Looking at these screenshots in order, would this person be able to complete the journey without confusion? Rate 0-3 + one-line reason. 0 = a typical persona-matching FLW would get stuck; 3 = obviously usable."
Threshold: ≥ 2/3 per app. Below → halt with verdict.
This is intentionally one LLM call per app (~2 calls total). Deep,
per-dimension UX grading is app-ux-eval running from /ace:qa-deep.
For the structural-quality dimensions:
takeScreenshot step)?Write the canonical structural verdict to
6-qa-and-training/app-screenshot-capture_verdict.yaml AND the
shallow smoke verdict to
6-qa-and-training/app-screenshot-capture_verdict-shallow.yaml. Both
shapes conform to lib/verdict-schema.ts so opp-eval can aggregate.
Structural verdict (6-qa-and-training/app-screenshot-capture_verdict.yaml):
skill: app-screenshot-capture
target: <opp-name>
ran_at: <ISO timestamp>
capture_path: 6-qa-and-training/app-screenshot-capture_manifest.yaml
overall_score: 8.5 # 0.0–10.0, weighted across dimensions
verdict: pass | warn | fail | incomplete
# `incomplete` is reserved for *upstream-incomplete* state the skill
# itself cannot remediate: app-test-cases.yaml missing or malformed,
# `3-commcare/recipes/journey-*.yaml` files absent, recipes carrying
# unfilled REPLACE_* selectors. That's "the rubric COULD NOT grade
# because Phase 3 didn't ship its full output set," not "the AVD is
# sick."
#
# AVD / Maestro driver health is **not** incomplete-state — it's
# `verdict: fail` with severity:BLOCKER. Pre-0.13.165 a hung Maestro
# driver wrote `incomplete` and the phase shipped placeholders; that
# escape valve let real Phase 6 capability problems hide behind a
# benign-looking yellow verdict run after run. Since 0.13.165
# `mobile_ensure_avd_running` includes an auto-heal (force-stop +
# uninstall + reinstall of `dev.mobile.maestro`), so a healthy
# workstation almost never needs to surface AVD failure to this
# rubric. When the heal exhausts, we want the BLOCKER, not the soft
# fail — the operator runs `/ace:mobile-bootstrap` and the run
# retries with screenshots. NEVER use `verdict: blocked`
# (off-schema; not in lib/verdict-schema.ts).
dimensions:
coverage: { score: 9.0, weight: 0.30 } # both smoke journeys covered
execution: { score: 8.5, weight: 0.30 } # every smoke recipe status == pass
artifact_quality: { score: 9.0, weight: 0.20 } # every screenshot is a valid PNG, non-zero bytes
manifest_integrity: { score: 8.0, weight: 0.20 } # manifest.yaml lists every screenshot actually present in Drive
per_item:
- ref: learn
score: 9.0
verdict: pass
note: "journey-learn walked to completion; 6 screenshots, all PNG"
- ref: deliver
score: 0
verdict: incomplete # or fail / pass
note: "journey-deliver.yaml missing — Phase 3 deferred it"
Top-level verdict from the two legs:
| Learn | Deliver | top-level verdict | phase proceeds clean? |
|---|---|---|---|
| pass | pass | pass | yes |
| pass | fail (ran, broke) | fail | no — blocks |
| pass | incomplete (recipe missing/scaffold) | incomplete | no — blocks |
| fail | blocked-by-learn | fail | no — blocks |
| incomplete | incomplete | incomplete | no — blocks |
A non-pass verdict still ships the Learn screenshots it captured. Operator-authorized whole-step skip remains the separate explicit escape (unchanged).
auto_surfaced:
- severity: WARN
message: "Recipe X timed out at step Y; partial screenshots captured"
Shallow smoke verdict (6-qa-and-training/app-screenshot-capture_verdict-shallow.yaml):
skill: app-screenshot-capture
target: <opp-name>
mode: shallow
ran_at: <ISO timestamp>
capture_path: 6-qa-and-training/app-screenshot-capture_manifest.yaml
overall_score: 2.5 # average of per-app smoke-judge scores (0-3 scale)
verdict: pass | fail | incomplete
# pass iff every per-app smoke-judge score >= 2/3.
dimensions:
ux_smoke: { score: 2.5, weight: 1.00 } # mean of per-app smoke judge
per_item:
- ref: "learn"
score: 3
verdict: pass
note: "Field labels clear, submission confirmation visible"
- ref: "deliver"
score: 2
verdict: pass
note: "Mostly clear; one screen has a developer-named field but recoverable"
auto_surfaced: []
Incomplete-mode shape (when blocked before grading):
skill: app-screenshot-capture
target: <opp-name>
ran_at: <ISO timestamp>
capture_path: phase5-block.md
overall_score: 0 # required by schema; not meaningful when incomplete
verdict: incomplete
live_state_verified: false
dimensions:
coverage: { score: 0, weight: 0.30 }
execution: { score: 0, weight: 0.30 }
artifact_quality: { score: 0, weight: 0.20 }
manifest_integrity: { score: 0, weight: 0.20 }
auto_surfaced:
- severity: PLATFORM
message: "ACE_E2E_PHONE family unset in .env; mobile_register_test_user cannot run"
- severity: PLATFORM
message: "app-test-cases.yaml missing or no is_smoke: true entries; re-run /ace:step app-test-cases <opp>"
If a sub-step here needs to POST to a Django-backed endpoint (the ACE
web app's /api/ingest/upload, an OCS / Connect write, etc.) from inside
a Playwright browser context, never use page.request.post(). The
Playwright request context has its own cookie jar that doesn't share
the page's session cookies, so Django rejects the write with HTTP 403
("CSRF cookie not set" or "CSRF token missing or incorrect"). This is
the same class of bug mcp/connect/backends/playwright.ts and
mcp/ocs-server.ts already work around in production (search those
files for X-CSRFToken for the canonical examples).
The two-step pattern is: read the CSRF cookie from document.cookie,
then issue the fetch from inside the page so the browser's own cookie
jar travels with the request:
// Inside the Playwright session, after a GET has warmed csrftoken_ace:
const csrf = await page.evaluate(() => {
const m = document.cookie.match(/(?:^|;\s*)csrftoken_ace=([^;]+)/);
return m ? m[1] : null;
});
if (!csrf) throw new Error('csrftoken_ace not set — GET base URL first to warm it');
const response = await page.evaluate(
async ([url, csrf, body]) => {
const r = await fetch(url, {
method: 'POST',
headers: { 'X-CSRFToken': csrf as string, 'Content-Type': 'application/json' },
credentials: 'same-origin',
body: JSON.stringify(body),
});
return { status: r.status, body: await r.text() };
},
[endpoint, csrf, payload],
);
Notes:
csrftoken_ace on the ACE web app (ace-web); on the
OCS / Connect Django backends it's plain csrftoken. Pick the right
one for the host you're targeting.CsrfViewMiddleware
has been hit — a single await page.goto(baseUrl) first is enough to
warm it. See skills/upload-transcript/SKILL.md § Shell reference for
the curl-cookie-jar equivalent of the same pattern.X-CSRFToken (capital T-O-K-E-N), not X-CSRF-Token.
Django's CsrfViewMiddleware only honors the former.ace-gdrive: drive_read_file, drive_upload_binary, drive_create_file, drive_list_folder.ace-mobile: mobile_ensure_avd_running, mobile_install_apk, mobile_run_recipe.mobile_run_recipe calls.dry_run: true flag.dry-run-success.| Dimension | Pass criteria |
|---|---|
| Coverage | both legs attempted; Learn always; Deliver iff Learn completed |
| Execution | every smoke recipe status: pass |
| Artifact quality | every screenshot is a valid PNG with non-zero bytes |
| Manifest integrity | manifest.yaml lists every screenshot path actually present in Drive |
| UX smoke (shallow verdict only) | per-app score ≥ 2/3 from the thin smoke judge |
| Date | Change | Author |
|---|---|---|
| 2026-07-13 | Step 5.5 distinctness check (dimagi-internal/ace#866). After each journey's captures land and before the manifest is assembled, hash every PNG in the journey (md5 -q / shasum); any byte-identical pair means two "steps" observed the same frame. The FIRST step stays canonical; later duplicates are marked duplicate_of: <first-step> in the manifest (not listed as distinct captures) and logged in the verdict's auto_surfaced list. Matching row added to the Step 5 recipe-error table. Surfaced by hh-poverty-targeting/20260702-1456, where two byte-identical pairs shipped as distinct steps and deck slides claimed distinct moments over duplicate frames. | ACE team |
| 2026-06-12 | Step 5 hard rule: screenshots only from a status: pass execution in THIS run (jjackson/ace#756). A status != pass mobile_run_recipe result for a required journey means that leg has NO screenshots — record the leg fail with the recipe failure inline; never source leftover PNGs (no "replay cache" exists); the shallow verdict must not be pass if any required journey's recipe failed. Pairs with the MCP-side fix: mobile_run_recipe now wipes the screenshot dir at execution start so stale PNGs are structurally absent. Surfaced by bednet-spot-check run 20260612-1220, where a failed deliver leg shipped prior-run PNGs under a confabulated "runner replay cache" note. | ACE team |
| 2026-06-03 | Step 6.5 auto-harvests selector drift. End of Phase 6 now runs scripts/probe-atlas-drift.ts over the run's screenshot dir automatically (best-effort), closing the consume-loop on the runRecipeWithDumps + *-FAILURE.xml dumps that previously just accumulated. The harvester is now FAILURE-aware: resource-ids seen on a *-FAILURE.xml screen but absent from the selector map surface as a priority "Drift suspects on FAILURE screens" section (candidate root causes for a recipe failure in this run — the #591/#618 drift class). Pairs with thrown-failure forensics capture (mobile_run_recipe now captures the failure screen on a thrown driver-death too, not just on returned status:'fail'). Canonical contract: playbook/integrations/mobile-integration.md § Failure forensics. | ACE team |
| 2026-05-05 | Path-scheme migration. Inputs repointed to 2-scenarios/pdd-to-app-journeys.md, 3-commcare/app-test-cases.yaml, 3-commcare/app-deploy_summary.md, 3-commcare/recipes/. Outputs repointed to 6-qa-and-training/screenshots/<recipe-base>/<step-name>.png, 6-qa-and-training/app-screenshot-capture_manifest.yaml, 6-qa-and-training/app-screenshot-capture_verdict.yaml, 6-qa-and-training/app-screenshot-capture_verdict-shallow.yaml (per manifest). Both verdict YAML examples' capture_path updated. No behavior change beyond paths. | ACE team |
| 2026-05-27 | Recipe naming convention. Screenshot dirs updated from <journey-id>/ to <recipe-base>/ (journey-learn/, journey-deliver/). Recipe read references updated from J*.yaml to journey-*.yaml. Structural verdict per_item refs changed from ref: "J1.yaml" to ref: learn / ref: deliver. No runtime behavior change. See spec 2026-05-27-phase6-learn-deliver-decoupling. | ACE team |
| 2026-05-31 | Meaningful journey ids. The manifest's journey-id reference is now a meaningful slug (learn-happy-path / deliver-yes) from app-test-cases.yaml instead of J<n>. Pairs with the descriptive recipe filenames / screenshot dirs already in place. See skills/app-test-cases/SKILL.md § Journey id convention. | ACE team |
| 2026-05-31 | journey- prefix. The manifest's journey-id reference examples now carry the journey- prefix (journey-learn-pass / journey-deliver-submit) to match the amended id convention. See skills/app-test-cases/SKILL.md § Journey id convention. | ACE team |
| 2026-05-31 | Tolerate recipe: deferred (lazy deep recipes, #605). Phase 3 now authors recipe files only for the two is_smoke: true journeys; deep journeys carry recipe: deferred and have no file (generated on demand by /ace:qa-deep). Step 2's pre-flight is scoped explicitly to the is_smoke: true subset, and a deferred deep journey with no recipe file is no longer a missing-recipe failure. This skill already only walked the smokes — this change makes the deferred-deep state explicitly non-fatal. | ACE team |
| 2026-05-27 | Two-leg capture split. Step 5 rewritten as independent Learn leg + Deliver leg. Learn always runs first; a Deliver failure (or missing recipe) no longer suppresses Learn screenshots. Per-app per_item verdict mapping table added (pass/fail/incomplete/blocked-by-learn outcomes). Coverage rubric updated: "both legs attempted; Learn always; Deliver iff Learn completed." See spec 2026-05-27-phase6-learn-deliver-decoupling. | ACE team |
| 2026-05-06 | Step 2 input-completeness pre-flight — restructured the post-Step-1 logic into an explicit failure-mode table that distinguishes upstream Phase 3 incomplete output (master yaml without recipes) from smoke-flag malformation. Each failure halts with a named PLATFORM auto_surfaced message + the exact /ace:step remediation command, and writes verdict: incomplete (not fail — upstream gaps aren't smoke failures). Surfaced by leep-paint-collection run 20260506-1440 where a Phase 3 dispatch paraphrased the app-test-cases SKILL contract and elided the per-journey recipe outputs; app-screenshot-capture halted correctly but the operator-facing message conflated the failure mode with general "missing input" diagnostics. See jjackson/ace#106 finding #3 + #16. | ACE team |
| 2026-05-07 | Step 5 anyone-with-link via drive_upload_binary({shareAnyoneWithLink: true}) — replaces the previous unfulfillable contract (the SKILL named drive.permissions.create but no MCP atom implemented it). The new flag sets role: reader, type: anyone atomically at upload time, eliminating the "deck builds without errors but slides are empty" failure mode. Standalone drive_set_anyone_with_link({fileId}) atom also added for retroactive sharing. See jjackson/ace#115 finding #3. | ACE team |