opice-reeval

name	opice-reeval
description	Diagnose and fix a failed opice CI run. Given a failed-run URL (or run id), pulls the failure details from the platform, reproduces each failing scenario locally in a real browser, and either fixes the test (selector drift, timing) or flags a genuine app regression — without ever gutting an assertion just to make it pass. Trigger when the user says "/opice-reeval <url>", "look at this failed run <url>", "this opice run failed, fix it", "re-eval the failing CI", or pastes a dashboard run link and asks what broke.
allowed-tools	Bash(opice:*), Bash(bunx opice:*), Bash(opice-browser:*), Bash(bun:*), Bash(gh:*), Bash(curl:*), Bash(git:*), Read, Edit, Write, Glob, Grep

opice-reeval — diagnose & fix a failed run

A CI run went red. Your job: find out why each scenario failed, reproduce it locally, and resolve it correctly — fix the test when the test is wrong, raise a flag when the app is wrong. The whole value of this skill is that judgment; do not destroy it by weakening assertions to force green.

The one rule that matters

Never make a failing test pass by removing or loosening the assertion that caught the failure. A red test is either:

• test drift — the app is fine, the test is stale (selector changed, timing, renamed label). → Fix the test.
• a real regression — the app is broken, the assertion did its job. → Do not touch the test. Report the regression with evidence (the step, the error, the screenshot) and stop.

When unsure which it is, treat it as a possible regression and ask. Silently deleting an assertion is the worst outcome this skill can produce.

Inputs

• A failed-run URL (paste from the dashboard, includes ?token=), or a bare run id plus a configured endpoint and a read credential — OPICE_READ_DSN (the project's read key, dropped into .env at project setup) or OPICE_READ_TOKEN.
• The repo checked out locally, with the playground runnable.

Workflow

1. Pull the failure digest

opice failures "<run-url-or-id>"        # human-readable
opice failures "<run-url-or-id>" --json # if you want to parse it

This gives you, per failed scenario: the failing step, the error, an absolute (tokened) screenshot URL, and the source file (test_file) — so you don't have to grep. The test IS the spec: each step carries its durable intent and the scenario carries its invariants, so everything you need to judge "is this the right assertion?" is in that one file (no separate .scenario.md). If opice failures can't reach the platform or the token is missing, fall back to CI logs:

gh run view <run-id> --log-failed     # the raw bun assertion + stack

CI logs have the exact stack but no screenshots and no step framing — use the platform digest as the primary source and CI logs to go deeper on a stack.

2. Look at the failure before touching anything

For each failed scenario, download the failure screenshot and read it — it usually tells you instantly whether the UI is broken or just shaped differently:

curl -s "<screenshot-url-with-token>" -o /tmp/opice-fail-<n>.png

Read /tmp/opice-fail-<n>.png and the step error together. Open the test_file — the failing step's intent (and any scenario invariant) tells you what the assertion was supposed to prove.

3. Reproduce locally

Make sure the playground is running. Then run just the failing test:

bun test <test_file>

• Fails locally too → reproducible. Go to step 4.
• Passes locally → not reproducible: flaky or environment-specific (timing, seed data, CI-only state). Don't "fix" code that passes — diagnose the divergence (race needing waitFor? data the scenario assumed?), propose the minimal robustness change, and say it was green locally.

4. Walk it live to find the cause

Reproduce the scenario in a real browser the same way opice-author does — re-snapshot, follow the scenario's steps, and find where reality diverges:

opice-browser launch <URL>#<hash>
opice-browser aria-snapshot main      # the agent's view; re-run after each step

Classify the divergence:

• Selector drift (testid/label/role changed, element moved): update the selector/locator to a stable one (el/byRole/byLabel). Test fix. ✔
• Timing (assertion ran before async UI settled): replace fixed waits with a retrying await expect(el(x)).toHaveText(...) on a stable marker. Test fix. ✔
• Intent wrong (the step's intent / an invariant describes behaviour the app never had / no longer should have): the spec is the bug — update the intent (or invariant) and its body together, in the test, and say so. ✔ (with the user's nod). Don't leave a body that contradicts its own intent.
• App regression (the app should do X, it now does Y, the assertion is right): stop fixing. Report it — scenario, step, expected vs actual, the screenshot — as a regression. ✘ Do not edit the test.

5. Verify the fix

After any edit, re-run until green for the right reason — the assertion still asserts the step's stated intent:

bun test <test_file>

If you changed selectors/timing, confirm the test still fails when the app behaviour it checks is actually broken (don't over-relax).

6. Report

Per failed scenario, one line: fixed (selector), fixed (timing), intent updated, flaky — green locally, or REGRESSION — app bug, test left red. Then a short summary grouping them. List exactly which files you changed.

Do not commit unless the user asks. When they do, commit atomically with an explicit file list (per the user's git rules), and never bundle a real regression's test into a "fix" commit.

Notes

• Multiple failed scenarios: handle them one at a time; they often share a root cause (one renamed testid breaks five tests).
• If many scenarios fail from one UI refactor, this is a re-author job — point the user at opice-batch to regenerate the suite, rather than hand-patching each test here.
• Re-eval reuses opice-author's live-walking and selector-resolution procedure for the fix step — when in doubt about how to pick a selector, defer to that skill's preference order (data-testid first).