| name | ultraqa |
| description | Use when a feature or flow needs adversarial end-to-end hardening before shipping because happy-path verification is insufficient. |
ultraqa
Adversarial end-to-end QA, run as a ralph profile: you drive oma ralph with a deliberately hostile scenario matrix and loop until every scenario behaves safely. The CLI owns the loop mechanics (counting, stall detection, stop judgment); you own the adversarial scenario design and run the checks yourself. ultraqa adds no new engine — it is oma ralph with an adversarial goal.
Use to harden a feature/flow before shipping — when "it works on the happy path" isn't enough.
Start
Frame the QA bar as verifiable, then start a ralph loop:
oma ralph start --goal "all ultraqa scenarios pass: <feature>" --max-rounds 10 --stall-window 3 --id qa-<slug>
The scenario matrix (design before looping)
Build a matrix across these axes — pick the ones that fit the target:
- Malformed / hostile input — empty, oversized, wrong-type, injection (SQL / shell / prompt), unicode & encoding edge cases.
- Lifecycle — cancel mid-operation, resume after interruption, double-submit, out-of-order calls, concurrency.
- Boundary — zero / one / max / max+1, empty collection, missing optional fields, expired or reused tokens.
- State / environment — stale cache, partial write, missing dependency, permission denied, network failure / timeout.
- Adversarial user — someone trying to break it, escalate privilege, or reach another tenant's data.
- Misleading success — a 0 exit with failure text in the output, a SUCCESS message over a partial write, a swallowed error reported as done. The exit code and the observable result must agree.
Each scenario states: setup → action → expected safe behavior (reject / degrade / recover — never crash, corrupt, or leak).
Each round
-
oma ralph next --json — continue:false (exit 4) means a terminal state; act on it, don't push on.
-
Run the next batch of scenarios YOURSELF; compare actual vs expected behavior.
-
Fix the defects the round surfaced.
-
Record the result:
oma ralph check --verifier-exit <0|nonzero> --note "<failing scenario signature>"
exit 0 only when EVERY scenario passes. The --note is the stall signature: same scenario failing → same note; a new failure → a new note.
Coverage-scored variant (optional) — for a large matrix where partial progress matters, drive the loop under --keep-policy score_improvement and report --score <fraction safe> (e.g. 7 of 9 → 0.78) each round instead of a binary pass; ralph keeps the best coverage and reports plateaued when fixes stop raising it (see the ralph skill).
Terminal states (from ralph)
- passed — every scenario behaves safely. Report the matrix + outcomes.
- exhausted — budget ran out: list scenarios still failing and the closest-safe state; ask to raise the bound or stop.
- stalled — the same scenario keeps failing the same way: the fix approach is wrong; present 2–3 genuinely different strategies.
- plateaued (coverage-scored runs) — safe-scenario coverage stopped improving for plateau_window rounds: the current fixes are mined out; report best coverage and switch to 2–3 different strategies.
Hard rules
- No new engine — never re-implement counting / stall / stop; that is
oma ralph.
- A scenario "passes" only on observed safe behavior, never assumption. You run the checks; you never fake an exit code.
- Expand the matrix when a defect reveals a missed axis — coverage grows with what you learn.
- Done = every matrix scenario green, recorded via
oma ralph check.
- Keep your OWN probes inside safe bounds: bounded timeouts, no destructive / production / secret-exfil actions, no unbounded spawns. You are firing hostile inputs — don't let the harness become the incident.
Parallel acceleration (optional, capability-gated): Delegate isolated, non-destructive scenario execution from the defined matrix; keep fixes sequential, and let the parent own the matrix and aggregate oma ralph check. Gate requires lifecycle-controllable subagent tools; at least two independent bounded lanes whose critical-path benefit beats coordination; no lane waits on user, peer, or unstable input; exclusive file/worktree writes, no generated/shared single-writer state; parent synthesis and final verification. Brief objective, inputs, output, boundary, stop conditions; normally no more than three, no nested delegation. Lane output is evidence, not a verdict. Parent owns questions, shared state, integration, and completion. If gate/lane fails, breaches scope, or conflicts: stop affected lanes, keep verified evidence, continue sequentially.
