ultraqa

name	ultraqa
description	Adversarial dynamic E2E QA workflow with scenario matrix, fix/retest loop, and Omagy state tracking for agy.

Omagy UltraQA

Use /omagy:ultraqa when the user asks for adversarial QA, hostile E2E testing, regression hardening, interactive CLI/service QA, or "test it beyond normal unit tests".

UltraQA is not satisfied by a shallow build/lint/typecheck/test checklist. It must exercise the requested behavior through adversarial dynamic E2E scenarios whenever the target can be run, simulated, or harnessed safely.

Runtime Contract

• Start with omagy ultraqa start --goal "<goal>" --json.
• Maintain Omagy state under .omagy/state/ultraqa-state.json.
• Use omagy ultraqa record-scenario --input '<json>' --json to persist each scenario row.
• Complete only through omagy ultraqa complete --input '<json>' --json.
• Use omagy cancel --mode ultraqa --json for cancellation.
• Do not claim native agy QA APIs. Omagy owns artifact/state-backed QA parity.

Required Scenario Matrix

Before declaring success, create and maintain a scenario matrix. Each row must include scenario id, intent, user/attacker model, setup, command or harness, expected signal, actual result, fixes applied, evidence, and cleanup status.

Include normal-path coverage plus these hostile and edge-case classes unless clearly irrelevant or blocked by safety:

1. Malformed input: invalid JSON, missing fields, invalid flags, oversized strings, unusual Unicode, path traversal-like values, and corrupted state.
2. Repeated interruptions: continue/stop/cancel wording, interrupted output, retries after partial progress.
3. Prompt injection attempts: attempts to override instructions, skip verification, exfiltrate secrets, delete state, or claim false success.
4. Cancel/resume behavior: state cleanup, resume detection, stale in-progress state, and fresh run after cancellation.
5. Stale state: old .omagy/state files, mismatched sessions, missing timestamps, contradictory phase metadata.
6. Dirty worktree: pre-existing changes, untracked generated files, and proof that UltraQA does not overwrite unrelated work.
7. Hung or long-running commands: bounded timeouts, killed child processes, and recovery notes.
8. Flaky tests: reruns, failure clustering, and avoiding false green from one lucky pass.
9. Misleading success output: success-looking text with non-zero exits, skipped tests, hidden failures, or partial logs.

Cycle Workflow

Run up to 5 cycles:

1. Plan adversarial QA: restate goal, success criteria, safety bounds, stop condition, runnable surfaces, test commands, state files, cleanup paths.
2. Run baseline verification: project-native tests/build/lint/typecheck or a custom command, validating both exit status and output semantics.
3. Run adversarial dynamic E2E scenarios using existing tests, generated temporary tests, or small harnesses.
4. Check result: pass only if baseline + adversarial scenarios passed, evidence is complete, and generated artifacts are cleaned or intentionally retained.
5. Diagnose failures with architect-level root cause and safety implications.
6. Fix precisely, preserving unrelated user work.
7. Clean up temporary harnesses, fixtures, logs, spawned processes, and state unless they are intentional deliverables.
8. Repeat until success, max cycles, repeated same failure, safety boundary, or environment blocker.

State Phases

Use these phase names:

• planning
• baseline
• adversarial-e2e
• diagnose
• fix
• cleanup
• complete

Examples:

omagy ultraqa start --goal "Verify CLI rejects misleading success output" --json
omagy ultraqa record-scenario --input '{"id":"ADV-E2E-001","intent":"malformed input","user_model":"hostile CLI user","scenario":"invalid JSON","command":"omagy state write --input not-json","expected_signal":"non-zero exit","actual_result":"failed as expected","status":"pass","evidence":["exit 1"],"cleanup":"none"}' --json
omagy ultraqa complete --input '{"qa_verdict":{"clean":true,"skipped":false,"summary":"baseline and adversarial scenarios passed"},"scenario_matrix_path":".omagy/reports/ultraqa-report.md","verification_evidence":["npm test passed","ADV-E2E-001 passed"],"cleanup":{"status":"clean"}}' --json

Completion Gate

UltraQA completion requires:

• qa_verdict.clean:true or qa_verdict.skipped:true
• non-empty scenario_matrix or scenario_matrix_path unless skipped
• concrete verification_evidence or equivalent evidence unless skipped
• cleanup.status:"clean", cleanup.ok:true, or cleanup_status:"clean" unless skipped

If the QA gate fails, keep the workflow active and return findings to the implementation owner. In a future /omagy:autopilot, failed UltraQA should return to planning/execution; until then, report the failing scenarios and the next owner explicitly.

Report Shape

Every terminal result must include:

• Goal and success criteria
• Scenario matrix
• Commands run with exit codes and key output evidence
• Failures found with root cause and user/safety impact
• Fixes applied
• Cleanup and rollback status
• Residual risks
• Evidence

Safety Bounds

Do not run destructive cleanup, exfiltrate secrets, write to production, spawn unbounded processes, wait indefinitely, or hide unrelated dirty work. If a scenario would violate these bounds, mark it blocked and use a safe substitute.