ultraqa

name	ultraqa
description	[OMX] Adversarial dynamic e2e QA workflow - generate hostile scenarios, test, verify, fix, report, and clean up

UltraQA Skill

Operating Contract

• Use outcome-first framing with concise, evidence-dense progress and completion reporting.
• Treat newer user updates as local overrides for the active workflow branch while preserving earlier non-conflicting constraints.
• If the user says continue, advance the current verified next step instead of restarting discovery.
• UltraQA is not satisfied by a shallow build/lint/typecheck/test checklist. It must exercise the requested behavior through adversarial dynamic e2e scenarios whenever the target can be run, simulated, or harnessed safely.

[ULTRAQA ACTIVATED - ADVERSARIAL DYNAMIC E2E QA CYCLING]

Overview

UltraQA finds real behavior failures by combining normal verification commands with generated end-to-end scenarios, hostile user modeling, temporary harnesses when useful, and a structured evidence report. The workflow repeats test → diagnose → fix → retest until the goal is met, a bounded stop condition is reached, or a safety boundary blocks further execution.

Goal Parsing

Parse the goal from arguments. Supported formats:

Invocation	Goal Type	What to Check
/ultraqa --tests	tests	Existing tests plus adversarial dynamic e2e scenarios for the changed behavior
/ultraqa --build	build	Build succeeds and generated smoke/e2e probes still run against the built artifact when applicable
/ultraqa --lint	lint	Lint passes and no generated harness/test artifact violates project hygiene
/ultraqa --typecheck	typecheck	Typecheck passes and generated typed harnesses compile when applicable
/ultraqa --custom "pattern"	custom	Custom success pattern is verified against behavior, not trusted as misleading success output
/ultraqa --interactive	interactive	CLI/service behavior is tested with generated hostile and edge-case interactions

If no structured goal is provided, interpret the argument as a custom behavior goal and derive a runnable e2e strategy from repository context.

Required Scenario Matrix

Before declaring success, create and maintain a scenario matrix. Each row must include: scenario id, intent, user/attacker model, setup, command or harness, expected signal, actual result, fixes applied, evidence, and cleanup status.

The matrix must include normal-path coverage plus adversarial dynamic e2e scenarios selected from the current goal and codebase. Unless clearly irrelevant or impossible, include these hostile and edge-case classes:

1. Malformed input: invalid JSON, missing fields, invalid flags, oversized strings, unusual Unicode, path traversal-like values, and corrupted state files.
2. Repeated interruptions: repeated continue, stop/cancel/abort wording, interrupted command output, and retries after partial progress.
3. Prompt injection attempts: user text that tries to override instructions, exfiltrate secrets, skip verification, delete state, or claim false success.
4. Cancel/resume behavior: active state cleanup, resume detection, stale in-progress state, and cancellation followed by a fresh run.
5. Stale state: old .omx/state files, mismatched sessions, missing timestamps, and contradictory phase metadata.
6. Dirty worktree: pre-existing modifications, untracked generated files, and verification that UltraQA does not hide or overwrite unrelated work.
7. Hung or long-running commands: bounded timeout handling, killed child processes, and recovery notes.
8. Flaky tests: rerun strategy, failure clustering, quarantine evidence, and avoiding false green from a single lucky pass.
9. Misleading success output: output containing success phrases with non-zero exits, hidden failures, skipped tests, or partial command logs.

Dynamic E2E and Temporary Harness Rules

• Generate temporary tests, scripts, fixtures, or harnesses when they materially improve behavioral confidence and no existing e2e surface covers the scenario.
• Prefer project-native test tools and small throwaway harnesses under a temporary directory or clearly named test fixture.
• Record every generated artifact in the scenario matrix, including whether it was committed intentionally or removed during cleanup.
• Use bounded runtimes and explicit timeouts for commands that can hang.
• Validate exit codes and output semantics; do not trust success-looking text alone.
• Do not delete, rewrite, or mask unrelated user work. Capture dirty-worktree evidence before and after generated harness work.

Cycle Workflow

Cycle N (Max 5)

1. PLAN ADVERSARIAL QA

   • Restate the goal, success criteria, safety bounds, and stop condition.
   • Inspect repository context enough to identify runnable surfaces, test commands, state files, and cleanup paths.
   • Build or update the required scenario matrix before running commands.

2. RUN BASELINE VERIFICATION

   • --tests: Run the project's test command.
   • --build: Run the project's build command.
   • --lint: Run the project's lint command.
   • --typecheck: Run the project's type check command.
   • --custom: Run the appropriate command and check the pattern plus exit status and failure markers.
   • --interactive: Use qa-tester or an equivalent CLI/service harness:

     Use `/prompts:qa-tester` with:
     Goal: [describe what to verify]
     Service: [how to start]
     Test cases: [normal, hostile, malformed, interruption, resume, stale-state, dirty-worktree, hung-command, flaky, and misleading-output scenarios]
     

3. RUN ADVERSARIAL DYNAMIC E2E SCENARIOS

   • Execute the scenario matrix using existing e2e tests, generated temporary tests, or generated harnesses.
   • Model malicious/hostile user behavior explicitly, including prompt injection and attempts to bypass safety or verification.
   • Exercise malformed input, repeated interruptions, cancel/resume, stale state, dirty worktree handling, hung commands, flaky tests, and misleading success output when relevant.
   • Capture commands, exit codes, important output excerpts, artifacts, and cleanup status.

4. CHECK RESULT

   • YES only if baseline verification and adversarial e2e scenarios passed, generated artifacts are cleaned up or intentionally tracked, and the report has complete evidence.
   • NO if any scenario failed, was skipped without justification, left debris, relied on misleading output, or lacked evidence. Continue to step 5.

5. ARCHITECT DIAGNOSIS

   Use `/prompts:architect` with:
   Goal: [goal type and behavior]
   Scenario matrix: [rows, commands, failures, evidence]
   Output: [test/build/e2e/harness output]
   Provide root cause, safety implications, and specific fix recommendations.
   

6. FIX ISSUES

   Use `/prompts:executor` with:
   Issue: [architect diagnosis]
   Files: [affected files]
   Constraints: preserve unrelated dirty work, clean temporary harnesses, keep safety bounds
   Apply the fix precisely as recommended.
   

Safety Bounds

UltraQA must stay inside these safety bounds:

• No destructive commands such as force resets, broad deletes, secret exfiltration, credential dumping, production writes, or unbounded process spawning.
• No reading or printing secrets beyond the minimum metadata needed to verify absence of leakage.
• No network or external-production side effects unless the user explicitly authorized them.
• No unbounded waits: use timeouts, retries with caps, and clear hung-command diagnostics.
• No hiding unrelated dirty work or generated debris.
• If a required scenario would violate these bounds, mark it blocked in the report with the safe substitute used.

Exit Conditions

Condition	Action
Goal Met	Exit with success: ULTRAQA COMPLETE: Goal met after N cycles plus the structured report
Cycle 5 Reached	Exit with diagnosis: ULTRAQA STOPPED: Max cycles plus failures, fixes attempted, residual risks, and evidence
Same Failure 3x	Exit early: ULTRAQA STOPPED: Same failure detected 3 times plus root cause, safety notes, and next owner
Safety Boundary	Exit: ULTRAQA BLOCKED: [destructive/credentialed/external-production/unbounded action] plus safe substitute evidence
Environment Error	Exit: ULTRAQA ERROR: [tmux/port/dependency/hung command issue] plus cleanup status

Structured Report

Every terminal UltraQA result must include this report shape:

# UltraQA Report

## Goal and success criteria
- Goal:
- Stop condition:
- Safety bounds applied:

## Scenario matrix
| ID | User/attacker model | Scenario | Command/harness | Expected signal | Actual result | Status | Evidence | Cleanup |
|----|---------------------|----------|-----------------|-----------------|---------------|--------|----------|---------|

## Commands run
- `[exit code] command` — purpose, duration/timeout, key output evidence

## Failures found
- Scenario ID, failure signal, root cause, user impact, safety impact

## Fixes applied
- Files changed, rationale, linked failing scenario(s), regression evidence

## Cleanup and rollback
- Generated artifacts removed or intentionally kept
- State/process cleanup performed
- Worktree status before/after

## Residual risks
- Untested or blocked scenarios with reasons and safe substitutes

## Evidence
- Test output, e2e logs, harness output, screenshots/transcripts when relevant, and rerun/flake evidence

Observability

Output progress each cycle:

[ULTRAQA Cycle 1/5] Planning adversarial scenario matrix...
[ULTRAQA Cycle 1/5] Running baseline tests...
[ULTRAQA Cycle 1/5] Running ADV-E2E-003 prompt-injection harness...
[ULTRAQA Cycle 1/5] FAILED - stale state resume accepted misleading success output
[ULTRAQA Cycle 1/5] Architect diagnosing scenario ADV-E2E-003...
[ULTRAQA Cycle 1/5] Fixing: src/hooks/... - validate exit code before success phrase
[ULTRAQA Cycle 1/5] Cleaning temporary harnesses and state...
[ULTRAQA Cycle 2/5] PASSED - baseline + 9 adversarial scenarios pass
[ULTRAQA COMPLETE] Goal met after 2 cycles

State Tracking

Use the CLI-first state surface (omx state ... --json) for UltraQA lifecycle state. If explicit MCP compatibility tools are already available, equivalent omx_state calls are optional compatibility, not the default.

Mode State Relationship Policy

UltraQA may run while another workflow mode is active. Treat that as an explicit state relationship, not as a reason to clear or overwrite parent state.

• child: UltraQA is verifying work for an active parent mode. Record mode_relationship:"child", parent_mode, and parent_session_id when those values are known. Keep the parent state intact.
• parallel: UltraQA is independent but shares the same repo. Record mode_relationship:"parallel" and keep dirty-worktree evidence in the scenario matrix.
• conflict: UltraQA would mutate the same state file, process, or artifact as another active mode. Return a structured non-fatal diagnostic with overlap_blocked:true, conflict_mode, conflict_resource, and next_route:"retry_with_child_state_or_cancel_parent".
• root: No active parent mode is relevant. Record mode_relationship:"root".

Never clear unrelated .omx/state mode files to make nested QA pass. Cleanup may clear only UltraQA state unless the user explicitly cancels another mode.

• On start: omx state write --input '{"mode":"ultraqa","active":true,"current_phase":"planning","iteration":1,"started_at":"<now>","mode_relationship":"child|parallel|conflict|root","parent_mode":"<mode-or-null>","parent_session_id":"<session-or-null>","scenario_matrix":[]}' --json
• On each cycle: omx state write --input '{"mode":"ultraqa","current_phase":"qa","iteration":<cycle>,"scenario_matrix":"<updated matrix path or summary>"}' --json
• On adversarial e2e transition: omx state write --input '{"mode":"ultraqa","current_phase":"adversarial-e2e"}' --json
• On diagnose/fix transitions: omx state write --input '{"mode":"ultraqa","current_phase":"diagnose"}' --json omx state write --input '{"mode":"ultraqa","current_phase":"fix"}' --json
• On cleanup transition: omx state write --input '{"mode":"ultraqa","current_phase":"cleanup"}' --json
• On completion: omx state write --input '{"mode":"ultraqa","active":false,"current_phase":"complete","completed_at":"<now>"}' --json
• For resume detection: omx state read --input '{"mode":"ultraqa"}' --json

Scenario Examples

Good: The user says continue after the workflow already has a clear next step. Continue the current branch of work, rerun the relevant adversarial scenario, and update the report instead of restarting discovery.

Good: The user changes only the output shape or downstream delivery step (for example make a PR). Preserve earlier non-conflicting workflow constraints and apply the update locally.

Good: A CLI prints SUCCESS while exiting 1. Mark the misleading success output scenario failed, fix the parser or reporting path, and rerun the generated harness.

Bad: The workflow runs only npm test, npm run build, npm run lint, or npm run typecheck, sees green output, and declares UltraQA complete without adversarial dynamic e2e coverage.

Bad: A generated harness leaves untracked files, state, or a child process behind and the final report omits cleanup status.

Bad: The user says continue, and the workflow restarts discovery or stops before the missing verification/evidence is gathered.

Cancellation

User can cancel with /cancel, which clears UltraQA state. Cancellation itself should be tested in cancel/resume scenarios when relevant, but UltraQA must not block an explicit user cancellation.

Important Rules

1. ADVERSARIAL E2E REQUIRED - Baseline build/lint/typecheck/test commands are necessary evidence, not sufficient completion proof.
2. SCENARIO MATRIX REQUIRED - Track normal, hostile, malformed, interruption, injection, cancel/resume, stale-state, dirty-worktree, hung-command, flaky, and misleading-output coverage.
3. GENERATE HARNESSES WHEN USEFUL - Create temporary tests or harnesses when they materially improve behavioral confidence, then clean them up or commit them intentionally.
4. PARALLEL WHEN SAFE - Run independent diagnostics while preparing potential fixes; do not parallelize commands that mutate the same state or worktree.
5. TRACK FAILURES - Record each failure to detect patterns and avoid false greens.
6. EARLY EXIT ON PATTERN - 3x same failure = stop and surface with root cause and residual risk.
7. CLEAR OUTPUT - User should always know current cycle, scenario, command, status, and evidence.
8. CLEAN UP - Clear UltraQA state and temporary artifacts on completion, cancellation, or early stop.
9. SAFETY FIRST - Never exfiltrate secrets, run destructive cleanup, write to production, or wait indefinitely to satisfy a scenario.

STATE CLEANUP ON COMPLETION

When goal is met OR max cycles reached OR exiting early, run $cancel or call:

omx state clear --input '{"mode":"ultraqa"}' --json

Use CLI state cleanup rather than deleting files directly. Also remove temporary e2e harnesses, fixtures, and logs unless they are intentional artifacts listed in the report.

---

Begin ULTRAQA cycling now. Parse the goal, build the adversarial dynamic e2e scenario matrix, and start cycle 1.