multi-provider-adversarial-review

name	multi-provider-adversarial-review
description	Dispatch parallel adversarial reviews to Codex and Gemini CLIs for plans or code artifacts. Use when the AI Review Routing Policy requires two- or three-provider review — architecture-heavy, security-affecting, cross-module, or high-stakes changes.
version	1.1.1
author	Hermes Agent
license	MIT
metadata	{"hermes":{"tags":["code-review","adversarial","multi-provider","codex","gemini","quality"],"related_skills":["codex","requesting-code-review","github-issues","code-review"]}}

Multi-Provider Adversarial Review

When to Use

Per AI Review Routing Policy:

• Three-provider adversarial review is the default for non-trivial plans, code, harness, file-structure, test-suite, docs/report, skill-transfer, governance, and workflow changes. Claude/orchestrator frames and synthesizes; Codex and Gemini (or explicit substitutes when unavailable) provide independent adversarial review.
• Reduction in reviewer count or prompt depth is allowed only under the policy reduction rules: explicit user scoping, provider unavailability/quota with evidence, or purely clerical changes with an explicit waiver note. A scaled sanity-check prompt is a lighter review shape, not a gate skip.

Do not skip adversarial review merely because a change is "docs-only", "skill-only", "harness-only", or "workflow/report-only" when it is meaningful to the repo ecosystem. Scale the prompt depth instead:

• Thorough review: harness, file-structure, test-suite, policy, governance, enforcement hooks/scripts, workflow-impacting changes, and changes to this review skill itself.
• Scaled sanity-check review: low-risk transfer reports, audit refreshes, and narrative documentation with semantic repo impact.
• Clerical waiver: typo-only, formatting-only, generated timestamp-only, or mechanically regenerated artifacts with no semantic/policy/workflow impact may waive one reviewer under the policy, but must record the waiver reason.

The agent/provider that authored the change does not count as an independent adversarial reviewer. Record reviewer verdicts (APPROVE, MINOR, MAJOR), the durable artifact path committed under scripts/review/results/, the issue/PR comment URL when one exists, and the commit SHA or exact revision identifier the review evaluated. All channels that exist must be recorded; any omitted channel must be explained.

If a single provider sustains MAJOR for 3+ rounds while the other two converge on MINOR or APPROVE, surface the consensus-vs-minority decision to the user rather than auto-cycling further rounds.

Two Review Checkpoints

The user expects adversarial review at BOTH stages — not just implementation:

1. Checkpoint 1 — Plan review: before any implementation begins
2. Checkpoint 2 — Implementation review: before closing the issue / merging the PR

Do NOT skip the plan review. Do NOT defer all review to implementation.

Step 1: Prepare Review Material

Write a self-contained review prompt to a temp file. The prompt must include:

• Reviewer role and expectations (be adversarial, no rubber-stamping)
• Full context (the reviewers have zero conversation history)
• The plan or diff to review
• Specific review questions to address
• Expected output format (verdict + severity-ranked findings)

# For plan review:
cat > /tmp/review-prompt.md << 'EOF'
# Adversarial Review Request: Issue #NNNN

## Your Role
You are an independent adversarial reviewer. Find gaps, risks, missing edge cases, and flawed assumptions. Do NOT rubber-stamp.

## Context
[Full background — what exists, what happened, what triggered this work]

## The Plan
[Complete plan content or issue body]

## Review Questions — Address ALL:
1. [Specific concern]
2. [Specific concern]
...

Provide verdict: APPROVE, MINOR (proceed with notes), or MAJOR (must address).
List every finding with severity (critical/high/medium/low).
EOF

# For implementation review:
# Use git diff instead of the plan
git diff main...HEAD > /tmp/diff-for-review.txt
# Then include the diff content in the review prompt

Step 2: Dispatch Reviewers in Parallel

Use codex exec and gemini exec via background PTY processes:

# Codex review — prefer stdin for large/markdown-heavy prompts
cd /path/to/repo && codex exec -C /path/to/repo - < /path/to/repo/.planning/quick/review-prompt.md 2>&1 | tee /tmp/codex-review.txt
# Run with: terminal(command=..., pty=true, background=true, timeout=300)

# Gemini review (only if three-provider trigger met)
cd /path/to/repo && gemini exec "$(cat /path/to/repo/.planning/quick/review-prompt.md)" 2>&1 | tee /tmp/gemini-review.txt
# Run with: terminal(command=..., pty=true, background=true, timeout=300)

Then wait for both:

process(action="wait", session_id="<codex_id>", timeout=300)
process(action="wait", session_id="<gemini_id>", timeout=300)

Retrieve full output:

process(action="log", session_id="<codex_id>", limit=500)
process(action="log", session_id="<gemini_id>", limit=500)

Step 3: Consolidate Findings

Deduplicate across providers. Structure as:

## Checkpoint N: [Plan/Implementation] Adversarial Review — Codex + Gemini

### Verdicts
- **Codex**: [VERDICT] (N findings: X critical, Y high, Z medium)
- **Gemini**: [VERDICT] (N findings: ...)

### CRITICAL findings (must fix)
| # | Finding | Codex | Gemini |
|---|---------|-------|--------|
| C1 | [description] | ✓ | ✓ |

### HIGH findings (should fix before implementation)
...

### MEDIUM findings (address during implementation)
...

### LOW findings (nice to have)
...

Step 4: Post to Issue/PR

gh issue comment NNNN --body-file /tmp/consolidated-review.md

For GTM/business-critical plan-review closeout, see references/gtm-plan-review-closeout-2026-04-29.md for a worked pattern covering live review replacement of UNAVAILABLE placeholders, owner-decision packets, numeric-claim verification, and concurrent-git closeout.

Post-Review: Acting on MAJOR Verdicts

If ANY reviewer returns MAJOR at plan stage, the plan is not approval-ready and must be revised before implementation or user approval. Do not average this down because another provider returned APPROVE or MINOR. Typical pattern:

1. Deduplicate findings across providers — they often converge on the same core problems independently
2. Phase the work — both Codex and Gemini consistently recommend splitting monolith issues into 3-7 independent deliverables when scope is too large
3. Revise the parent issue body with findings incorporated, then create child issues for each phase
4. Post the consolidated review as an issue comment for traceability
5. Do NOT proceed to implementation until CRITICAL and HIGH findings are addressed in the revised plan

Common reviewer recommendations that recur across reviews:

• "Split into phases" (scope creep)
• "X is not a real health/smoke check" (inadequate verification)
• "Rollback is underspecified" (no atomic model)
• "Windows/cross-platform not addressed" (Linux-centric thinking)
• "Logging is not alerting" (passive vs active failure reporting)

Retroactive Review Pattern

When commits were pushed without review (e.g., overnight batch runs), dispatch retroactive reviews. This was proven end-to-end on 2026-04-02 (40 commits, 0 reviews → 3 MAJOR verdicts, 27 findings, 9 follow-up issues):

1. Audit: git log --oneline --since="..." | grep -vE '^(docs|chore|test|ci|style)' to find unreviewed feature/fix commits
2. Group by work stream: cluster commits by issue number into 2-4 review batches
3. Embed code in prompts: Read files via terminal("cat ...") (NOT read_file which may cache). Truncate to ~20K chars per prompt. Codex sandbox CANNOT read mounted volumes.
4. Write prompts to workspace: Use terminal("uv run python -c \"...open().write()...\"") to write prompt files to the repo dir where $(cat .planning/quick/review-X.md) works in real shell.
5. Dispatch parallel: codex exec "$(cat .planning/quick/review-X.md)" via terminal(background=true, pty=true)
6. Consolidate: Save to scripts/review/results/TIMESTAMP-retroactive-review-codex.md with tabular findings
7. Create follow-up issues: One issue per CRITICAL/HIGH finding (create labels first!)
8. Comment on parent: Link all follow-ups from the parent issue

This catches real bugs even after the fact — the 2026-04-02 retroactive review found shell injection, race conditions, schema mismatches, and ToS compliance gaps across solver queue and GTM scanner code.

Writing Prompt Files for Codex

The /tmp/ trap: Hermes execute_code and write_file write to a sandbox overlay — NOT the real filesystem. So $(cat /tmp/review-prompt.md) in a terminal() call will fail with "No such file or directory" because the file only exists in the sandbox.

The workspace overlay trap: Even write_file or execute_code's write_file() targeting the workspace mount (e.g., /mnt/local-analysis/workspace-hub/.planning/quick/file.md) goes to sandbox overlay on mounted volumes.

The fix: Write prompt files via terminal() using Python:

terminal("cd /mnt/local-analysis/workspace-hub && uv run python -c \"
content = '''... your prompt ...'''
with open('.planning/quick/review-prompt.md', 'w') as f:
    f.write(content)
\"")

Then dispatch: codex exec "$(cat .planning/quick/review-prompt.md)"

Alternatively, for short prompts, embed code content directly in the $(cat) heredoc — but beware shell metacharacters in code will break heredocs. The Python open().write() approach is most robust.

Pitfalls

1. Codex sandbox blocks file reads — Codex exec runs in a bwrap sandbox that may block filesystem access. Pass ALL context in the prompt text itself, not via file references. The prompt must be fully self-contained.

2. Argument-size limits on giant inline prompts — very large review prompts can fail before the provider even starts with shell errors like Argument list too long when you do codex exec "$(cat prompt.md)" or gemini exec "$(cat prompt.md)".

   • Symptom: the shell fails immediately; no provider verdict is produced.
   • Fix: write a compact review prompt containing only the essential context, exact artifact under review, specific questions, and required output format.
   • Keep the full context in a separate workspace file if needed, but do not force the entire issue history/diff corpus into argv.
   • Save the compact prompt as its own artifact (for example .planning/quick/review-<issue>-implementation-compact-prompt.md) so the recovery path is reproducible.
   • Prefer compact self-contained prompts over retrying the same oversized command.
   • Codex-specific recovery: if codex exec "$(cat prompt.md)" exits 0 but the tee/raw artifact is empty or contains no verdict, treat it as a failed dispatch, not a successful review. Retry with stdin: codex exec -C /path/to/repo - < /path/to/repo/.planning/quick/review-prompt.md 2>&1 | tee ....
   • Always validate the raw artifact after each provider run by checking both non-zero length and a verdict/findings marker; process exit code alone is insufficient.

3. Gemini capacity limits — gemini-3.1-pro-preview can hit 429 MODEL_CAPACITY_EXHAUSTED errors. Gemini CLI retries automatically but may take longer. Allow extra timeout.

   • In large parallel review waves, Gemini may fail repeatedly and never produce a usable verdict.
   • Treat that as missing provider evidence, not as approval or as a silent pass.
   • Continue with Codex (and any existing Claude/Hermes evidence), but post a GitHub comment explicitly noting that Gemini re-review was blocked by provider capacity exhaustion.
   • Do not mark the plan fully cross-reviewed if the Gemini artifact is only a 429/capacity log with no substantive verdict.
   • If the remaining provider returns MAJOR, keep the issue in status:plan-review and proceed with revision work instead of waiting indefinitely for Gemini capacity to recover.

• docs/plans/README.md
• scripts/review/results/
• .planning/plan-approved/*.md
• gh issue view <n> --json labels,state,... Treat missing Codex/Gemini artifacts as pending cross-review even when GitHub labels suggest a more advanced state.

19. Provider CLI warnings can be non-fatal; distinguish startup noise from failed review output — In live plan-review runs on 2026-04-14:

• Codex emitted a startup warning about a missing .claude/skills/skills symlink.
• Gemini emitted agent-loading warnings about .gemini/agents/*.md containing unsupported permissionMode keys. Despite these warnings, both CLIs still produced valid review content. Do not treat these warnings alone as review failure. Confirm success by reading the tee'd output file and checking for a complete verdict/findings block before deciding whether to retry.

20. For one-by-one plan review waves, always materialize three artifacts per issue — The reliable pattern is:

• prompt file: .planning/quick/review-<issue>-prompt.md
• raw provider logs: .planning/quick/review-<issue>-codex.out and .planning/quick/review-<issue>-gemini.out
• canonical saved reviews: scripts/review/results/YYYY-MM-DD-plan-<issue>-codex.md and ...-gemini.md Then post a concise GitHub issue comment summarizing verdicts, shared blockers, provider-specific emphasis, and artifact paths. This keeps raw CLI noise separate from the durable review artifact and makes later governance audits much easier.

21. When a provider is unavailable, still save an explicit review artifact instead of leaving the slot blank — In live use on 2026-04-15, Gemini repeatedly returned 429 RESOURCE_EXHAUSTED / MODEL_CAPACITY_EXHAUSTED for plan reviews. The reliable pattern is:

• save the successful reviewer artifact(s) normally
• save a provider-specific scripts/review/results/YYYY-MM-DD-plan-<issue>-gemini.md artifact with Verdict: UNAVAILABLE
• include the concrete failure reason (for example model capacity exhaustion) and point to the raw CLI log path
• treat the plan review wave as incomplete unless repo policy explicitly allows reduced-provider review for that run This avoids ambiguous "pending" review state, preserves evidence for governance audits, and makes it clear the blocker was provider availability rather than missing execution.

22. Claude CLI review can fail silently or exhaust turns — claude -p may time out with an empty tee file, or exit with Error: Reached max turns before producing a usable verdict. Treat both as failed dispatches, not review evidence. Retry with a compact prompt that lists exact files, known review state, required output format, and owner-decision questions; increase --max-turns enough for the review. Save only the successful substantive output as the canonical Claude artifact.

23. Patched-after-MAJOR plans are not automatically approval-ready — If provider artifacts still record MAJOR but the main session patches the plan afterward, keep the local plan status conservative (draft / patched-after-review) until one of these happens: (a) a fresh provider re-review returns no blocking findings, or (b) the review summary explicitly distinguishes historical provider MAJOR artifacts from main-session inline/r3 patches and names any remaining durability blocker. Do not apply status:plan-review while the plan/review artifacts are still untracked or unpushed; first commit/push the durable plan + review artifacts, then post the GitHub summary/comment and only then transition from status:needs-plan to status:plan-review. This prevents claiming cross-review closure from stale MAJOR artifacts or non-durable local files.

24. GTM numeric claims need source-file calculation, not prose review — For brochure/outreach plan reviews, force at least one lane to compute headline numbers from source JSON/report files. A 2026-04-29 review caught a 108 cases caption that should have been 156 by summing the matrix; text-only reviewers had missed it. Mark each number as either verified-now or render-time recompute-required.

25. Concurrent background agents can make broad git status/add unusable — In active workspace-hub sessions, global git status or broad git add -A can hang behind other agents/VS Code git operations and can stage unrelated work. For review closeout, use scoped verification and staging: git diff -- <target-files>, git ls-files <target-files>, git add <target-files>, and git diff --cached --name-only. Commit only the intended review artifacts and plan patches.

Post-Review: Batch Follow-Up Issue Creation from Findings

When review findings produce multiple follow-up issues (common with retroactive reviews across multiple streams), create them efficiently:

1. Create labels first — gh issue create silently fails if ANY label doesn't exist. Check gh label list | grep <name> and create missing labels with gh label create "<name>" --description "..." --color "<hex>" BEFORE creating issues.
2. Write body files via terminal("uv run python -c '...open().write()...'") — avoids both sandbox overlay and shell escaping issues.
3. Loop in execute_code — create all issues in one script, collecting URLs.
4. Comment on parent issue — link all child issues with a consolidated summary using gh issue comment <parent> --body-file.

Post-Review: Creating Phased Child Issues

When the revised plan splits into phases, create child issues efficiently using execute_code with a loop rather than manual gh issue create calls. Each child issue should:

• Reference the parent issue number in the title (e.g., Phase 1: ... (#1668))
• Include ## Parent: #NNNN as the first line of the body
• Use --body-file /dev/stdin << 'BODY' ... BODY heredoc pattern to avoid shell escaping issues
• Share the same labels as the parent
• Have independent acceptance criteria that don't depend on other phases

After creating all child issues, update the parent body to link them (replace #PENDING_N placeholders with actual issue numbers), then use gh issue edit --body-file.

Consolidating Review into Issue Revision

The full workflow after MAJOR verdicts is:

1. Write consolidated review findings as issue comment (for traceability)
2. Write raw reviewer output as a second comment (in <details> blocks)
3. Revise the parent issue body — incorporate all CRITICAL/HIGH findings
4. Create phased child issues
5. Update parent body with child issue links
6. Comment noting the review gate policy (both checkpoints)

Do all issue body edits via write_file to temp path + gh issue edit --body-file. Never try to pass complex markdown through shell arguments.

Three-Provider Trigger Checklist

Add Gemini when ANY apply:

• Architecture-heavy change (cross-module/cross-repo structural)
• Research-heavy task (synthesizing external sources)
• Ambiguous requirements (third interpretation reduces risk)
• High-stakes delivery (production, security, data integrity)
• Context saturation (Claude's context is full)

Skip Gemini for: routine implementation, standard refactors, test additions, docs-only changes.

Provider-unavailable artifact rule

If a required provider cannot complete review (for example Gemini returns repeated 429 RESOURCE_EXHAUSTED / MODEL_CAPACITY_EXHAUSTED), do NOT leave the review slot missing.

Always write a canonical artifact file anyway, for example:

• scripts/review/results/YYYY-MM-DD-plan-<issue>-gemini.md
• scripts/review/results/YYYY-MM-DD-implementation-<issue>-gemini.md

That placeholder artifact should record:

• reviewer/provider name
• timestamp
• verdict: UNAVAILABLE or equivalent explicit status
• concrete failure reason (capacity, auth, CLI crash, etc.)
• whether startup warnings were non-fatal
• path to raw CLI output/log if captured
• operational decision taken (retry later, proceed with reduced-provider review, or block approval)

Why this matters:

• governance audits can distinguish "provider unavailable" from "review never attempted"
• plan metadata stays truthful when it references expected artifact paths
• later reruns can replace a documented placeholder rather than reconstructing what happened from chat history