Diagnose "clean sweep" reports from scripted multi-agent review crons (Hermes `bug-hunt-daily.sh`, future siblings in the family — `wa-*-review-*`, `*-nightly-review-*`) when the report's PR/bug/agent-failure counts are all zero but the scan didn't actually happen. Triggers when a daily bug-hunt / nightly-review cron posts a Slack report with all-zero numbers, a sibling report from the same minute surfaces the real failure, a known review CLI was rate-limited / preflight-failed and the cron ran anyway, or the user pastes such a report and asks "is this right?". Classifies into FOUR buckets (PR-discovery suppressed, agent-preflight suppressed, agent-execution suppressed, fail-closed-not-armed), identifies whether the script's FAILURE_WARNING gate masked the failure, and produces the recipe for the next sibling cron. v1.1.0 (2026-07-23) — adds Bucket E for concurrent codex-review processes racing for the same model-list cache; verified on bug-hunt run 20260723_090145.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Diagnose "clean sweep" reports from scripted multi-agent review crons (Hermes `bug-hunt-daily.sh`, future siblings in the family — `wa-*-review-*`, `*-nightly-review-*`) when the report's PR/bug/agent-failure counts are all zero but the scan didn't actually happen. Triggers when a daily bug-hunt / nightly-review cron posts a Slack report with all-zero numbers, a sibling report from the same minute surfaces the real failure, a known review CLI was rate-limited / preflight-failed and the cron ran anyway, or the user pastes such a report and asks "is this right?". Classifies into FOUR buckets (PR-discovery suppressed, agent-preflight suppressed, agent-execution suppressed, fail-closed-not-armed), identifies whether the script's FAILURE_WARNING gate masked the failure, and produces the recipe for the next sibling cron. v1.1.0 (2026-07-23) — adds Bucket E for concurrent codex-review processes racing for the same model-list cache; verified on bug-hunt run 20260723_090145.
When a Hermes scripted multi-agent review cron (currently bug-hunt-daily.sh, future siblings in the same family) posts a Slack report that looks like a clean sweep (PRs reviewed=0, Bugs found=0, Agent failures=0/0), the dispatcher is staring at the script's failure-shape: every layer of the pipeline (PR discovery, agent preflight, agent execution) silently coerces its failure to "zero work done", and the FAILURE_WARNING block is gated only on agent-execution failures — not on the upstream gates. The 4-bucket classification below is the diagnostic; the recipe at the end is the durable fix.
This is distinct fromwa-daily-cron-failure-diagnosis (which covers GCP Cloud Run cron jobs for your-project.com with email alerts). This skill covers Hermes-side launchd crons with Slack alerts — different channel, different agent fleet, different failure shape. The two skills share Pitfall 5 ("trust the report's zero count") and Pitfall 8 ("re-verify against live state"); the buckets diverge.
User pastes such a report: "is this right?", "this says zero PRs — there should be some", "the daily bug hunt is broken".
Sibling-report signal: a bug-hunt-<TS>.md file from the SAME minute has a ## PR Discovery Failure section, OR a sibling has Agent failures: N/N instead of 0/0. Sibling reports at ls /tmp/hermes/bug_reports/bug-hunt-<same-minute-prefix>* are the smoking gun.
Cron ran but didn't surface what it should have: the launchd plist (launchctl print gui/$(id -u)/ai.hermes.schedule.<job>) shows last exit 0, the log is fresh, but Slack got a zero-result report.
The five failure buckets — classify BEFORE doing anything
Always classify into ONE of these five buckets before recommending a fix. The bucket determines the recipe.
Sibling report file contains ## PR Discovery Failure block with a GraphQL: API rate limit already exceeded for user ID <N> line, OR an HTTP 5xx, OR a could not resolve host error.
The "clean sweep" report's own JSON output files for all agents are ( size = 0, size = 0). Agents were never started because the script took the path on .
0 bytes
bug-hunt-<agent>-<TS>.json
bug-hunt-<agent>-<TS>.err
continue
if [ "$PR_COUNT" -eq 0 ]
gh api user confirms the rate-limited user ID matches the active token (gh api user --jq .id should equal the user ID in the rate-limit message).
Root cause:get_merged_prs "$REPO" 2>/dev/null || echo "[]" silently coerces a non-zero exit (rate limit, auth failure, network error) to []. The script has no way to distinguish "no PRs merged in the window" from "couldn't even ask".
Recipe:
Identify which token hit the rate limit (gh api user --jq .id vs the user ID in the rate-limit message — should match).
Wait for the rate-limit window to pass (GitHub GraphQL resets hourly; REST has its own counter).
Manually re-run discovery with gh pr list --state merged --search "merged:>=<since>" against each scanned repo.
The "fix now" recipe: replace the silent-coercion line with a typed status return (printf 'ERROR\t%s\t%s\n' "$repo" "$out" >&2; return 1) and stop coercing to []. The FAILURE_WARNING block must surface discovery failures too.
bug-hunt-review-preflight-<TS>.err contains ERROR: installed codex CLI lacks supported review mode or ERROR: installed codex CLI lacks explicit model selection or ERROR: codex CLI not found.
The "clean sweep" report has Agent failures: 0/0 — agents never started because REVIEW_CLI_AVAILABLE=0 took the continue path.
codex review --help 2>&1 | grep -q -- '--base' returns non-zero on this machine.
Root cause:configure_review_cli() returns 1 when the local codex CLI lacks the --base/--model flags the script requires. The script then skips the agent loop entirely, leaving AGENT_PIDS=() empty, so AGENT_FAILURES stays 0 (no agents to fail). The FAILURE_WARNING block only fires on ALL_AGENTS_FAILED (every spawned agent failed), not on "no agents spawned because preflight short-circuited".
Recipe:
Install or update codex: brew upgrade codex (or npm i -g @openai/codex).
Verify the model tier per SOUL.md ## COMMIT: subagent model routing (mandatory, 2026-07-14) — cheapest correct tier for review lanes (mini/haiku-class for pollers/monitors, mid-tier Sonnet/Codex Spark for standard review/evidence lanes).
The "fix now" recipe: change configure_review_cli to return 0 on missing CLI but still mark a typed PREFLIGHT_FAILED status; the loop should set AGENT_FAILURES=${#AGENTS[@]} when preflight fails so the warning fires.
Bucket C: Agent-execution suppressed (spawned but produced 0 bytes / errors)
Symptom markers:
JSON output files exist but are 0 bytes: bug-hunt-<agent>-<TS>.json size=0.
Error files contain codex: command not found, Error: 401, Error: 429, or timeout messages.
The "clean sweep" report has Agent failures: N/N — the warning fires correctly here, but the bug count is still 0 because all agents crashed before producing findings.
Root cause:codex review was spawned but the underlying model invocation failed (auth, quota, network). The script's validate_finding_evidence() correctly rejects empty/malformed output and counts as AGENT_FAILURES. This is the bucket the script handles correctly. The bug is upstream — quota exhaustion on the chosen tier, transient OAuth expiry, or codex binary segfaulting.
Recipe:
Read the .err files: cat /tmp/hermes/bug_reports/bug-hunt-<agent>-<TS>.err.
Identify the failure class (auth → re-login; quota → switch tier per SOUL.md subagent model routing; binary → reinstall).
Retry with a different model tier (BUG_HUNT_REVIEW_MODEL=...) or different agent fleet.
If the underlying cause is quota exhaustion on the cheapest tier, escalate per ao-spawn-minimax-worker (mid-tier fallback).
Bucket D: Fail-closed-not-armed (cron disabled / plist not loaded)
Symptom markers:
launchctl print gui/$(id -u)/ai.hermes.schedule.bug-hunt-daily shows state = not running and last exit code = 0 BUT the log mtime is hours/days old.
bug-hunt-<TS>.md files in /tmp/hermes/bug_reports/ are from older runs; no fresh timestamps.
~/.hermes/launchd/ai.hermes.schedule.bug-hunt-daily.plist exists but Disabled=true (canonical example: ai.hermes.schedule.<job>-watcher.plist pattern).
Root cause: Plist was disabled, never re-enabled after a previous test run, or KeepAlive{SuccessfulExit=false} caused launchd to remove it from the run queue. This is NOT a script bug — it's an infra bug. See dropped-thread-watcher-of-watchers for the canonical silent-cron-death pattern.
Verify next scheduled run lands and produces a real report.
Bucket E: Parallel-CLI-process race (concurrent agents hit shared resource lock)
Symptom markers:
PRs were discovered correctly (Bucket A excluded — TOTAL_PRS > 0 in the log, e.g. Found 9 merged PRs).
Preflight passed (Bucket B excluded — bug-hunt-review-preflight-<TS>.err is empty).
Per-agent output files exist and are 0 bytes: bug-hunt-<agent>-<TS>.json size=0.
bug-hunt-<agent>-<TS>.err is 0 bytes for SOME lanes but NON-empty for OTHERS — the discriminating signal. The non-empty err shows the underlying CLI's banner (OpenAI Codex v0.144.5, model: gpt-5.3-codex-spark, workdir: ...) and then a fatal line like failed to refresh available models: timeout waiting for child process to exit or codex_models_manager::manager: failed to refresh available models: ....
The "clean sweep" report has Agent failures: N/N where N == number of agents — FAILURE_WARNING fires correctly here.
ps aux | grep -E "<cli-binary>" shows multiple instances of the same CLI binary running concurrently with overlapping start times.
Root cause: The script spawns N agents in parallel as background processes from a single bash subshell, all targeting the SAME CLI binary with the SAME model. Each CLI process tries to refresh the local model-list cache on startup; the cache has a single-writer lock and the loser times out. One process wins, the others produce 0-byte output and either swallow the failure to a 0-byte .err (race winner ate the error stream) or write the banner + timeout to .err before exiting.
This was confirmed in the 2026-07-23 09:01 PT bug-hunt run: three concurrent codex review -c "model=\"gpt-5.3-codex-spark\"" - processes from bug-hunt-daily.sh:230-238. bug-hunt-claude-20260723_090145.{json,err} and bug-hunt-codex-20260723_090145.{json,err} were both 0 bytes. bug-hunt-minimax-20260723_090145.json was 0 bytes; bug-hunt-minimax-20260723_090145.err contained the actual Codex banner + codex_models_manager::manager: failed to refresh available models: timeout waiting for child process to exit. All three agents used the SAME gpt-5.3-codex-spark model — the labels claude/codex/minimax were cosmetic, all three were the same Codex CLI invocation.
Recipe:
Detect: count err-files-with-content vs total.for f in /tmp/hermes/bug_reports/bug-hunt-*-<TS>.err; do echo "$f $(stat -f%z "$f") bytes"; done. Mix of 0-byte and non-zero = Bucket E.
Stop spawning the same CLI in parallel. Two durable fixes, pick one:
Switch to per-agent CLIs (the canonical fix in PR #792): hermes -z -m <model> per agent label where each label maps to a DIFFERENT CLI (claude → Claude Code, gemini → Gemini CLI, minimax → minimax Anthropic-API shim). Each CLI has its own model-list cache, no contention.
Serialize the agents: change the bash loop from & backgrounding to wait after each spawn. Loses parallelism but eliminates the race.
Surface the race in the report. Add a per-agent stderr-capture guard that copies non-empty .err content into the ## Results section BEFORE the FAILURE_WARNING block, so a single-line codex_models_manager::manager: failed to refresh available models: timeout waiting for child process to exit shows up in Slack instead of being swallowed.
Add a regression test that monkey-patches the agent spawn to return the timeout error string and asserts the report's ## Results section contains the literal failed to refresh available models.
Related skill for the canonical fix: the durable fix lives in jleechanorg/jleechanclaw PR #792 (fix/durable-bug-hunt-harness, MERGEABLE, CodeRabbit APPROVED, blocked only by Gate 3 stale state + Gate 5 unresolved comment as of 2026-07-23). Reference: references/2026-07-23-bug-hunt-codex-review-race.md.
The 7-step investigation protocol (always run in order)
Step 1 — Read the report file the bot posted
cat /tmp/hermes/bug_reports/bug-hunt-<TS>.md
Capture: PRs reviewed, Bugs found, Agent failures. If all three are zero, suspect suppression (Buckets A/B/C/D).
Step 2 — List sibling reports from the same minute
ls -la /tmp/hermes/bug_reports/bug-hunt-<same-minute-prefix>*
If you see ≥2 reports from the same minute, the cron ran more than once and one of them likely surfaced the real failure. Sibling reports are the smoking gun — Bucket A/B always leaves a sibling that reports correctly.
for f in /tmp/hermes/bug_reports/bug-hunt-{claude,codex,gemini,minimax}-<TS>.{json,err}; doecho"=== $f ($(stat -f%z "$f" 2>/dev/null) bytes) ==="
[ -s "$f" ] && head -c 500 "$f"echodone
Empty JSON + empty err = Bucket A or D (agents never spawned). Empty JSON + non-empty err = Bucket C (agents spawned, failed). Mixed empty-JSON-and-err across agents = Bucket E (parallel-CLI-process race) — see Bucket E's symptom markers; the lane that captured the banner-then-timeout is the one that lost the model-list cache lock. Preflight .err present = Bucket B.
(Adjust <TS> and mergedAt >= "<since>" to the actual window.) This is the live ground truth; the report is the suspect. Compare counts. Mismatch = Bucket A.
Step 5 — Classify into one of the four buckets
Apply the markers from each bucket section above. Buckets can compound (e.g. Bucket A on first repo → all subsequent repos loop returns [] → no agents → clean sweep). When in doubt, start with Bucket A — it is the most common and the most silent.
Switch to per-agent CLIs (e.g. hermes -z -m <model> per label) OR serialize the spawn loop; surface the cached timeout in the report
Step 7 — Produce a named end-state
Per diagnosis-requires-followthrough-or-handoff, every diagnosis must end with one of:
"apply now" — fix is ≤10 lines, single-file, reversible (e.g. broaden FAILURE_WARNING gate). Apply inline in same turn. Reply: "Diagnosis complete. Applied the fix inline — <summary + file:line + verification>."
"dispatch now" — multi-file or needs PR (the script fix + pytest). ao spawn with a one-line brief OR bring-to-green babysit cron. Reply: "Diagnosis complete. Dispatched on branch with the recipe."
"hand off explicitly" — fix is destructive/expensive (data migration, schema, billing, prod deploy) OR user has been upset by auto-fixes. Paste exact 3-5 line shell block + offer to dispatch. Reply: "Diagnosis complete. NOT applying because . To finish: ."
Bare "Investigate" / "look at this report" (no /a, /finish, /auto) means diagnose-then-hand-off: do NOT auto-dispatch. Reply with the diagnosis + end-state ask.
Pitfalls
Pitfall 1: Trusting the "clean sweep" report's zero counts
The script's three zero counts (PRs=0, Bugs=0, Agent failures=0/0) are all three independently suppressible by upstream failures. Always run Step 2 (sibling reports) and Step 4 (re-derive ground truth) before believing any of the three. The 2026-07-22 case: zero counts on the main report, sibling at the same minute had ## PR Discovery Failure with the rate-limit message verbatim, and Step 4 found 8 merged PRs the bot should have reviewed.
Pitfall 2: Diagnosing the report's surface format instead of the script's pipeline
The user often asks "is this report formatted right?" — that's the wrong question. The format is fine; the content is suppressed. Pivot to the pipeline: PR discovery → agent preflight → agent execution → report generation. Each stage has a silent-failure path; identify which one is firing in this run.
Pitfall 3: Skipping the launchd state check (Bucket D) on first encounter
If bug-hunt-<TS>.md files are stale (no fresh timestamps in the last 24h), check the launchd plist state FIRST. No amount of script debugging helps if the cron isn't running. launchctl print gui/$(id -u)/ai.hermes.schedule.bug-hunt-daily 2>&1 | grep -E 'state|last exit' is the 5-second check.
Pitfall 4: Reporting "all three agents failed" when the actual cause was preflight
The "clean sweep" report with Agent failures: 0/0 often masks a preflight failure (Bucket B). The script's FAILURE_WARNING gate only fires on AGENT_FAILURES == ${#AGENTS[@]} (every spawned agent failed), not on REVIEW_CLI_AVAILABLE == 0 (no agents spawned because preflight short-circuited). Read the preflight .err file before declaring "agents failed".
Pitfall 5: Trusting prior-session conclusions about cron health
EA sweeps (e.g. memory/briefings/YYYY-MM-DD/<time>-ea-sweep.md) often cite "bug-hunt cron is healthy" or "no PRs to review" from brief-time telemetry. Re-derive live per Step 4. The same "session-of-record drift" bug that wa-daily-cron-failure-diagnosis Pitfall 10 catches for daily-cron telemetry applies here.
Pitfall 6: Reading the bot's Slack-token vs the user's GitHub-token
The Slack post lands under the bot's identity (XOX-B). The GitHub gh pr list rate-limit is on the user's GitHub token (XOX-P for the user, or GITHUB_TOKEN for the bot identity). When triaging rate-limit errors, the user ID in the message is the GitHub user (gh api user --jq .id), not the Slack user. They're different credentials.
Pitfall 7: Posting the diagnostic to the wrong Slack thread
Per SOUL.md slack-reply-inherit-thread-ts + slack-channel-routing-policy, the diagnostic reply belongs in the originating thread (the bug-hunt report thread) — NOT a home-channel orphan. The report itself is the durable artifact; if you reply home-channel instead of in-thread, the user will treat the home-channel post as "your reply" and re-ask in-thread. Always verify ThreadTs == <correct_ts> on the verification call (conversations.replies).
Pitfall 8: Auto-dispatching on bare "is this right?"
no-confirmation-gate: bare "is this right?" / "look at this report" / "the bug hunt is broken" are NOT dispatch commands. Only /a, /fullrun, /finish, /auto, /f, /fin bypass the confirmation gate. If the user's message is bare, diagnose and name the end-state — let the user invoke the dispatch command. Posting "Should I spawn an AO worker? Y/N" is a SOUL.md violation.
Pitfall 9: Skipping the "named end-state" — finishing with "want me to fix?"
Every diagnostic reply must end with ONE of: "apply now" / "dispatch now" / "hand off" named explicitly. A reply ending with "Want me to ...?" or "Should I ...?" is a SOUL.md no-pick-one-menus violation. The user explicitly opted into autonomous execution by building this skill family, but a structured diagnosis with a named handoff lets them pick precisely.
Pitfall 11: Treating a mixed empty/non-empty .err pattern as Bucket C only
The Bucket C markers say "empty JSON + non-empty err = Bucket C". When ALL lanes show that pattern, it's Bucket C. But when the pattern is mixed (some lanes 0-byte err, some non-empty err, ALL lanes 0-byte JSON), it's Bucket E — a parallel-CLI-process race where the cache-lock winner ate its error stream and the loser(s) got the timeout. The single signal that disambiguates Bucket C from Bucket E is which CLI is in the .err file. If every non-empty .err file shows the SAME CLI banner and the SAME model — and the script's TASK_PROMPT was sent to N distinct labels that all resolved to the same CLI/model — it's Bucket E, not C. The "fix CLI" recipe in Bucket C won't help; the fix is "stop running this CLI N times in parallel" (Bucket E recipe).
A subtler pitfall inside this one: the cosmetic agent labels (claude/codex/minimax) often DO NOT correspond to distinct CLI binaries. When the script dispatches via a single codex review -c "model=\"…\"" for all three labels, the labels are cosmetic and all three agents are the same Codex CLI. The bug-hunt run 20260723_090145 had labels claude/codex/minimax and a single banner OpenAI Codex v0.144.5 / model: gpt-5.3-codex-spark in the surviving .err. Diagnosing this as "claude and codex failed but minimax worked" is wrong — minimax didn't work either; it just produced a non-empty .err before the timeout.
Pitfall 10: Including a recipe without proof
Per SOUL.md proof-before-claim, the diagnostic reply must include raw terminal output proving the classification — sibling report contents, .err file lines, gh api output for the re-derivation. A "Bucket A" classification with no cat output of the sibling report's ## PR Discovery Failure section is unprovable and is a SOUL.md fabricated proof violation.
Reference and cross-links
references/2026-07-22-bug-hunt-rate-limit-suppression.md — session-specific evidence: the three sibling reports from 16:40:07, 16:40:09, 16:40:11 PT; the GraphQL rate-limit message verbatim; the script's lines 112-114, 137, 405-416 trace; the 8 PRs the bot failed to review; the durable-fix recipe for the script + pytest.
references/2026-07-23-bug-hunt-codex-review-race.md — session-specific evidence for Bucket E: three parallel codex review processes racing for the same model-list cache; one lane captured the banner + codex_models_manager::manager: failed to refresh available models: timeout waiting for child process to exit, the other two produced 0-byte .err; the cosmetic agent labels all resolved to the same CLI; in-flight durable fix is jleechanorg/jleechanclaw#792 (fix/durable-bug-hunt-harness).
~/.hermes/skills/worldarchitect/wa-daily-cron-failure-diagnosis/SKILL.md — sibling skill for GCP-cron failures (4-bucket classification pattern shared, but different cron family).
~/.hermes/skills/devops/gh-rate-limit-and-transient-failures/SKILL.md — the GraphQL rate-limit-and-recovery playbook that Bucket A relies on.
~/.hermes/skills/dropped-messages/SKILL.md — Bucket D detection (silent launchd death).
~/.hermes/skills/hermes-imports/dispatch-task/SKILL.md — AO worker dispatch mechanics for the "dispatch now" end-state.
~/.claude/skills/drive-pr-to-green/SKILL.md — for driving the resulting PR through to merge.
~/.claude/skills/advice/SKILL.md — for /advice second-opinion on the script fix.
~/.cursor/rules/pr-hyperlink.mdc — for the PR-hyperlink rule when reporting the missed PR list to the user.
~/.hermes/scripts/bug-hunt-daily.sh — the script under diagnosis; lines 112-114 (silent coercion), 137 (|| echo "[]"), 405-416 (FAILURE_WARNING gate).
One-line summary
Treat every "clean sweep" report as suspect until proven otherwise — sibling reports in the same minute and live gh api re-derivation are the two smoking guns. Classify into A/B/C/D/E (E = parallel-CLI-process race: mixed empty/non-empty .err across lanes, all targeting the same CLI/model), fix the right surface, name the end-state explicitly.
