Diagnose daily cron FAILs for Your Project (Dice Audit, Level Up Test, and any future GCP cron job in the `wa-*-daily-*` family). Triggers when a `[GCP Cron] ... - FAIL (...-YYYY-MM-DD)` email arrives, when the `worldai:daily-level-up-and-dice-test-watcher-12h` cron (`8ccfba727015`) posts a FAIL to Slack `#worldai`, when the user says "Daily Dice Audit failed again" / "the daily cron is failing" / "look at my email and fix using /a", or when an AO worker receives a `fix the GCP cron` brief without further context. Classifies failures into FOUR buckets (infra, audit assertion, data-class regression, watcher pipeline), identifies the matching PR fix surface (existing-open vs needs-reopen vs needs-new), and produces a named end-state per `diagnosis-requires-followthrough-or-handoff`. v1.0.0 (2026-07-09) captures the 4-bucket classification, the "PR
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Diagnose daily cron FAILs for Your Project (Dice Audit, Level Up Test, and any future GCP cron job in the `wa-*-daily-*` family). Triggers when a `[GCP Cron] ... - FAIL (...-YYYY-MM-DD)` email arrives, when the `worldai:daily-level-up-and-dice-test-watcher-12h` cron (`8ccfba727015`) posts a FAIL to Slack `#worldai`, when the user says "Daily Dice Audit failed again" / "the daily cron is failing" / "look at my email and fix using /a", or when an AO worker receives a `fix the GCP cron` brief without further context. Classifies failures into FOUR buckets (infra, audit assertion, data-class regression, watcher pipeline), identifies the matching PR fix surface (existing-open vs needs-reopen vs needs-new), and produces a named end-state per `diagnosis-requires-followthrough-or-handoff`. v1.0.0 (2026-07-09) captures the 4-bucket classification, the "PR
When a Your Project daily GCP cron job (wa-daily-dice-audit, wa-daily-level-up-test, or a sibling) produces an exit=1 email, the dispatcher is staring at a recurring multi-class failure surface that has already consumed 5+ consecutive days of debugging cycles across multiple sessions. This skill encodes the fast-path triage: classify into one of four buckets, identify the right PR fix surface, and produce a named end-state.
When this skill fires
Email arrives:[GCP Cron] Daily Dice Audit - FAIL (daily-dice-audit-YYYY-MM-DD) (subject format [GCP Cron] <Job Name> - FAIL (<work-name>-<date>)) from $USER@gmail.com.
Slack alert fires::rotating_light: <U09GH5BR3QU> Daily Dice Audit: FAIL (work=daily-dice-audit-YYYY-MM-DD, exit=1) in #worldai (C0AH3RY3DK6), posted by the worldai:daily-level-up-and-dice-test-watcher-12h cron (8ccfba727015).
User requests: "Daily Dice Audit failed", "the daily cron is failing", "look at my email and fix using /a", "dice audit cron FAIL again", or any class-level equivalent. Bare "Investigate" (no slash trigger) is in-scope — diagnose, name the end-state, do not auto-dispatch.
AO worker brief contains "fix GCP cron" / "green the daily dice audit" / "daily-level-up-test failing" without further context — load this skill FIRST before reading the brief.
The four failure buckets — classify BEFORE doing anything
Always classify the failure into ONE of these four buckets before recommending a fix. The bucket determines the right PR surface.
Bucket A: Infra (cron job itself broke)
Symptom markers (look for these in the email body):
Container called exit(2) before any audit logs
Default STARTUP TCP probe failed
OOM, SIGKILL, Worker (pid:NNN) was sent SIGKILL
Error: Could not create cloud SQL proxy, missing service-account env var
ModuleNotFoundError, ImportError before audit script runs
Cloud Run revision logs show the container never started, not the audit output
Fix surface: Docker / Dockerfile / Cloud Run job config. NOT in scripts/audit_dice_rolls.py. Reference skill for the canonical infra-debug protocol.
wa-cloud-run-deploy-failure-debug
Open PRs in this bucket: None as of 2026-07-09 (most infra PRs already merged).
Audit summary block includes [Integrity Failure] lines AND [Ignored Warning] lines (mixed severity)
Sub-buckets (further classify):
B1: Same campaign, same failure shape as yesterday — PR #7605, #7812, #7729, #7607 already shipped benign-skip / brand-new-skip / watermark-empty fixes. The campaign is now stale-data the audit can't triage. Fix surface: backfill or campaign skip.
B2: Different campaign, same failure shape as prior weeks — The audit heuristic is buggy regardless of campaign. Fix surface: open new PR hardening the audit script.
B3: Different campaign, NEW failure shape never seen before — The audit script's regex/parser can't handle a NEW class of dice notation the LLM started emitting. This is the 2026-07-09 case (xK3fp5XrV24oarIINTF7 / Visenya v7 with concatenated 1d8+3 + 2d6 notation). Fix surface: reopen the structural fix that was merged in pieces but never completed.
Same campaign as yesterday, but DIFFERENT failure shape (e.g. yesterday was 1d20=22 impossible values, today is 1d20+10+1d15 concatenated notation on the same campaign)
New notation types appearing in dice_audit_events that weren't there before
The number of integrity failures per campaign is INCREASING across consecutive days
Recent prompt-schema-merge PRs in $PROJECT_ROOT/prompts/shared/ (especially dice_notation_contract.md)
Fix surface: Re-examine the prompt contract — LLM may have started emitting a NEW notation style after a contract clarification. Two-pronged:
Audit script: handle the new notation (backwards compat)
Prompt contract: anti-examples for the new notation (forward-looking)
Gateway ~/.hermes/logs/gateway.error.log shows Slack API error: message_limit_exceeded around the time of the cron fire
Fix surface: Launcher / cron-scheduler / Slack channel-routing. NOT in audit script. Reference dropped-messages and slack-thread-routing-investigation skills for the canonical recovery.
The 7-step investigation protocol (always run in order)
Step 1 — Verify the watcher fired (or didn't)
python3 -c "
import json
d = json.load(open('$HOME/.hermes/cron/jobs.json'))
for j in d.get('jobs', []):
if 'dice' in j.get('name','').lower() or 'wa-daily' in j.get('name','').lower():
print(f\"id={j.get('id')} last_run={j.get('last_run_at')} next_run={j.get('next_run_at')} enabled={j.get('enabled')} repeat={j.get('repeat')}\")"
Also check the marker file:
ls -la ~/.cache/wa_daily_test_watcher/dice/<TODAY>.posted 2>/dev/null && cat ~/.cache/wa_daily_test_watcher/dice/<TODAY>.posted
If the watcher didn't fire, you're in Bucket D. Stop; route to watcher-pipeline fixes.
If the watcher fired and wrote FAIL to the marker — go to Step 2.
Read the email body's === Results: block and the === Log Tail (last 80 lines) block. Apply the bucket markers above. Most failures are Bucket B — within Bucket B, classify into B1/B2/B3.
Step 4 — Campaign-fingerprint the failure
Use gh pr list --search "<campaign_id>" to find which PRs already touched this campaign. Pattern:
gh pr list --repo $GITHUB_REPOSITORY --state all \
--search "<campaign_id OR campaign-name>" --json number,title,state,mergedAt,url
This answer has three parts:
Which PRs already addressed this campaign (and whether they merged or closed) — tells you what's already been tried.
Whether the structural fix is open/merged/closed — for the dice audit family, this is PR #7695 (fix/dice-audit-modifier-guard). As of 2026-07-09 it is CLOSED not merged, which is the most-likely cause of any new B3-class failure.
Adjacent open PRs that could absorb this fix — e.g. #7873 (cron-exit-semantics-and-oom-watchdog) for infra-vs-assertion separation, #7596 (emailer-scenario-loader) for the email render bug.
Step 5 — Identify the right fix surface
Bucket
Right fix surface
A
Dockerfile / Cloud Run config — see wa-cloud-run-deploy-failure-debug
B1
New PR: campaign-skip / backfill (~30 lines)
B2
New PR: hard-code the auditor's regex to reject the bad input before chi-square
When B3 hits and PR #7695 was closed without merging, the fix recipe is: open a NEW PR off origin/main with the structural audit refactor + ship a smaller parallel PR for the tactical stop-gap. Do NOT rebase #7695 onto main — recreate the branch from origin/main, cherry-pick the valuable commits, drop the stale parts, push.
Step 6 — Produce a named end-state
Per diagnosis-requires-followthrough-or-handoff, every diagnosis must end with one of these explicit names:
"apply now" — fix is ≤10 lines, single-file, reversible, NOT in audit script (e.g. add a campaign-skip rule). Apply inline in same turn. Reply: "Diagnosis complete. Applied the fix inline — <summary + file:line + verification>."
"dispatch now" — multi-file or >10 lines or needs PR (the B3 reopen case, or any B2). ao spawn with a one-line brief OR bring-to-green babysit cron. Reply: "Diagnosis complete. Dispatched on branch with the recipe."
"hand off explicitly" — fix is destructive/expensive (data migration, schema, billing, prod deploy) OR user has been upset by auto-fixes. Paste exact 3-5 line shell block + offer to dispatch. Reply: "Diagnosis complete. NOT applying because . To finish: ."
Bare "Investigate" / "look at this email" (no /a, /finish, /auto) means diagnose-then-hand-off: do NOT auto-dispatch. Reply with the diagnosis + end-state ask: "Reply /a to spawn AO worker on the recipe; apply now to do the inline fix; hand off to paste the exact shell block for you to run."
Step 7 — Always create a status-cron
Per one-time-status-cron-after-every-task, after posting the diagnosis reply, immediately create exactly ONE one-time hermes cron at +20m targeting the same Slack thread:
CRITICAL: --at 20m (one-shot, fires once); --delete-after-run; NEVER --every (recurring spams the thread). Include the cron job ID in the reply.
Pitfalls (do NOT do these)
Pitfall 1: Treating today's FAIL as the same bucket as yesterday's
Bucket B1 and Bucket B3 look similar in the email log tail — both show [Integrity Failure] lines and [Integrity Failure] Campaign ... has unparseable dice notation. The discriminator is the campaign ID and shape: if it's a NEW campaign or NEW notation shape, you're in B3 (structural), not B1 (data stale). Misclassifying B3 as B1 means shipping another partial fix that closes without addressing the parser.
Detection: in the email body, look at === Results: 0/N passed — if the failing campaign ID is NEW (search gh pr list --search "<campaign_id>" and get ≤1 prior result), it's B3.
Pitfall 2: Assuming PR #7695 will get reopened automatically
PR #7695 (fix/dice-audit-modifier-guard) has been the structural answer since June 2026 but was CLOSED without merging on every reopen cycle. The fix recipe does NOT include "go look at #7695" — it includes "open a NEW structural PR off origin/main with the disjoint-union of #7695's commits + tactical stop-gaps." Searching for "#7695" surfaces it; expecting it to be actionable is the trap.
Pitfall 3: Mistaking a Deploy FAIL for an Audit FAIL
If the GCP cron email shows Container called exit(2) or OOM BEFORE any audit-script output, it's Bucket A (infra). The audit script never ran. Don't waste cycles reading scripts/audit_dice_rolls.py. Hand off to wa-cloud-run-deploy-failure-debug and ask whether Dockerfile / requirements.txt / Cloud Run job memory was changed recently.
Pitfall 4: Dispatching on bare "Investigate" without an explicit trigger
no-confirmation-gate: bare "Investigate" / "look at my email" / "fix this" are NOT dispatch commands. Only /a, /fullrun, /finish, /auto, /f, /fin bypass the confirmation gate. If the user's message is bare, diagnose and name the end-state — let the user invoke the dispatch command. Posting "Should I spawn an AO worker? Y/N" is a SOUL.md violation.
Pitfall 5: Skipping the "named end-state" — finishing with "want me to fix?"
Every diagnostic reply must end with ONE of: "apply now" / "dispatch now" / "hand off" named explicitly. A reply ending with "Want me to ...?" or "Should I ...?" is a SOUL.md no-pick-one-menus violation. The user explicitly opted into autonomous execution by building this skill family, but a structured diagnosis with a named handoff lets them pick precisely.
Pitfall 6: Bundling Bucket B + Bucket C fixes into one PR
If you're in Bucket C (data-class regression — LLM is emitting a new notation shape), the fix is two PRs in parallel: (1) audit-script backwards-compat (treat new notation as parseable + add to integration test), (2) dice_notation_contract.md anti-examples (forward-looking prevention). Bundling them into one PR forces the reviewer to accept both or neither and stalls on review.
Pitfall 7: Claiming the cron "passed" because no GCP email arrived
The cron can FAIL with exit=1 and the email can be silently dropped (Gmail quota, sandbox block, etc.). Always check ~/.cache/wa_daily_test_watcher/<job>/<TODAY>.posted AND gog gmail search 'subject:"[GCP Cron] <Job>" newer_than:3d'. If the marker file is missing but the email exists, the Slack-alert pipeline died silently. If both are missing, the GCP cron job itself died (Bucket A).
Pitfall 8: Trusting prior-session PR state instead of re-verifying live (added 2026-07-14)
When a prior session's diagnosis recorded mergeStateStatus: CLEAN, mergeable: MERGEABLE, CodeRabbit APPROVED, the next session MUST re-verify those values via REST (curl -fsS -H "Authorization: Bearer $(gh auth token)" https://api.github.com/repos/<owner>/<repo>/pulls/<N>) — NOT inherit them as truth. Concrete failures observed 2026-07-14 on PR #8290: the prior session on 2026-07-11 recorded mergeable: MERGEABLE, mergeStateStatus: CLEAN on head 7822304264b. By 2026-07-14T20:19Z, the actual state was state: OPEN, mergeable: False, merge_state_status: None, head: f81c860e (new push invalidated the prior head), with the underlying cause being main drift — main HEAD advanced from 658445f2 (the PR base) to 69282e01 over the week the PR was open, putting the PR 197 commits behind origin/main. Additionally, the evidence bundle authored at 7822304264b became stale when the new head f81c860e was pushed (Evidence Staleness Tolerance rule requires the bundle be re-issued). The symptom was that 3 prior investigation reports all claimed #8290 was "ready to merge" but the live state was 1-fail/20-pass/9-pending with mergeable: False and a stale evidence gate. Generalizes: never inherit mergeable, mergeStateStatus, headRefOid, check counts, OR verdict-line text from a prior session's claim — always curl the API and parse the response in the current turn. Treat prior-session conclusions as leads, not facts. This rule is universal across PR-fix surfaces, not specific to the daily-cron path.
Pitfall 10: Trusting an EA-sweep telemetry number instead of querying live (added 2026-07-21)
EA sweeps (memory/briefings/YYYY-MM-DD/<time>-ea-sweep.md) capture telemetry at brief-time (e.g. 12:00 PT) and reference it again at 16:00 PT. The numbers in the sweep are not the numbers the user sees now. Real instance 2026-07-21: sweep reported is_test IS NULL Gemini rows went 134 → 161 in 24h and was still "firing daily"; the live bq query on worldarchitecture-ai.llm_forensics.llm_payloads for the same 24h window returned 0 new NULL rows (1372 Gemini rows, 0/1372 = 0.00%), and is_test populated climbed 67.54% → 82.10% → 92.23% → 93.36% over 4 days. The sweep's "still firing daily" was the watcher's alert-history-count (stale burst from 2026-07-20 17:00 UTC), not live 24h data.
When an EA-sweep BQ/telemetry number drives a Dice Audit action item:
Run the live query the sweep referenced: bq query --project_id=worldarchitecture-ai --use_legacy_sql=false "<the sweep's SQL>".
Report the live count alongside the sweep's count, named explicitly: "Sweep reported X (12:00 PT); live 24h count is Y (16:30 PT)."
If the live number is healthier than the sweep, say so — do not surface the sweep number as the action item without the live qualifier.
This complements Pitfall 8 (PR-state staleness) — both are "session-of-record drift" bugs where a prior snapshot was re-reported as current state. The fix is the same shape: re-derive in the current turn from the canonical source (REST API for PR state; BQ for telemetry; df -h for disk), and label any prior number as historical.
Pitfall 9: Posting the diagnostic via MCP without probing the bot-token scope first (added 2026-07-14)
When the slack-mcp conversations_add_message API returns not_in_channel (Failure 5f per slack-thread-routing-investigation), do NOT stall on the error. Probe auth.test with HERMES_SLACK_BOT_TOKEN from ~/.bashrc and SLACK_USER_TOKEN from ~/.profile (grep '^export SLACK_USER_TOKEN=' ~/.profile | sed 's/^export SLACK_USER_TOKEN=//;s/"//g'), then conversations.info?channel=<chan> with both tokens — the one that returns is_member: true is the working path. In practice on jleechanorg/* channels, the XOX-P user token (~/.profile) is the working path; the XOX-B bot token is workspace-scoped and fails. Then post via Path B curl with the user token, include an identity-disclosure note in the body, and verify via conversations_replies. The full sub-class 5f recipe lives in slack-thread-routing-investigation Failure 5f — but the daily-cron-specific lesson is: the diagnostic reply IS the durable artifact; if you post the diagnosis as a home-channel orphan instead of in-thread, the user will treat the home-channel post as "your reply" and re-ask in-thread. Always verify ThreadTs == correct_ts on the verification call.
Reference and cross-links
references/2026-07-09-dice-audit-failure-buckets.md — session-specific evidence: the GCP cron email body, the four corrupted dice sequences, the d6/d8 impossible value table, and the campaign fingerprint for xK3fp5XrV24oarIINTF7 (Visenya v7).
references/2026-07-14-pr-8290-stale-state-[REDACTED_SLACK_TOKEN] — session-specific evidence: 5th-consecutive-day Level Up FAIL (4/8) + Dice Audit (1/2), PR #8290 live state re-verified (head f81c860e, base 658445f2, main HEAD 69282e01, Evidence Gate FAIL run 29356796837 step "Validate evidence bundle structure"); Pitfall 8 (never trust prior-session PR state) and Pitfall 9 (XOX-P fallback when bot token has no channel scope) — both operationalized here. 2 NEW failure classes (POST_FINISH_FALLBACK_CHOICES + claudem_review 300s timeout) not in #8290; Visenya V8 (8Q3ipgQIxRs2YvK1flng) dice failure linked to PR #8398 /repro evidence.
references/2026-07-20-sariel-compound-notation.md — NEW — Sariel Valyria × 2 (EROaUnSbmDhqBedTbJMg + Cg2m2TkGFFez7XBynEah) FAILED 0/2 with the same sequence numbers (96, 296, 486, 488, 832), proving the upstream is the LLM's compound notation class (1d6+8+2d6+2d8, 5x 1d20+8, 54d6+2d8+24, Contested Insight (Tully 25 vs ...)), not the audit script's regex alone. Captures the two-defect coupling that today's FAIL surfaces: (D1) _is_single_die / dice_pattern rejects compound notation → emits "unparseable dice notation" warnings; (D2) when individual_rolls[] is missing on a modifier-bearing d20, _bucket_d20_from_structured at audit_dice_rolls.py:613-619 and the chi-square path at :1283-1299 fall back to using the modified total as the face → emits "impossible values" with samples like [25, 27, 37] / [9, 13, 216] / [6]. The d4 impossible=6 is the same defect — modifier-bearing d6/d8/d20 totals landing in the d4 bucket. Generalizes: any time "unparseable notation" and "impossible values" both appear in the same audit run, suspect this coupled defect, not two separate bucket failures. Includes the GCS-evidence fetch recipe (%2F URL-encoded path under download/storage/v1/b/.../o/<encoded-name>?alt=media), the campaign-fingerprint proof that same-sequence-numbers → same-upstream-cause, and the concrete fix shape (compound-notation expander + skip-not-bucket on modifier totals). Pairs with hermes-deploy-pipeline for the orphaned-plist aspect that let this 12:00 UTC FAIL sit 9h silent.
~/.hermes/skills/worldarchitect/wa-cloud-run-deploy-failure-debug/SKILL.md — Bucket A (infra failures) and the Cloud Run revision log discipline.
~/.hermes/skills/software-development/convergent-bug-triage/SKILL.md — sibling-investigation pattern when 3+ issues pile up on one campaign in 24h (a different but adjacent class).
~/.hermes/skills/hermes-imports/dispatch-task/SKILL.md — AO worker dispatch mechanics for the "dispatch now" end-state.
~/.claude/skills/drive-pr-to-green/SKILL.md — for driving the resulting PR through to merge.
~/.claude/skills/advice/SKILL.md — for /advice second-opinion on Bucket B3 reopen PRs (the recipe has a history of failing review).
~/.cursor/rules/pr-hyperlink.mdc — for the PR-hyperlink rule when reporting PR list to the user.
Your Project repo CLAUDE.md "Merge safety" section — MERGE APPROVED gate on $GITHUB_REPOSITORY PRs.
One-line summary
Classify into one of four buckets, identify whether PR #7695-class structural fix exists and was merged (or was closed), name the end-state explicitly, do not auto-dispatch on bare "Investigate" — produce diagnosis + named handoff.
