Menu
Use when classifying a stuck, failed, empty, or zero-scoring evaluation as infrastructure or policy failure.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Use when improving a Coworld player through one diagnosed failure and completed comparison evidence.
Use when compiling, validating, and uploading a notsus change as a new Crewrift policy version.
Use when decoding a Crewrift replay for authoritative deaths, survival, movement, roles, tasks, or rewards.
Use when verifying that the notsus Bedrock vote advisor fires before an advisor-sensitive evaluation.
Use when finding and statistically confirming a reproducible Crewrift crux before changing the policy.
Use before evaluations or after roster churn to resolve live policy versions and restore the champion flag.
| name | diagnose-stuck-or-failed-run |
| description | Use when classifying a stuck, failed, empty, or zero-scoring evaluation as infrastructure or policy failure. |
The reflex when a run looks dead is to blame the policy and revert, or to assume a crash and resubmit. Both are usually wrong. The vast majority of "dead run" signals are infra, not the policy. This recipe classifies the failure first.
Golden rule (evidence discipline): only a TERMINAL, completed round with real per-player scores is policy evidence. Pending / submitted / claimed / running rounds, empty logs, and rounds that fail before a player process starts are non-evidence. Never replace a champion or revert a policy from non-terminal or failed-before-startup rounds alone.
Match your situation to a branch, then run that branch's checks:
| Symptom | Branch |
|---|---|
| Detached/background eval logfile is empty or stuck; "is it dead?" | A — Empty/buffered log |
| A submit printed a JSON decode error or "failed" — about to resubmit | B — uv preamble corrupts JSON |
Round stuck pending / running forever; hosted dispatch not picking up | C — Backend / dispatch |
Empty division leaderboard, or coworld results crashed | D — Empty leaderboard trap |
Round reached terminal failed, or completed with zero/garbage scores | E — Failed/zero terminal round |
A 422 on submit, or fields the server rejects | F — API shape drift |
Don't skip ahead. Each branch ends with a classification (INFRA or POLICY) and a success check.
A long-running Python eval launched detached (nohup, background, output redirected
to a file) block-buffers stdout: the logfile stays empty until the process exits,
even while the run is healthy and working. An empty log is NOT failure evidence.
Before declaring it crashed, check liveness and side-effects (in order, cheapest first):
Process is alive:
ps -p <PID>
Alive → it is buffering, not dead. Stop here unless other signals contradict.
Side-effect artifacts exist: the eval writes a run directory as soon as the
submit succeeds and the poll loop starts. A created run dir = submit succeeded and
the loop is running. Check the run-dir tree (per-xreq folder with episode_NN/
subfolders + a summary.json for the antfarm_run.py artifact path).
Live docker container count (local docker path only):
docker ps -q | wc -l
A non-zero, structured count means games are running. (Session-derived example: 54 containers = 6 concurrent games × 9 containers each.) (session-derived, unverified)
Query the server for the request's status directly rather than trusting the log — poll the experience-request / round status via the API (see Branch C for the status check). The server is the source of truth, not the buffered file.
The fix (so this never recurs): force unbuffered stdout in the eval/diagnostic
script — PYTHONUNBUFFERED=1, or python -u, or
sys.stdout.reconfigure(line_buffering=True). Apply this even to throwaway diagnostic
scripts; the same mistake recurs there. (session-derived, unverified)
Long-running watches: run a continuous watch loop via the Monitor tool with an
until-loop, NOT a detached Bash shell with a long foreground sleep — the detached
shell gets reaped mid-sleep and the watch dies silently. The monitor.py tool (the
standalone league rank/round watcher) is the reference pattern for "always-on,
print-only-on-change". (session-derived, unverified)
Classification: process alive OR artifacts present OR server says running → INFRA (buffering), run is healthy. Do not resubmit, do not revert. Wait for terminal status or query the server.
uv run writes an install/sync preamble to stdout, which intermittently corrupts
machine-parsed JSON output. A CLI whose JSON you pipe into a parser will throw a JSON
decode error even though the underlying command SUCCEEDED — only the parse failed.
Do this:
Do NOT resubmit on a parse error. The submit may have created the request. Resubmitting double-submits. (Session-derived: three consecutive "failed" submits had all actually been created.) (session-derived, unverified)
Verify server/queue state before any resubmit — query the experience-request / round list and confirm whether your request already exists (see Branch C status check). Only resubmit if the server has no record.
Fix the invocation: when a CLI's JSON must be parsed, invoke the project's venv
python directly instead of through uv run:
.venv/bin/python <script> ...
This skips the uv preamble so stdout is clean JSON.
Classification: parse error with a created request on the server → INFRA (uv preamble), submit succeeded. Not a policy or backend failure.
First, decide the execution backend. For crewrift experience requests,
execution_backend="k8s" is the dependable hosted path. antfarm is slow-dispatch and
unreliable — measured ~61% episode failure rate, a ~60s/episode cold-start floor, and
multi-hour outages; a matched probe had k8s complete 4/4 in 239s while antfarm sat
~243s pending then failed all 4 on 502/read-timeout. If you have a choice, use k8s.
The A/B eval engine (eval.py → league_eval.run_league_ab / run_arm) already
encodes a formula-derived timeout budget and a wedge detector — let it run rather than
declaring failure early:
~ num * maxticks/24 * 3 + 900 seconds. Games run at realtime
sim pacing (TargetFps 24), so a fixed short timeout falsely fails slow-but-healthy
episodes. One episode at maxTicks 10000 ÷ 24 ≈ 7 minutes wall time; episodes
parallelize fully server-side, so the budget is dominated by the longest single
episode + warm-up, not num × episode-time.maxticks/24*3 + 600 seconds, a participant is wedged — fail LOUD. run_arm does
this with stall_grace_seconds (transient statuses 404/425/429/5xx are retried;
zero completions past the grace = declared wedged). Poll cadence ~10s.Status-probe a stuck round / request directly:
GET/list on /v2/experience-requests
(the parent) and its child episodes. A status of submitted/claimed/running is
non-evidence — keep waiting up to the budget above./healthz and /results. status=finished + 404
results = the game completed but emitted no artifacts (a game-side gap, not
slowness). (session-derived, unverified)sha256(policy_version_id)[:12] as a suffix; if you must stay on antfarm, intersect
the registrar's public /list with live league memberships to compute the
dispatchable set, then pin all seats explicitly.) (session-derived, unverified)live_url=None ran on k8s, not antfarm (only antfarm dispatches carry a
live_url) — don't count a k8s success as antfarm proof. (session-derived, unverified)Classification:
coworld results crashAn empty division leaderboard does not mean an empty division. The leaderboard only
shows SCORED champions, so 0 rows can coexist with a populated division (observed: 0
leaderboard rows alongside member_count=10; a champion-eligibility pool of 5 while the
broader competing-membership pool held 136). This is the trap that makes server-side
top_n champion auto-fill silently fail to fill a 7-seat field even though plenty of
runnable opponents exist.
Do this:
softmax_pull.py tool
already surfaces division leaderboards, rounds, and submissions from the endpoints
that work today (and coerces a null leaderboard to []), so use its
fetch_softmax(league_id) path for the outcome side rather than reading the
leaderboard alone.coworld results <division_id> (0.1.16) CRASHES on an empty leaderboard: the API
returns null and the client validates against list[LeaderboardEntryPublic]
(pydantic "Input should be a valid list"). Divisions with entries work fine.
Workaround: coworld memberships --mine to check a qualifying bot's status.
(Filed as Metta-AI/coworld issue #11.) (session-derived, unverified)Classification: empty leaderboard with a non-empty membership pool → INFRA /
expected state, not a dead division. Don't dismiss top_n or conclude "no opponents".
A hosted round that comes back failed or with zero/garbage scores is often a
runtime/infra failure, not a policy regression. Documented cases where reverting the
policy would have been wrong:
KeyError: 'authorization' / missing
in-cluster auth). No player ran → the round is zero policy evidence.404s. Packaging/infra skew,
not a policy bug.results.json, but a player process stayed blocked in websocket teardown and had to
be force-killed. If the player's process timeout is too short, the player disconnects
before the server records the round → no usable results even though game logic
completed. (And the runner-side image gate's own timeout must be long enough that a
results.json arriving just after a stingy deadline isn't discarded as a runner
timeout.)The classification procedure:
results.json + game logs + per-agent policy logs — the antfarm_run.py harvest
path pulls all of these into episode_NN/ folders), and any submission dry-run notes.timeout with a generous duration and
--kill-after, and MAP the timeout exit code to success — a player reaped after
the game is done is normal lifecycle, not a failure. Set the timeout comfortably
longer than a full episode (tens of seconds disconnected players before results were
written; minutes fixed it). This is try/finally-style lifecycle handling, not
error-swallowing.If the live policy genuinely is broken: fix it operationally — upload a clean replacement and let the new placement de-champion the old version — rather than adding a backward-compat shim for an obsolete import path.
Classification: failed/zero round with no started player or a teardown race → INFRA, non-evidence. Real per-player scores reproducing a regression → POLICY.
422 on submit / API shape driftConform to the deployed server's OpenAPI, not your local client models — the request
shape drifts. POST /v2/experience-requests was reworked on 2026-06-12 to a unified
roster body (one entry per seat, policy_ref). The old requester/opponents /
policy_version_ids / top_n / target shape now 422s.
eval.py → league_eval.build_request_body) is already on the
post-rework shape: the seat is pinned inside the requester object
(requester.slot) — the server rejects a top-level requester_slot supplied next to a
requester.antfarm_run.py (build_request) still emits the pre-rework shape and will 422
until patched to the unified roster body. Re-fetch live members and validate every
pvid right before launch; the league rolls policy versions mid-session.Classification: a 422 on submit → INFRA (API shape drift). Patch the request
body to the deployed schema; not a policy failure.
You can state, for the specific run, one classification with the evidence that backed it:
timeout). You did not revert or replace the policy on this evidence.If you cannot produce a terminal round with real scores, the answer is INFRA-or-unknown, and the run is not policy evidence — do not touch the policy.