tooling-analysis

name	tooling-analysis
description	Use for Crewrift-specific tooling analysis recipes when optimizing a player.

Tooling & analysis — recipes (crewrift tier)

On-demand recipes (8). Trigger→action heuristics; pull the relevant one when its situation arises.

1. Crewrift local replay viewing: env var, singular /client/ path, image parity, confirm playback advanced

crewrift · negative result

To view a Crewrift .bitreplay locally, launch the game image with COGAME_LOAD_REPLAY_URI=file:///coworld-replay/replay.json (path inside the container) and open the SINGULAR /client/replay page -- this is the only mechanism the image honors. Do NOT use 'coworld replay <replay.json>': it only sets COGAME_REPLAY_SERVER=1 (which Crewrift never reads; replayServerMode is set at boot from replayLoaded), so the uri is ignored and the container streams a live game stuck at 'waiting for players: 0/8'. URLs are singular /client/ (no .html): live=/client/global, replay=/client/replay?uri=...; the CLI's advertised PLURAL /clients/replay is dead and reports 'disconnected'. When mounting a replay output dir, do NOT mount over /workspace/crewrift (it hides bundled assets like data/tiny5.aseprite and the server fails before players connect) -- mount only config.json there and the replay dir separately. Playback requires producer and viewer image versions to MATCH or it aborts on a replay-hash mismatch; refresh both together (uv tool upgrade coworld + pull current crewrift image, then regenerate). CONFIRM playback actually LOADED before trusting visual checks: health and 'the map rendered' do not prove it. The /replay stream carries a 'replay tick ' sprite -- confirm N advances ~24/sec (resets to 0 each loop, plays once start-to-finish then holds the final frame; looping defaults false). During playback the logs/viewer show the NORMAL lifecycle ('waiting for players 3/8','game starting: players=8, imposters=2') because the replay feeds Join records -- that is the re-simulated recorded game and the expected sign it works; ticks 0-120 lobby and 120-240 role reveal genuinely show 'waiting for players', so account for that recorded phase before diagnosing 'stuck'. NEVER set COGAME_SAVE_REPLAY_URI together with COGAME_LOAD_REPLAY_URI (server refuses: 'Cannot save and load a replay together'); local server runs save no replay by default -- set COGAME_SAVE_REPLAY_URI to capture a .bitreplay. Episodes do not auto-end (run to the tick cap, ~2100 ticks ~= 3-3.5 min), so set a background timer to kill the container. Playback is real-time ~24fps with no scrubber (~587KB initial frame, ~1.2KB deltas; playback time = tick/24). The MettaScope viewer has macOS/Nim gotchas (the bundled 'cogames replay' wrapper fails; invoke MettaScope directly with --path to ~/.nimby/pkgs/*/src incl opengl, launch from a temp cwd pointing data at the matching MettaScope bundle's real cached assets NOT the metta checkout whose atlas is Git LFS pointer text, run in a foreground Terminal); for pure metric extraction parse the compressed replay JSON directly. _{sources: claude-code:962c8ab4-4ea3-4aaa-9f6d-20ea3cfce003, codex:claude-code:ff9f693c-b88a-419d-be7a-efc46750dc23, codex:019e8fc7-ebad-7b00-bc95-df1727641ce0, players_checkouts/players/players/crewrift/crewborg/docs/crewrift-repl (+5)}

2. A crewrift replay is INPUTS not state; recover events by re-running the real sim via expand_replay

crewrift · tool: expand_replay.nim

A crewrift .bitreplay/CREWRIFT replay is a byte stream of per-tick player INPUT masks, not state, so recover events by replaying inputs through the real simulator (tools/expand_replay.nim) and reusing the sim's exported public procs (e.g. tryReport for body-report attribution, the sim's ordered side effects for kill attribution) -- never reimplement game rules in Python; the CLI text just renders the typed API expandReplayTimeline(data: ReplayData): ReplayTimeline. Read a finished game from two aligned sources: the objective expand_replay event timeline (every player, true roles, all kills/bodies/votes/tasks/chat -- policy-independent ground truth) for WHAT happened, and each player's own per-slot subjective trace log for WHY, aligned by tick. You CAN read format CONSTANTS from game source to decode without the SDK (the older .rep format was a BITWORLD magic header plus little-endian join/leave/input/hash records with a JSON config blob; a standalone decoder extracted per-slot policy names, per-player input counts, last-active tick, button-press timings), and you can extract the CREWRIFT header/gameConfig (timing params, player names), but you canNOT decode per-tick events without the native decoder -- any tick derived only from gameConfig timing is an ESTIMATE and must be flagged. Chat is NOT stored in the replay stream -- agent logs are the only source for whether anyone chatted. Per-player input counts plus last-active tick make a fast liveness check (one opponent logged only 4 inputs and went silent; another left mid-match). Keep the replay parser PURE (no feature logic): produce a typed Game (players/roles, sampled positions, visibility intervals, kills/bodies/ejections, meetings with votes, chats, task completions, map geometry) shared by every downstream consumer, with the event vocabulary and correctness rules in exactly one place. Build position/visibility lookups queried by bisect: state_at(slot,tick) returns the latest sample at or before the tick, sees(observer,target,tick) is true when any visibility interval covers the tick; sort samples/intervals by tick in a post-pass before querying. suspicion_lab's replay_parse.parse_game expects a pathlib.Path, not a str (a str crashes with AttributeError on .suffix). A one-off offline tool can reuse this corpus/parsing infra without being wired into the nightly pipeline -- keep exploratory tools descriptive (no pass/fail gate), emitting a text report plus optional JSON. _{sources: claude-code:00f61c5a-0c3b-41ca-a586-969b3ce1ce83, claude-code:4c0f63b0-f7eb-408e-ac3f-00feb1dcc193, claude-code:10f74020-e0e6-4515-8d0c-f071d41766ad, coworlds/coworld-crewrift/README.md (+6)}

3. Version-match expand_replay to the recording build; hash-validation catches skew; trust results.json for kill COUNTS not replay ATTRIBUTION

crewrift · negative result · tool: expand_replay

expand_replay validates a per-tick state HASH and stops on the first mismatch, so it correctly re-simulates ONLY replays recorded by the same crewrift build (e.g. build 3ea899eb matched game v0.1.51); a binary from a different version diverges within a few hundred ticks, prints 'hash failed', and emits almost nothing. Keep one expander per version (expand_replay-), tag each scraped episode with the version that expanded it, and bump the ref when the league redeploys and fresh replays start hash-failing. Build a version-matched host-native binary with crewrift_lab/tools/build_expand_replay.sh (fetches public crewrift source as a tarball -- no credentials, just the host Nim toolchain) tracking the pinned CREWRIFT_REF; the coarse embedded gameVersion is the constant '1' and does NOT catch a build mismatch, so match the git ref. When you cannot supply that ref (older season, someone else's build), fall back to the visual replay viewer (loads into the episode's own game image, always re-simulates correctly) plus the version-independent player logs. Do NOT trust the bundled tests/replays/notsus.bitreplay fixture as an oracle (stale, hash-fails ~tick 36) nor a tools/expand_replay binary already in a checkout (matches whatever it was last built from). Even version-matched, trust results.json for kill COUNTS (server-authoritative kills[i]) but NOT replay-derived kill ATTRIBUTION: the re-sim attributes a kill to whichever imposter is near the body when it appears, mis-assigning at simultaneous-body ticks (it showed crewborg killing two players in two rooms on the same tick under cooldown) -- use the replay for kill TIMING and positions/rooms, results.json for counts. --snapshot-every N thins ONLY periodic player/body/visibility state sampling; every discrete event (kills, votes, room transitions, task completions) still emits on its exact tick and state rows are force-emitted on event ticks, so thinning never drops events (verified: kills land at ticks not divisible by the cadence). An observability-only addition (e.g. a JSONL visibility-interval emitter) does not affect the hash and can be cherry-picked onto an older ref. Compile replay-analysis helpers as optimized/release builds for bulk passes (a debug build managed ~37/560 replays in minutes while an optimized rebuild finished all 560 quickly). _{sources: claude-code:63658f66-83c7-482e-a805-d58527e10975, codex:019eaea8-4f78-7072-b4b0-539df963b7df, codex:019eaa2f-3bfd-73b3-aaec-4d709e098c39, personal_labs/crewrift_lab/crewrift/crewborg/docs/designs/suspicion-le (+5)}

4. Use results.json scores as primary evidence; apply inference rules and score fingerprints, but do not invent state the replay lacks

crewrift · ⚠ session-derived, unverified

When the replay is opaque/binary, results.json is the most reliable evidence and a crewrift episode-report is assemblable from results.json plus the replay header alone (no raw logs). It exposes per-slot arrays: scores, win, tasks, kills, imposter, crew, vote_players, vote_skip, vote_timeout. Inference rules: (a) a player with vote_skip=0 while all other alive players have vote_skip=1 was almost certainly dead at that meeting (cross-check the kills array for the killer's slot); (b) dead crew still complete tasks (ghost completion), so do NOT infer a player is alive from high task/score. Score-fingerprint triage flags anomalous episodes cheaply before pulling replays: a 'successful' crewmate score is 8 (all tasks, lost) or 108 (all tasks, won); a 'successful' imposter is 20/30/120/130/140 (2+ kills) -- auto-flag any score off those values. But do not infer state the replay does not record: among_them 'likely_dead' and estimate_meetings heuristics were REMOVED because there is no clean way to know deaths or meetings from the artifact -- surface only what is directly present and drop derived fields whose ground truth the format cannot supply. Note: optimizer run-*-stdout.log files are agent/session metadata (thinking tokens), NOT decoded game events -- do not mine them for action.kill/vote.cast. _{sources: claude-code:4c0f63b0-f7eb-408e-ac3f-00feb1dcc193, claude-code:5ba30745-2180-4e0c-9cb5-13846d2aa6a6, claude-code:f70c9801-7ee7-499b-97f9-4fd0848a6b8e, claude-code:e32290d2-800b-4c28-8f9d-ac6f65e9ae76}

5. Map slot->policy by NAME AND VERSION from episode metadata; crewborg logs do not self-identify version or role

crewrift

Map slot->policy by NAME AND VERSION from episode metadata before reading per-slot logs, never by eyeballing which logs are JSON: the 'first line is JSON' heuristic only proves a log is from the crewborg JSON-logging family, not which version, and ONE league episode can run several versions of the SAME policy at once (slot 0 = v15, slot 5 = v6) emitting identical trace schema, so a head/jq sniff proves 'a player log' not which version. The slot map lives in episode.json/the episode record, NOT results.json (which has outcome arrays but no slot map): league episodes expose policy_results[]=[{position,policy:{name,version}}]; experience-request episodes expose participants[]=[{position,policy_name,version}]. crewborg's per-tick stderr trace (perception/belief/mode decisions plus metric lines) is the richest behavioral record and comes ONLY from the Observatory policy-logs endpoint indexed by SLOT, so resolve crewborg's slot from participants first, then request policy-logs/{slot}. An EMPTY result from a vote/kill/suspicion jq recipe is AMBIGUOUS -- crewmate-role logs and no-meeting episodes legitimately emit nothing on imposter-only kill paths, so count domain.phase_change and check domain.decision_snapshot role before concluding the recipe is broken. meeting_context_serialized is multi-kilobyte; extract only the fields you need (e.g. per-slot {slot,color,alive,self}). For mechanic audits, grep the installed package for EVERY handler touching the mechanic (hp/death/kill); finding exactly ONE death handler is itself a load-bearing negative result. _{sources: claude-code:63658f66-83c7-482e-a805-d58527e10975, personal_labs/crewrift_lab/crewrift/crewborg/docs/trace-logs.md, personal_labs/crewrift_lab/best_practices.md, personal_labs/crewrift_lab/docs/crewrift-replays.md (+3)}

6. Anchor underperformance to a quantitative baseline, split by role, and surface behavioral gaps

crewrift · ⚠ session-derived, unverified

When diagnosing a role's underperformance, anchor to a concrete quantitative baseline before reading traces: framing a vague 'imposters are inefficient' (0-1 kills per 3-min match) against 'a competent imposter averages a kill every 30-45s when cooldown allows' converts it into measurable lost-time accounting (e.g. '13% of the match completely idle') attributable to specific phases. Split a dominant opponent's win-rate by ROLE first, because the edge is usually concentrated in one: league leader RowDaBoat's edge was crew discipline (+14pp crew win from near-zero complicity in crew ejections, burning the emergency button every game to reset imposter kill cooldowns, always 8/8 tasks), not kills or task speed -- the crew-win gap accounted for essentially the whole leaderboard distance while imposter kills/game were identical. A run summary should surface behavioral GAPS, not just win/loss: a scripted bot logged vent_suspected twice but never reached a meeting to report it -- the actionable finding is the missing detect-to-report linkage, which a bare score hides. _{sources: claude-code:0919285d-1374-4827-b596-6e7107d95526, claude-code:0e7b14ca-ffd5-4137-9456-47485d3c6f87, claude-code:962c8ab4-4ea3-4aaa-9f6d-20ea3cfce003, claude-code:dfa4502e-c404-40e8-bf86-8d15c72ba5af}

7. Decompose Crewrift episode analysis by role, cheap-to-deep, flagging outliers

crewrift

Always decompose Crewrift episode analysis by role -- crewmate and imposter are effectively two policies, so an aggregate distribution hides one role being broken; keep the split in every distribution. Connect/disconnect timeouts (scored -100) are crashes, not strategy flaws -- give them their own category and exclude from behavioral distributions. Analyze a batch cheap-to-deep across three tiers: Tier 1 reads results.json + episode.json for role flags/scores/tasks/kills/votes/timeouts without parsing replays (scales to hundreds); Tier 2 runs version-matched expand_replay on flagged episodes to recover killed-by-imposter vs ejected vs survived, itemized score breakdown, and vote correctness; Tier 3 reads the policy's own logs for the subjective 'why' at the tick of interest. Flag an episode as 'unusual' when it is an outlier on some feature via robust z-score or rank, and report the specific deviating feature (e.g. 'alive only 180 ticks vs median 3000', '0 chats vs median 4') rather than a generic anomaly flag. _{sources: personal_labs/crewrift_lab/.claude/skills/crewrift-report/SKILL.md}

8. Local debug launchers for the Crewrift/Among Them bots and their lifecycle footguns

crewrift · tool: run_debug_game.sh

Run the among_them Python smart bot via among_them/bot-policies && ./run_debug_game.sh (launches server, Nim opponents, the LLM-brained bot, and a debugger UI), or scripted/no-LLM with python3 -m sidecar.bot --name smartbot --debug; supply LLM via the softmax AWS Bedrock profile or ANTHROPIC_API_KEY/OPENROUTER_API_KEY with a matching --provider from env. Run the notsus Nim bot headless by setting COWORLD_PLAYER_WS_URL to the ws://127.0.0.1:8080/player slot+token URL, or with the visual debugger via -d:notsusGui (renders walkability mask, viewport, A* path, current/roam goal, input mask, velocity, stuck state). Know launcher footguns: quick_player prepends the standard players dir to bare names (a nested bot needs its full path), propagates --gui to EVERY spawned instance, and kills ALL child bots when any one exits -- to watch one bot, spawn N-1 headless via the launcher and one GUI'd directly. _{sources: bitworld/among_them/players/how_to_make_a_bot.md, coworlds/coworld-crewrift/players/notsus/README.md, opencode:ses_224322057ffexj7SYlcfVA0zgy}