experiment-discipline

name	experiment-discipline
description	Use for Crewrift-specific experiment discipline recipes when optimizing a player.

Experiment discipline — recipes (crewrift tier)

On-demand recipes (9). Trigger→action heuristics; pull the relevant one when its situation arises.

1. Engineer the match to reach the rare behavior under test, run a canonical reproducible config, then re-validate at real timings

crewrift · ⚠ session-derived, unverified

Rare events (kill, body report, meeting, vote) must be engineered, not waited for: meeting/voting validation fails across many seeds because filler bots never call meetings and imposters almost never kill (kill cooldown ~1200 ticks / ~50s in a ~4000-tick match, opportunistic kills needing exactly one visible crewmate), so tune the instrument — shorten the kill cooldown in server ticks via --imposter-cooldown-ticks (e.g. 1200->600->450, not vague seconds), run 2 imposters (1-imposter games never surface coordination/double-kill/alibi dynamics), slow task-win pressure so crew cannot win before any body is reported, mix in aggressive bots, or force the condition. Derive minimum player counts from the win condition first: with 1 imposter a 2-player game instantly satisfies aliveImposters>=aliveCrewmates and triggers instant game-over, so any Playing/Voting scenario needs >=3 players. Match the run LENGTH to the symptom: a 40s/120s run can end clean without ever firing a meeting/kill/scoring event, and under tasksPerPlayer=1/300 ticks '7 of 8 finished 0 tasks' is the expected order of magnitude, not a stuck-bot bug. Flag modes you could not exercise as 'structurally verified but not validated against live play.' Before a live diagnosis run, clear the traces directory and set trace level to full (or you analyze stale/under-detailed traces), and formalize the working commands into a self-contained 'full live game' script long enough that kills happen. Document a canonical 'small live test' config (crewrift: 4 players, 1 imposter, 1200-tick kill cooldown, 600-tick meeting time, 3-min playtime, port 2000) next to the larger 'standard' config; scale down to the smallest player count that still exercises the mechanic (8 bots -> 4) for shorter games and easier attribution. Enumerate ALL named presets and design for the full range — actual play happens on longer presets, so a strategy validated only on the short test preset mis-budgets time in tournament. Because shortening changes the event distribution, never tune prompts on a trace lacking the behavior and re-validate at real tournament timings afterward. _{sources: claude-code:0bd5f20d-10ad-41ef-8372-ae778cd59000, claude-code:309519c5-1e0c-4dfe-b7df-126611184851, claude-code:73fa0a3d-37fb-4408-bae8-245db6e62c56, claude-code:d48f9347-7884-4ea2-8659-230051d86410 (+17)}

2. Force the Crewrift role via seed+slot or per-slot override, verify the role was OBSERVED, and confirm the dispatch path allows it

crewrift · negative result · ⚠ session-derived, unverified

For any role-specific evaluation, force the role yourself instead of trusting auto-rotation: the experience-request API's rotate_seats and slot=-1 round-robin do NOT give matched coverage (across 100 random-role games a policy ended 72 dead, 27 crewmate, 0 imposter-alive), so issue many small (often 1-episode) experience requests that lock roles via custom game config. Use the seed+slot lever (roles assigned by a seeded shuffle over slots) or a per-slot game_config.slots[*].role override — override is slot-based (tag a SLOT imposter and seat a policy there; you cannot tag a policy as imposter by identity), and the assignments field is unavailable on the opponents path. Beware RNG coupling: role and task assignment draw from the same seeded RNG with roles first, so the same seed + same imposter slots yield different task assignments depending on how many roles were pre-fixed. A force-role flag is only trustworthy if you verify the policy actually OBSERVED the forced condition (--force-role imposter was passed but the bot self-detected crewmate every time because pixel detection disagreed and won) — confirm via server trace/stdout, not the flag, and prefer pinning roles via the engine's per-slot config over fishing for a seed. Confirm the hosted dispatch path can pass slot-role overrides — tournament/commissioner scheduling may expose only variant_id, policy_version_ids, optional seed, and tags, so 'engine can do it' is not 'this dispatch path can request it.' _{sources: claude-code:e5b86992-9b8a-498e-a47a-1ead507cc54c, claude-code:f70c9801-7ee7-499b-97f9-4fd0848a6b8e, codex:019e941c-8eb9-7402-a77f-8fe75a5d19bf, codex:019ea9af-7020-7550-b427-d3478fed2057 (+5)}

3. Bind every episode claim to evidence, keep OBSERVATION/HYPOTHESIS/RECOMMENDATION distinct, and make each recommendation name change, metric, risk, and evidence

crewrift · see related: S010 (other tier)

Anchor each factual claim about a Coworld/Crewrift episode to four things: the real decoded tick (flag estimates as est), the player (slot index and/or display name), the event key (action.kill, vote.cast, lifecycle.game_end, meeting.called), and the source artifact (results.json, replay, policy_log, config.json, timeline). Never invent ticks, players, roles, votes, scores, or outcomes; state missing evidence as missing, and treat replay logs/code/docs/live artifacts as the source of truth rather than inferring mechanics from behavior. In reports keep three voices distinct and labeled: OBSERVATION (evidence), HYPOTHESIS (inferred cause), RECOMMENDATION (testable change), and tie each observed behavior to the metric it moved. Each recommendation in the prioritized-improvement section must name the change, the expected metric movement (metric + direction + magnitude), the risk, and the supporting evidence ticks/keys, keeping each change small and testable and prioritized P0/P1/P2 so a downstream optimizer can act without re-reading raw logs; vague advice like 'play better' is disallowed. _{sources: claude-code:5ba30745-2180-4e0c-9cb5-13846d2aa6a6, optimizers/.cursor/skills/episode-evidence-binding/SKILL.md, optimizers/instructions/mvp/reporter.md, archive/cogames_playground/BULBACOG_LEADERBOARD_GOAL.md}

4. Triage Crewrift losses per-player and use fixed game invariants as correctness assertions

crewrift · ⚠ session-derived, unverified

Separate 'our policy's fault' from 'teammate/environment' with a per-player task-completion breakdown: of 10 crewmate losses ~6 were task-timeout losses where our agent finished 8/8 but a teammate sat at 1/8, capping the crew below the task threshold — this distinguishes fixable policy weaknesses from uncontrollable teammates. Use fixed game invariants as assertions across a match: each crewmate has a fixed task count, so one bot appearing to 'steal' a task from another (or a station corrupting task confirmation) flags a deeper correctness bug worth a dedicated fix pass. _{sources: codex:019e9656-7cfc-7032-ae70-82b6dbc0a053, ses_1fa63b520ffeQP7WMCZiDebllG}

5. Validate crewrift sim/replay changes against the right tests, with hash-correct seeded synthetic fixtures

crewrift · ⚠ session-derived, unverified

Several crewrift sim/replay fixtures are pre-existing stale-fixture problems, not your regression: the eventlog reporter test (reporters/eventlog/test_eventlog.nim) and the bundled notsus.bitreplay hash-mismatch at tick 158, failing BEFORE event-row validation and BEFORE roles are assigned at startGame (so it yields no manifest row). Validate via tests/test_discrete_buttons.nim, tests/test_replay.nim, tests/tests.nim, and by compiling tools/expand_replay.nim; do not assert manifest-role correctness against the bundled notsus replay — put manifest-role regression tests in a dedicated synthetic two-player fixture with a configured imposter slot that runs past startGame. The replay loop stops on recorded hash ticks, so a synthetic fixture cannot use placeholder hashes — compute matching hashes by stepping a live sim with the same joins/config, and pin the random seed (the default sentinel is resolved separately for expected and replay sims). When fixing kill attribution, add a regression test for simultaneous kills (two imposters in the same tick), since the single-killer path passed under the old buggy heuristic. After a failed check in a Nim test, guard subsequent assertions so they don't dereference a missing value (an absent manifest row) and cascade into a confusing crash. Grader pre-commit validation is python3 -m unittest discover -s tests, python3 -m compileall graders, git diff --check; boto3 s3:// paths are imported lazily and NOT covered by local tests. _{sources: codex:019e66c3-638e-7e70-afdf-8f6df9299f6a, codex:019eb8c0-1b42-7a92-843b-c5db259d9ac3, codex:019ebd2d-defe-7032-b3d7-1b960d832853}

6. Gate every Crewrift change through an in-image Gate-1 smoke plus full tests, run env-flag A/B arms, and validate locally with the slot-matrix runner

crewrift · tool: crewrift_slot_matrix_runner

Gate every Crewrift policy change through a Gate-1 local smoke exercising the actually-changed in-image path (the flag-on search/kill path, zero tracebacks) AND a full test+lint pass BEFORE any hosted A/B or league submission, recording test counts and ruff status per version. Gate candidates behind an env-var flag (CREWBORG_FRONT_BIAS, CREWBORG_NO_ISOLATION) so the same built image runs both arms, the baseline is provably inert when off, and rejected experiments are committed gated-off rather than reverted. Keep a prior hand-tuned model reachable behind a flag (CREWBORG_SUSPICION_WEIGHTS=0) with its tests pinned when swapping in a fitted model, so regressions are isolatable. Validate a candidate image locally with the crewrift_slot_matrix_runner gate against the pinned coworld_manifest.json, setting --max-candidate-vote-timeouts 0 so the run fails if the candidate ever times out on a vote. _{sources: personal_labs/crewrift_lab/crewrift/crewborg/version_log.md, personal_labs/crewrift_lab/WORKING_CONTEXT.md, players_2/players/crewrift/suspectra/README.md}

7. Drive the Crewrift loop from failed-replay to single capability to one function to rerun

crewrift

Start from replays where your bot scored low, died early, stood still, missed a body, failed to vote, or killed in front of witnesses; expand the replay, name the SINGLE failed capability, jump to the responsible function via players/notsus/README.md (decideNextMask for per-frame action, findPath/navigateToPoint for navigation, decideImposterMask for imposter pretend-then-hunt, parseVotingScreen/desiredVotingDecision for voting), make one small change, and run again. _{sources: coworlds/coworld-crewrift/README.md, coworlds/coworld-crewrift/players/notsus/README.md}

8. Clean Crewrift batches first: filter artifact and non-scoring-slot episodes, join by policy_version_id, decompose by role

crewrift

Filter platform-artifact episodes BEFORE concluding and join scores by policy_version_id, never slot or name. Two signatures: connect/disconnect-timeout episodes score -100 for ALL players and read as losses (a 48%-timeout batch looked saturated ~5% crew/~100% imp, while a clean 50-game batch showed crew 29-40%, imp 38-80%) — drop episodes with nonzero connect_timeout+disconnect_timeout; instant-GameOver episodes score everyone 0 with all timeout flags zero and a half-assigned roles array, so the -100 filter MISSES them — filter by whether the game reached the Playing phase or detect 0 score AND 0 tasks AND no votes. The policy_version_id is authoritative (a crewborg v17 round also contained another player's crewborg-v23 fork). A round can seat the SAME policy in two of eight slots ('Name (2)'), so strip the ' (N)' suffix and decide seat-vs-episode as the unit. Decompose evaluation by role (crewmate vs imposter are two policies) and opponent, pinning role via roster slot + game_config_overrides.slots. Detect ejection from the trace (role->dead) or replay, not the '1-imposter GameOver-after-vote' trick (fails in 2-imposter games). 'Finished all 8 tasks' is not a clean win: an all-tasks crewmate scored -2 from a vote-timeout penalty (-10) while another scored 98 — a pure score-anomaly filter catches these for free. Beware non-scoring slots too: in Among Them there are 10 slots (0-9) but only slots 2-9 produce scores; a participant in slot 0 or 1 runs correctly but receives no score and the CLI shows the episode completed with 8 scores, so check the assignments array for non-scoring slots before concluding your agent is 'missing' or that a win-rate sample is complete (a reported '10 slots, only 8 score' pain point was normal slot rotation). _{sources: personal_labs/crewrift_lab/crewrift/crewborg/version_log.md, personal_labs/crewrift_lab/lessons_archive/TENTATIVE_LESSONS-20260612-, personal_labs/crewrift_lab/WORKING_CONTEXT.md, personal_labs/crewrift_lab/AGENTS.md (+2)}

9. Assert relational suspicion-test properties and log weight provenance honestly

crewrift

Write suspicion tests to assert RELATIONAL properties (evidence raises P; one cue alone stays below the near-certain bar; corroboration crosses it; a brief body-proximity outscores a long one) rather than exact numbers, so they survive re-tuning and fail only when a cue's qualitative shape changes. Maintain a provenance log with one row per weight-setting event recording whether each suspicion weight is a hand guess or earned from N games -- keep it honest because hand-tuned weights fit from zero games are the default failure mode (every initial crewborg weight was a guess fit from zero games). _{sources: personal_labs/crewrift_lab/crewrift/crewborg/docs/designs/suspicion.md}