build-submit-packaging

name	build-submit-packaging
description	Use for build submit packaging recipes in scripted Coworld policy optimization.

Build, package & submit — recipes (loop tier)

On-demand recipes (41). Trigger→action heuristics; pull the relevant one when its situation arises.

1. Two non-interchangeable submission surfaces: use the first-class coworld policy-image flow, not legacy cogames/zip-S3

loop · negative result · tool: coworld

For new Coworld/Among-Them submissions the maintained path is the policy-image flow via metta's packages/coworld CLI (uv run coworld ...), NOT the legacy cogames Python bundle commands (create-bundle/upload/ship) nor the classic zip/bundle+S3 path (cogames upload). The classic zip path creates a version with s3_path set and container_image_id null; a season error 'Policy version has no container_image_id' means you used it and must build an image and upload a NEW version via the container path (no retrofit). Two overlapping surfaces also break attribution silently: the legacy public path (cogames.coworld upload-policy from a public clone) is NOT the newer first-class CLI, and an image uploaded via the legacy surface can exist as a policy yet have NO league submission so it never plays while an older v2 upload is the one actually entered — when a 'submitted' policy isn't playing, check WHICH surface uploaded it. Prefer the first-class coworld upload-policy CLI because it exposes --use-bedrock and repeatable --secret-env for LLM credential plumbing the legacy path omits. Write the player to read its websocket URL from env COGAMES_ENGINE_WS_URL and open exactly that socket; the runner owns slot/token assignment so never hardcode ports/slots/tokens. Add tooling through the repo UV project, not ad hoc scripts; the game source / Nim wrappers are reference-only, not the contract. When public play docs and in-repo metta source disagree, trust the repo source. _{sources: bitworld/among_them/README.md, bitworld/among_them/players/SUBMIT_TO_TOURNAMENT.md, bitworld/among_them/players/how_to_submit_coworld_policy.md, claude-code:962c8ab4-4ea3-4aaa-9f6d-20ea3cfce003 (+5)}

2. Submit under one persistent name with an OBSERVED-state ledger; the leaderboard is the only truth

loop

Submit repeatedly under ONE persistent league name and let the server auto-increment the league version (alpha-cog -> :v1, :v2) rather than baking version numbers into the name. Three gotchas: (1) a different --name creates a SEPARATE league policy with its own v1 history that name-keyed tooling won't find (deliberate one-off only); (2) version auto-increments GLOBALLY per name, not per season, so shipping the same bundle to two seasons in one run yields :v1 then :v2; (3) the duplicate-check keys on git SHA + season only (not name), so re-shipping the same SHA to an already-submitted season triggers the duplicate guard even under a new name — override with --force or ship to an unused season. Because server league-versions don't align 1:1 with internal experiment versions, maintain an append-only ledger (e.g. experiments/submissions.json) recording OBSERVED state per submission: internal version, full git SHA, flag snapshot, A/B provenance, validation result, submission id, membership id, division, season, round/score; commit ledger updates separately from code and never let a row imply an unverified win. Treat the LIVE leaderboard, not the local ledger or recorded tags, as source of truth for which version ran where (they disagreed in CvC: V5 showed in beta-cvc on the leaderboard but beta-teams locally). The empirical tournament noise floor, from two near-identical entries of the same code (V11 at 22.07 vs 22.28), is ~0.2 points; improvements below that are indistinguishable from noise. _{sources: codex:019e4c28-d516-7ca0-b145-153c152b4f43, opencode:ses_2094d724cffebAsFmXFHGyRB7d, opencode:ses_21a1839daffe5QnSh33ozoq3rB, alpha_cog/experiments/README.md (+4)}

3. Acceptance is not placement: leaderboard lags, divisions are tiered, and scheduling can gate a strong policy

loop

Submission acceptance is not placement and membership flags are not performance. After submit, capture the identifiers (version :v1, league name, submission id, status pending) and debug a missing policy as four INDEPENDENT stages — image, policy version, submission, league membership — because success at one does not imply the others. A submitted policy first qualifies via self-play (~1-2h), enters a competition pool, and the first reliable leaderboard position appears only after ~20 competition matches (~2-6h total; e.g. beta-cvc runs 20 matches across 2+6/6+2/4+4 splits on machina_1 at 10k steps, scored junctions_held/max_steps). 'active'/'champion=true' means the account's active entrant, NOT winning (minimum_champions: 8 means top-8); a fresh policy showing all zeros is usually 'not scheduled yet', not 'failing'. Divisions are TIERED (Qualifiers -> Dirt -> Wood); a new submission lands in Qualifiers status 'qualifying', only an active-champion membership schedules rounds while non-champion memberships sit 'qualified but idle', a division with too few champions won't run, and uploading a new version under an existing name does NOT update the active membership in place. A division leaderboard endpoint returns null while its round runs; a hidden/older 'Daily' division (an old div_... id) can be replaced by tiered divisions mid-history, so re-confirm which membership/division is active before trusting an old rank. Don't idle on the scheduler; mine the freshest completed round for the next fix while the new version qualifies. _{sources: codex:019e18fd-64ea-7040-a62b-f54856a9e6d8, codex:019e1d2c-7707-7223-bd6c-d2c41104a10a, claude-code:307f1069-21fb-469e-bcd2-8a07f2dba0d8, opencode:ses_21a1839daffe5QnSh33ozoq3rB (+6)}

4. Inspecting league submissions and standings: which endpoints work, list don't GET

loop · negative result

There is NO GET-by-id route for Coworld league submissions: GET /v2/league-submissions/{sub_id} returns 404 even on success — check status via the list endpoint GET /v2/league-submissions?league_id=...&limit=N and match on id (the Observatory 'Submit policy' form just POSTs /v2/league-submissions, so you can replicate it). The league-policy-memberships list ignores a league_id filter (returns empty) — query by division_id, resolving divisions (Qualifiers + main) via GET /v2/divisions?league_id=.... Via the CLI: coworld submit <policy>:<ver> --league <id> (omit version for latest), then verify placement by LISTING (coworld submissions --mine --league <id>, expect pending->placed), coworld results <league>, coworld rounds --league <id>. AVOID coworld memberships and coworld episodes: they may crash with a pydantic ValidationError because the server returns status:'qualifying' while the CLI model still requires removed is_active/is_champion fields. Pick the current league via coworld leagues (a stale id submits to the wrong game); treat league/division/coworld IDs as rotating and re-query (coworld leagues --json). _{sources: players_checkouts/players_1/docs/coworld-player-swap.md, metta/agent-plugins/kitchensink/skills/ks.build-submit-policy/SKILL.md, bitworld/among_them/players/how_to_submit_coworld_policy.md, archive/cogames_playground/bulbacog/COGS_V_CLIPS.md (+1)}

5. Certification/run-episode is protocol-only and ignores variants[]; always also test a real variant

loop · negative result

A clean Coworld certification run does NOT prove gameplay and a zero cert score is not a broken player: the default local run uses a short cert fixture with a tiny step cap that only validates protocol correctness (connect/handshake/disconnect) and is often degenerate (e.g. kills impossible when killCooldownTicks exceeds maxTicks), always scoring zero, whereas the live league uses a different mission with a much larger step cap. coworld run-episode runs the manifest certification game_config and IGNORES manifest.variants[] (no --variant flag); behavioral diagnostics require coworld play --variant <id> (e.g. machina-1-daily). To run a specific variant headless, build an episode request with build_manifest_episode_job_spec(pkg, variant_id=..., player_images=[img], player_run=[...]).model_dump(by_alias=True, exclude_defaults=True, mode='json') from coworld.certifier, write it to JSON, and run coworld run-episode <manifest> <req>.json. A manifest selects the variant named default when present else variants[0]; for talk strategies select the talk variant from manifest config (not a CLI flag) and run with -v talk so the replay surfaces the talk channel. The smoke-test episode does NOT execute graders: certifier.py (~line 208) only validates each declared grader image is reachable and never launches it. _{sources: metta/agent-plugins/kitchensink/skills/ks.build-submit-policy/SKILL.md, metta/agent-plugins/kitchensink/skills/ks.coworld-player-gotchas/SKILL, archive/cogames_playground/bulbacog/COGS_V_CLIPS.md, archive/cogames_playground/bulbacog/designs/OUTER_LOOP_AND_MODES.md (+2)}

6. Confirm a local episode actually succeeded before trusting it; iterate locally before uploading

loop

Before trusting a Coworld run, confirm a clean start->play->exit, a written replay, and no tracebacks in logs/policy_agent_.log; audit the generated artifacts directly to confirm WHAT ran before claiming success (a starter can connect, send correct binary action packets, decode packed frames, and run a trivial movement loop while understanding nothing — 'connects and moves' is not 'capable policy'). Do not submit blind: run the bundled baseline, run your own image against the manifest, and inspect the replay and logs first. Iterate locally before uploading: coworld download a package, docker buildx build --platform=linux/amd64 --load the image, then coworld run-episode <manifest> <image>, and inspect the per-episode workspace under /tmp/coworld-cert-/ (config.json, results.json, replay.json, per-slot logs). If the image's default CMD is not the player entrypoint, pass the SAME --run argv to run-episode you'll pass to upload-policy. --timeout-seconds is the START-failure health check (60s play, 3600s run-episode), NOT the episode step cap (that lives in the manifest game config); an episode is roughly ticks/24 seconds wall-clock, so set it generously for long episodes. run-episode's stdout is buffered behind | tail, so watch the output dir and docker ps rather than waiting on stdout. _{sources: metta/agent-plugins/kitchensink/skills/ks.build-submit-policy/SKILL.md, coworlds/coworld-crewrift/README.md, archive/cogames_playground/bulbacog/COGS_V_CLIPS.md, archive/cogames_playground/bulbacog/designs/OUTER_LOOP_AND_MODES.md (+3)}

7. Player-image positional semantics for play/run-episode; resolve-and-upload is a GAME flow

loop

Coworld play/run-episode player-image positional semantics: zero images runs the manifest's bundled starter image in every slot; one image fills all slots with it; N images (N = player_count) assigns one per slot, enabling adversarial tests pitting your policy against a different one on a subset of slots. For local dev depend on the coworld[auth] extra and drive runs with coworld play <manifest> [<player_image>...]; it accepts a manifest by path, URI, or bare cow_ id with no prior download. run-episode fills all 8 slots with your single image and produces results.json, replay.json, and per-slot logs/policy_agent_*.txt. The runnable run field is a FULL launch-command replacement: empty run=[] uses the image's ENTRYPOINT+CMD, non-empty run is passed as --entrypoint run[0] <image> run[1:]; each --run X appends exactly ONE argv token (force python /app/foo.py via --run python --run /app/foo.py; a single quoted --run "python -m x" parses as one executable and fails). coworld resolve-and-upload COMPOSE TEMPLATE VERSION OUTPUT is a GAME-upload wrapper (digest-pins manifest refs), NOT a policy flow — use upload-policy for policies. Pick a command by the artifact/flag you need, not by assuming one subcommand is a superset (in coworld==0.1.4 run-episode produced episode artifacts but only play exposed --variant). _{sources: archive/cogames_playground/bulbacog/COGS_V_CLIPS.md, archive/cogames_playground/bulbacog/designs/LOOP.md, players_checkouts/players/archive/players/among_them/coborg/PLAN.md, player_labs/.claude/skills/coworld-policy-lifecycle/SKILL.md (+7)}

8. coworld download layout and reading the game image tag

loop

coworld download <game> requires a running Docker daemon and pulls every image in the manifest (game, all players, optional grader/reporter — Cogs-vs-Clips ~1.7GB; a missing daemon yields CalledProcessError). It writes /<coworld_id>/ (default ./coworld, override -o; --refresh re-pulls): coworld_manifest.json (authoritative game + player[] manifest), coworld_images.json, and AGENTS.md, and prints the coworld-id (cow_); pass that bare id or the manifest path to play/run-episode. Read the game image tag straight from manifest game.runnable.image (rewritten to a local tag like crewrift--N:downloaded) rather than guessing the -N suffix — a package bundles several images and only game.runnable.image is the server. Older releases wrote a flat ./coworld/coworld_manifest.json which editable Metta source no longer produces, so don't assume the flat path. The downloaded manifest is an untracked LOCAL artifact, not repo source; coworld download retags the canonical image locally as ':downloaded' and overwrites the local manifest, so when debugging 'did the game change?' inspect THIS manifest's digests and add a TEMP variant carrying your match params rather than editing it in place. _{sources: archive/cogames_playground/bulbacog/COGS_V_CLIPS.md, metta/agent-plugins/kitchensink/skills/ks.build-submit-policy/SKILL.md}

9. Leave manifest player env empty and pin your own policy ref in the image; env is preserved across substitution

loop

For a single-policy image do NOT declare the policy-selecting env var (COGAMES_POLICY_URI) in the manifest player[].env block (keep PLAYER_ENV_JSON='{}'): the runner injects env only from that block and falls back to the Dockerfile ENV for anything undeclared, so declaring it lets a runner silently swap policies while leaving it empty lets the in-image default win. A policy ref is live executable config (resolves via registry/class-path with an S3 fallback), so an accidental override loads OTHER policy code into a fixed-identity image rather than failing. Manifest env is PRESERVED across player substitution, so a substitute honoring that var can be hijacked into loading the upstream starter; do NOT edit the upstream manifest - hardcode your ref in your entrypoint's main() and verify by grepping your player's log for its ref. The certifier enforces three substitution rules (image-count match, run-replacement, env-preservation), so keep image count and manifest env. For an image hosting MULTIPLE policies emit one manifest entry per variant (distinct id, distinct baked env, same image), or hand-author an entry overriding COGAMES_POLICY_URI (e.g. nlanky?miner=4&scrambler=2) to field a non-default short_name / retune a role mix at runtime via query params rather than rebuilding. _{sources: auggie:c9f833a1-50e8-4526-8839-f438394f903e, claude-code:52cdf7a0-e04b-4105-8678-39efc85c46da, claude-code:67ff1fe3-48c0-4738-b7fb-8b994e567434, claude-code:c6fec691-565c-406d-b7dd-37e9387f5475 (+2)}

10. The packaging contract is the manifest binary path plus runner argv injection: use CMD, not ENTRYPOINT

loop · negative result · ⚠ session-derived, unverified

The runner packaging contract is the manifest 'binary' path plus the runner's argv injection, not your local ENTRYPOINT ergonomics. ENTRYPOINT ['agent'] causes the runner's binary command (e.g. /usr/local/bin/pystack ...) to be appended as extra positional args and break parsing — use CMD ['', ...defaults] with NO ENTRYPOINT so the runner command replaces it cleanly (a manifest's explicit run:[...] overrides the Dockerfile CMD, so a CMD still passing --address/--port can be dead, misleading code to remove). The player should take connection config (engine WS URL, auth token, slot) from env/CLI, not hardcoded Dockerfile host args (COGAMES_ENGINE_WS_URL, --url, --token, --slot). The bitworld runner uses Nim-style colon flags (--address:HOST --url:ws://...?name=...&slot=... --slot:N) that Python argparse rejects; normalize colon flags but split ONLY on the option separator before URL/query content, because the --url: value's '=' query string is mangled by a naive split. A clean game-end disconnect can be misread as a failure: the BitWorld server closes the websocket WITHOUT a close frame at episode end and the websockets library turns that into a nonzero exit, so treat normal game-end disconnects as clean shutdowns (exit 0). Add a regression test capturing the exact runner argv shape that broke. _{sources: codex:019e1df4-08f8-7b33-9a82-18024a199075, codex:019e1dec-8fd1-7b53-8cab-c4ad94229695, codex:019dfece-78f8-7722-8301-f4e0fafc2cd9, codex:019e184d-2de6-7b22-b8c0-853edbf53265 (+1)}

11. Build reporter/player images from the right context, install deps in-image, overlay owned source via build-context

loop · negative result

Build a Coworld reporter Docker image with the build context at the reporters/ repo ROOT, not the reporter dir, or COPY ../../reporter_sdk cannot reach the shared SDK; the Dockerfile must both install dependencies (pip install -r requirements.txt) AND ship the SDK or the container dies with ModuleNotFoundError (e.g. pydantic). Fetch deps in-image via nimby --global sync nimby.lock (the repo-root pattern) rather than copying vendored dep dirs. When a player's source is owned in one repo but its Docker build needs a separate monorepo checkout as the build context, use docker buildx build --build-context to overlay the owned source on top of the build context, guaranteeing the image ships the freshest in-repo source; make the build script auto-detect that secondary checkout in fixed precedence (explicit env override, then sibling dir, then a known home path) so the common case needs no config. When a player installs a dependency from a remote tarball at a mutable build-arg ref (e.g. PLAYERS_SDK_REF defaulting 'main'), resolve that ref to the exact commit recorded in uv.lock and pass the SHA as the build-arg, so the Docker layer cache busts exactly when the lock moves; a literal 'main' build-arg plus a cached remote-tarball fetch silently reuses a stale SDK layer. Test in two layers: pytest for fast iteration, then a containerized smoke.sh that builds the image and runs it against checked-in fixtures to confirm the PACKAGED image satisfies the contract. _{sources: claude-code:10f74020-e0e6-4515-8d0c-f071d41766ad, claude-code:1402d3ec-c565-4576-ba22-97942b9f9d0b, claude-code:2e0ca80b-d747-4358-a5dc-650397742760, personal_labs/crewrift_lab/docs/designs/building_players.md}

12. Pin the submission stack to what the policy was tested against; pin mettagrid explicitly and verify actual imports

loop · negative result

Pin the submission image's stack to what the policy was developed against. The Cogs vs Clips example Dockerfile installs mettagrid and cogames from PyPI UNPINNED, so the built image can ship a different stack (e.g. cogames 0.27.2 / mettagrid 0.26.20 vs developed-against 0.26.19 / 0.26.17) and a short 3-tick certification smoke completes but cannot catch the behavioral drift. A working build order: install the game package (e.g. cogsguard) from git at its locked SHA, then pip install --no-deps the local policy package (avoids dragging in conflicting transitive pins), then install the player runtime (cogames). Pin mettagrid explicitly in BOTH pyproject.toml and the Dockerfile rather than relying on it being transitive, and do NOT try to mirror the player's mettagrid with the game image's (they intentionally differ in artifact contents even at the same version; version mismatches cause observation-decode errors). A declared dependency does not imply use (torch shipped in alpha_cog's bundle declared-but-unused) — verify actual imports. Make optional runtime deps truly optional so the policy loads inside the submission container: guard env-specific imports (e.g. mettagrid.bitworld) with inline fallback constants and add **kwargs to the policy init for Docker/script compatibility. The canonical player-image template lives at metta/packages/coworld/src/coworld/examples/cogs_vs_clips/player/. _{sources: claude-code:6064640e-b01d-47af-b5d8-b4d7d982e559, archive/cogames_playground/bulbacog/designs/COWORLD_MIGRATION.md, opencode:ses_21a181ab8ffe3nrttiVWHpM4LJ, opencode:ses_21af0e707ffeWhoJeouK9agHhn}

13. LLM features silently fall back without credentials; the runtime credential mechanism is EKS IRSA, not ECS/local

loop · negative result

A policy that calls an LLM can silently fall back to deterministic play in the hosted pod while passing local smoke tests, because the dev box has credentials and the pod may not; an LLM flag with no creds behind it is a silent no-op. Enabling the LLM in the Dockerfile is not enough, and a 'grant cloud LLM access' flag (--use-bedrock setting USE_BEDROCK=true plus a Bedrock service account) does NOT guarantee creds reach the process (images uploaded with --use-bedrock still ran with AWS_ROLE_ARN/AWS_ACCESS_KEY_ID unset). The credential resolver must cover the ACTUAL runtime's mechanism: production was EKS injecting via IRSA/web-identity (AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE), NOT the ECS/local paths the bot handled — a resolver knowing only ECS task-role and AWS CLI paths fails every call with no_key/no_key; fix by adding STS AssumeRoleWithWebIdentity (IRSA) support plus a startup llm_init trace summarizing provider selection and env presence so future failures self-diagnose. Container ENV applies only on the image path; if the default LOCAL test path runs a native build inheriting host env, set the flag on the host launch too. The hosted reporter runner also does NOT pass AWS_* env to reporter pods, so real Bedrock calls fail without a reporter-pod Bedrock service account. An in-image smoke verifies resolution (use_llm==True, correct model/client) only up to the provider boundary, NOT the live call (a creds/inference-profile failure falls back silently) — confirm bedrock:InvokeModel at packaging and confirm the live call via episode logs after placement. _{sources: claude-code:00f61c5a-0c3b-41ca-a586-969b3ce1ce83, claude-code:0bd5f20d-10ad-41ef-8372-ae778cd59000, claude-code:77e158b0-bd16-41f8-8473-d1b1522d451b, claude-code:7b965041-3e1e-4d77-bc3a-a48662331630 (+2)}

14. Support direct-Anthropic and Bedrock behind one toggle; keep a scripted fallback that alone passes the gate

loop · ⚠ session-derived, unverified

To make an LLM player work locally and in the league, support direct Anthropic(api_key=...) and AnthropicBedrock(...) behind one boolean, accepting multiple toggle var names (yours plus CLAUDE_CODE_USE_BEDROCK). Bedrock needs the inference-profile model id form (e.g. us.anthropic.claude-haiku-4-5-...-v1:0), differing from the direct API id, so default the model by active backend; it authenticates via standard AWS env creds injected by the league's Bedrock service account when uploading with --use-bedrock (also sets USE_BEDROCK=true). Keep boto3 in a scoped optional extra so only Bedrock images pull it; locally Bedrock works via the softmax SSO profile in us-east-1. Keep a graceful-degradation gate: the scripted policy alone must pass the harness's N-step submission gate (e.g. cogames 10-step) with the LLM disabled. Local coworld play does not support Bedrock (no --use-bedrock/--secret-env/env passthrough), so exercise the LLM via submission with --use-bedrock and plan to validate LLM-backed behavior via hosted runs rather than blocking on running it locally. A per-version runtime flag is NOT proven by a shared container image (alpha_cog v13-v15 reused the same digest but USE_BEDROCK lives in the version's secret env), so trace where each flag lives per version. Don't assume runtime LLM use is forbidden; detect it by grepping source for anthropic/openai/bedrock/httpx/requests (absence proves a fully scripted policy). _{sources: claude-code:77e158b0-bd16-41f8-8473-d1b1522d451b, claude-code:7b965041-3e1e-4d77-bc3a-a48662331630, claude-code:829e80cb-6031-4b81-af72-52b3a12289f3, claude-code:940b2eb9-3677-4c05-8754-c9cfa163f531 (+4)}

15. Attach secrets and experiment knobs at upload time, not in the image

loop · negative result

Treat the image as the code and the environment as the experiment: bake only what is needed to run the policy and set secrets/behavior knobs at upload time so one image equals one codebase and the version log stays honest. Attach LLM/cloud keys ONLY via coworld upload-policy --secret-env KEY=VALUE (repeatable; stored in AWS Secrets Manager and injected at episode time); --use-bedrock additionally sets USE_BEDROCK=true. Behavior knobs (e.g. CREWBORG_BE_DUMB) also go at upload time. Make custom player modes default-OFF in the shipped image and gate them behind explicit env flags; audit Dockerfile/image markers against intended runtime state (a stale DICK_MODE=1 marker / dumb-dick image tag once contradicted the intended 'keep it off' state). Exception: operational config that merely wires the policy to its own files (e.g. SUSPECTRA_LLM_* paths) is part of policy identity and may be baked. Gate an optional meeting-LLM helper behind a flag (e.g. SUSPECTRA_LLM_MEETINGS=1) supporting both Anthropic Bedrock (USE_BEDROCK=1, tournament-injected AWS creds) and direct Anthropic (ANTHROPIC_API_KEY). _{sources: archive/cogames_playground/bulbacog/designs/COWORLD_MIGRATION.md, personal_labs/crewrift_lab/docs/designs/building_players.md, players_checkouts/players/docs/coworld-player-packaging.md, players_checkouts/players/docs/coworld-integration-guide.md (+3)}

16. Don't trust your local clone or installed CLI as authority; the deployed coworld can be ahead of public master

loop · negative result · ⚠ session-derived, unverified

Confirm you build against the game version the league actually runs: a stale cached image can mask a submission against an old version (a build pulled cached among_them v0.1.20 because the newer 403'd), and the published league image can be AHEAD of your public source clone — a fresh git pull to origin/master does NOT guarantee parity (after pulling crewrift to current master the manifest still read 0.1.40 while live replays were 0.1.48 / 0.1.24, and the freshly rebuilt helper hash-failed on both checked-in and live replays). Pull the published manifest / re-download by the game's coworld id before smoke-testing. Don't trust the installed coworld CLI as schema authority: it can lag metta's packages/coworld src and miss validation rules; diff the installed manifest_validation.py against src. Before regenerating a plan or running commands, read the CLI source and --help directly (coworld/cli.py, play.py) since the surface and flags churn faster than the policy API — grep docs for every referenced CLI command and re-run --help against each for the installed version. Distinguish BINARY version skew (0.1.13 calls /v2/episode-requests, 404 everywhere, while /episodes returns 200) from manifest SCHEMA skew (0.1.13 wants players[].name not slots[].name; 0.1.1 fails on 'tokens') by probing which routes 404 vs 200 before explaining causation. _{sources: auggie:c9f833a1-50e8-4526-8839-f438394f903e, claude-code:25488cd9-61d5-4231-bd87-e2744b6db043, claude-code:3d84ea03-c7bb-4cb2-9a06-345e1ae3c84a, claude-code:77e158b0-bd16-41f8-8473-d1b1522d451b (+5)}

17. Coworld certify/upload-coworld pitfalls: branch-matched source_url, ECR Public auth, digest-pin, every image a pullable compose service, upgrade the CLI

loop · negative result

coworld certify validates each manifest source_url against the GitHub contents API at the EXACT branch named, so a tree/main URL on a master-default repo (or vice versa) fails HTTP 404; match each repo's real default branch (gh api repos/<o>/<r> --jq .default_branch; crewrift uses master, commissioner/grader repos use main). Anonymous public.ecr.aws pulls during certify hit 'toomanyrequests' surfacing as 'image not available locally or reachable remotely'; authenticate first. coworld upload-coworld (GAME upload, not policy) runs certify internally (no skip flag), asserts every player[].image is locally reachable via docker image inspect (the certifier does NOT build images), runs the cert fixture to verify HTTP routes/token rejection/results-schema/replay, then uploads images and rewrites manifest image strings to Softmax IDs (every certification.players[].player_id must match a declared player[].id); it rejects mutable refs (:latest/untagged) demanding coworld resolve-and-upload to digest-pin rather than hand-editing tags. coworld build resolves EVERY manifest runnable via docker image inspect after only docker compose pull/build, so any grader/reporter image referenced must also be a pullable compose service (add it as an image-only service so docker compose pull --ignore-buildable fetches it; a manifest image absent from compose passes on a cached machine but fails on a clean builder). Before mutating a manifest to appease the tool, diagnose stale-CLI vs wrong-manifest: a too-old CLI (v0.1.14 crashed on a pydantic ValidationError because the server's image-upload response shape changed) should be upgraded first (uv tool install --reinstall coworld); hand-editing source_url refs to satisfy an old CLI's certify was the wrong fix and had to be hard-reverted. _{sources: claude-code:c24c7575-fe83-4650-952b-e1817c930d01, coworlds/coworld-crewrift/README.md, archive/cogames_playground/bulbacog/COGS_V_CLIPS.md, bitworld/among_them/README.md (+2)}

18. ECR manifest-HEAD 403 on push: disable Buildx provenance/SBOM, else aws ecr put-image / save+crane; isolate by changing one variable

loop · negative result · ⚠ session-derived, unverified

coworld upload-policy (Docker 29 / Buildx) hits an ECR 'HEAD 403 Forbidden' on the final image-manifest push after layers push fine; root cause is the manifest carrying a provenance/SBOM/attestation manifest, so rebuild as a plain single-platform linux/amd64 image with Buildx provenance/SBOM/attestation disabled. The failure persists across both legacy cogames and v2 coworld paths because coworld.upload shells out to plain docker push; isolate image-shape vs server-side push by changing one variable at a time (a unique name rules out dedupe, disabling provenance rules out image shape) — a hash change alone did NOT fix the 403, confirming the problem is the push path, not image shape. If it still fails, the working bypass is aws ecr put-image with the temporary upload credentials, or save the image as a tarball and push with crane, then finish the image-plus-policy upload through the API using the current coworld.upload package. Identical image content dedupes to the same upload record. Track 'built locally' and 'pushed/available' as distinct states: docker build only creates a local tag, but catalog/manifests must reference a ghcr.io/... ref and the image must be PUSHED (the graders repo defaulted build.sh to a bare among-them-grader:latest so nothing reached GHCR); do not open manifest PRs referencing tags until those tags are pushed, and verify GHCR (not just docker image ls). _{sources: codex:019e18ed-f5bd-7083-a170-7833d83d5236, codex:019e18fd-64ea-7040-a62b-f54856a9e6d8, codex:019e6696-698a-7953-b210-3f541350ce2a, codex:019e66c3-638e-7e70-afdf-8f6df9299f6a (+3)}

19. Coworld/softmax auth: separate credential stores, endpoint-scoped 401s, and verify the actual auth surface

loop · negative result

Coworld/softmax CLIs can spuriously report 'Not authenticated' even with a valid token because the standalone softmax CLI and the coworld install read DIFFERENT credential stores (coworld reads the older ~/.softmax/credentials.yaml); coworld download <game> --refresh via canonical name can hit this while refreshing directly by Coworld ID works. softmax set-token/softmax status are IRRELEVANT to what coworld uploads as — verify via coworld's own token/whoami. cogames auth status succeeding does NOT mean every endpoint accepts the token: a 401 can be endpoint-specific (e.g. the upload/presigned-URL endpoint) while season list and auth-status succeed; do NOT log out on a single 401 (that destroys a working token) — treat a localized 401 as endpoint-scoped first and only re-auth after confirming the token is broadly invalid. The 'Run softmax login first' message is misleading — there is no separate softmax binary, cogames auth login (GitHub OAuth) is all you need. For upload, reconcile the token store or bypass the CLI via the v2 API with Bearer auth. _{sources: codex:019e8afb-efcd-7ca0-820f-0320501ef2fa, opencode:ses_21faa7b9bffezLlGGJpB1MVzZS, bitworld/among_them/players/mod_talks/cogames/README.md, bitworld/among_them/players/modulabot/cogames/README.md (+1)}

20. Coworld identity: versions bind to a player at upload time; the classic player-swap footgun is two installs

loop · negative result

A Coworld user (usr_... token, scope write) owns many players (sub-identities, ply_... tokens, no scopes); leaderboard entries are per-player so one account fields several competitors. A version's owning player is stamped at upload-policy time from the active identity's subject (there is NO player_id field in the upload payload) and the binding is fixed once stamped; to submit as player P you must upload while authenticated as P (or reuse a P-bound version), and on POST /v2/league-submissions the body player_id must equal the version's owner or it is rejected ('policy version ... is already assigned to player ...'). The classic silent swap failure: a machine can have two non-interchangeable coworld installs — the bare coworld (uv tool at ~/.local/bin/coworld) and uv run coworld (project .venv); only the bare tool may support player sessions, so mint the player session AND run upload-policy with the SAME bare install (invoke as command coworld upload-policy, never uv run coworld), prefer keeping the minted token in-process and passing it directly as the Bearer header. Verify true binding the right way: a player token's whoami name may show the account owner, so trust subject_id/subject_type ('player'), not name; on GET /stats/policy-versions every row serializes player_id:null and ?mine=true is USER-scoped — the ONLY read of true binding is ?player_id=<ply_...>; a freshly-minted token may intermittently return 'anonymous' from read-replica lag (re-check; TTL ~24h). Player tokens are least-privilege (upload/submit as their own player only; cannot mint tokens, list/manage players, set default, or retire memberships — those need a user token). After a swap, restore the default by DELETING the player session (auth._delete_player_session); do NOT touch POST /players/{id}/default (unneeded, often unauthorized). Retire an unwanted membership with POST /v2/league-policy-memberships/{lpm_id}/retire (user token). _{sources: /Users/jamesboggs/.claude/skills/coworld-player-swap/SKILL.md, player_labs/.claude/skills/coworld-policy-lifecycle/SKILL.md, players_checkouts/players_1/docs/coworld-player-swap.md}

21. Treat live submissions and ownership/attribution APIs as read-only; attribute via the player's own session

loop · ⚠ session-derived, unverified

Treat a live league submission as read-only: when porting or reimplementing, build the new version as a sibling package and never edit the production submission in place; to capture ground-truth output do NOT instrument it (even a read-only JSON dump counts as modifying it), instead build a standalone CLI importing the production perception modules read-only. Do NOT trial-and-error against an ownership/attribution API: it creates stray uncleanable state (mis-attributed memberships, inert uploaded versions) since there is no delete-version endpoint, so stop creating things, read the API/auth internals, and flag external state you did not create rather than touching it. To attribute a version to a specific player, write that player's player-scoped session into coworld's credential store (mint a token via POST /players/{id}/login, save with the bundled save_player_session, upload, then restore the original identity) rather than changing the default player. _{sources: claude-code:7b965041-3e1e-4d77-bc3a-a48662331630, claude-code:91060045-312c-4787-9819-a1421d228d61, claude-code:f886e63d-1c92-4afd-bb2b-f4ff2cd9b788}

22. Confirmation isn't execution: verify the fix ran via episode logs and that the runner image carries the feature

loop · negative result · ⚠ session-derived, unverified

A submission ID confirms upload, not that the fixed code executed. Verify a hosted-submission fix took effect by pulling episode logs: coworld episodes --mine -r ROUND_ID --json, then coworld episode-logs EREQ_ID --mine, and grep for the expected startup/trace event. Decide a policy's trace/log channel by checking what the SPECIFIC runner actually captures, because runners disagree — one captured stderr only (anything on stdout or local files was lost), another scraped stdout; read the runner's capture path before choosing and target whichever channel it scrapes from the start. Likewise a merged feature PR does not guarantee the runner image ships it (a per-player artifact-upload feature landed in a metta PR but the coworld-runner image lacked it, needing a follow-up build); verify the runner IMAGE carries the feature and fire a FRESH probe episode since already-running pods predate the change. A league 'champion' flag can mean top-N not first (minimum_champions: 8 means is_champion top-8, and a new submission can displace your own prior one), so read the league config before celebrating. _{sources: auggie:ac491a46-e7f7-4271-a1c0-f0e69987d424, claude-code:d48f9347-7884-4ea2-8659-230051d86410, claude-code:f70c9801-7ee7-499b-97f9-4fd0848a6b8e, claude-code:b2b2f1e5-537c-4acf-abfb-4490b0abc27a}

23. 'Timed out waiting for agents' usually means the player never connected; diagnose with a log ladder, not skip-validation

loop · negative result

'Server timed out waiting for agents to connect' usually means the player process never reached the websocket connect step, not that the bot is slow. Diagnostic ladder: check submission status, read the job's raw artifacts, look for policy-logs. If policy-logs is empty and the debug zip has only game.stdout.log ('waiting for players: 0/8') with zero policy_agent_N.txt files, no player log of any kind means the player containers never started or were rejected before emitting stdout — points at orchestration/scheduling (service-account/pod admission), not bot code; a crash or wrong-URL bug would instead leave a player-side traceback or connect log. Server-side validation runs against the DEFAULT SEASON's pinned pool config even with --no-submit or an explicit --image, installing the framework from the season's pool requirements (so --image latest does NOT fix a version mismatch); a newer-framework policy fails on a ModuleNotFoundError that is not a bot bug — localize by comparing against an already-shipped sibling using the identical import. The dry-run/Docker gate is only ~10 hard-coded steps and rejects any bundle emitting no non-noop action, so a perception/localization bot that idles (action 0) during warm-up FAILS while healthy. Use --skip-validation ONLY for a documented benign failure (too-short gate, or no matching season/pool), never to paper over a real Nim build/import/ABI bug, and only when the same bundle was already dry-run-validated; surface the residual risk and get explicit human consent first. _{sources: codex:019e2886-d576-7770-be1b-6d92ca97761e, codex:019dfece-78f8-7722-8301-f4e0fafc2cd9, opencode:ses_22085eca7ffeS682vQ9JkzWot0, bitworld/among_them/players/how_to_submit_coworld_policy.md (+8)}

24. Verify a season is live and matches your game before investing; submission is not local validation

loop · negative result

Game/season availability is volatile: a target season can be a phantom (API 404 even when listed) or vanish mid-project. Before investing, verify the target season is actually live (e.g. cogames season list) AND that it matches your player's game (don't submit an AmongThem bot to a beta-cvc/beta-teams season); if no matching live season exists, surface the blocker and pivot to local match testing rather than forcing a meaningless submission — treat season churn as expected, not an auth bug. Promoting/validating a policy version locally (validate + historical tournament + commit) does NOT submit it: submission is a separate explicit step (e.g. experiments/ship_to_league.py ship) run OUTSIDE the sandbox because it builds/uploads a Docker bundle needing the league path and network I/O (a 'policy localized and ran successfully inside the episode runner' message during upload is the good sign the bundle validated); a full ship is not done until submitted to the default league seasons with a status check and ledger entry. After a real submission teaches a correction, update the how-to-submit doc in the same task (live corrections: a season can accept submissions while showing non-active, and the local Docker validator can fail at player-connection time before the policy runs). In the cogames CLI 'submit' means attach-a-policy-to-a-season; treat an ambiguous 'submit the bot as ' for a game with no public season as upload-without-attaching (cogames upload ... --no-submit) and confirm before any irreversible server action. _{sources: opencode:ses_21af0e707ffeWhoJeouK9agHhn, opencode:ses_21b899dadffeBiSiDW3XWFqEdz, opencode:ses_21ffeb5ffffeA1Pk6sdyHDzwQW, opencode:ses_22033c323ffeaukf7T2ZfJOoMR (+8)}

25. Include EVERY source dependency in cogames -f; diagnose packaging errors by message

loop · negative result

When uploading a CoGames policy whose native build pulls transitive source deps, include EVERY on-disk dependency in the -f file list because the build script runs inside the validation/worker Docker image and needs them on disk; for the Nim modulabot/mod_talks bot that means among_them/sim.nim, common/, src/bitworld/, and nimby.lock (a missing nimby.lock causes a source-dir-not-found / Nim build failure in the worker). Diagnose by message: 'Could not locate modulabot source directory' = wrong cwd or a missing source -f include; 'does not export an ABI version' = build predating the ABI change, rebuild clean; 'has ABI version N, expected M' = stale .dylib, delete libmodulabot.* and let the wrapper rebuild. When validation passes but matches still fail, pull logs with cogames matches <id> --logs and cogames match-artifacts <id> logs|error-info. _{sources: bitworld/among_them/players/mod_talks/cogames/README.md, bitworld/among_them/players/modulabot/cogames/README.md, opencode:ses_219e032f9ffeqy8OzInbZDP27R}

26. Packaging a native (Nim/FFI) policy: portable paths, gated exports, ABI checks, release builds, ctypes wrapper

loop · tool: ship.sh

To make a native-backed (Nim) policy portable outside its monorepo, vendor the shared library next to the package and resolve its path env-first with one consistent chain in every module: an env override (e.g. $AMONG_THEM_PLAYERS_DIR / BITWORLD_REPO_PATH), then a shipped vendor/native/ dir, then a walk-up to the monorepo layout (resolve the repo root by walking up until nimby.lock); remove hard-coded absolute paths. Gate the FFI exports behind a Nim compile-time define (e.g. -d:evidencebotV3Library) so one source serves both the standalone CLI build (omits the define) and the FFI library build with no symbol collision; compile nim c --app:lib -d:<libDefine> --out:<lib> --path:<common> --path:<src> <bot.nim>, maintain variants as independent compile units, ship the toolchain (Nim+nimby) pre-installed in the worker image so it compiles natively (no cross-compilation). Package as a Python policy with a thin ctypes wrapper that keeps the source one dir up: it locates the source tree, imports the build script via importlib by file path (the bundle has no init.py), compiles the .so on demand when missing or when the ABI stamp mismatches, then loads via ctypes and routes step_batch to the native step. Write an explicit ABI-version stamp next to the library at build time and refuse to load on mismatch; bump BOTH sides together when the FFI surface changes (ffi/lib.nim:ModulabotAbiVersion and build_*.py:MODULABOT_ABI_VERSION). The entry module must locate source in BOTH the in-repo AND flattened-bundle layout (search source FIRST so an in-repo run never picks up a stale bundled copy; raise with searched paths when not found); for a bundled native binary, re-add the +x bit idempotently on load (zip extraction drops it). Verify a fresh build: confirm the export symbols are actually present (nm) — a clean compile does not prove the ABI symbols the runtime loads are exported. Build competitive bots with -d:release (debug ~5x slower, drops frames under heavy per-frame work; verify via config.nims), make phase 0 a compilable empty-stub skeleton the host wrapper can load, and raise an actionable typed error (FFIError) naming a missing native dep. To ship, run the bot's ship.sh with SEASON/POLICY_NAME (dry-run then ship); prefer a no-ABI-change fix over a new FFI argument. NEVER commit FFI/native build artifacts — gitignore them per-player (a build script once left libmodulabot.dylib ~900KB and its .abi stamp untracked). _{sources: claude-code:eb751ed4-c198-4b0d-819e-1a3002f4f341, codex:019dfece-78f8-7722-8301-f4e0fafc2cd9, codex:019e05ac-239c-7bb0-8e34-1ec371fbabcd, bitworld/among_them/players/evidencebot_v3/build_evidencebot_v3.py (+18)}

27. Ship discipline: default-off flags, two-commit activation, regression identity, frozen snapshots

loop

In a scripted policy whose 'versions' (V1..V7) are progressive flag configs of one class (e.g. AlphaPolicyFull), make every optional behavior default OFF and preserve a regression guarantee: with all flags off the policy must be action-identical to the original starter on every fixed seed. Ship a winning flag with a two-commit pattern — commit 1 lands the implementation (default off, a submission no-op), commit 2 activates it and records the A/B command and result — so git-blame on the activation line recovers provenance and reverting commit 2 cleanly disables it. Every ship that flips an enable_* flag must add a new AlphaFullV historical-snapshot class in the SAME commit, freezing the shipped config for later A/B. Ship a validated change as two commits: an 'experiment:' commit recording the A/B result, then a 'ship:' commit promoting the flag into the canonical policy and snapshotting a historical version (add to historical_policies.py, update docstring/AGENTS.md/strategy doc, regenerate the index, all together). Keep ship commits to one logical change (do not bundle pre-existing uncommitted tooling), with messages leading with version then experiment id, e.g. 'Ship V7: goal-aware stuck recovery (E044, T005)'. For a focused non-shipping worktree patch, SKIP the submission-log and mission-doc steps. _{sources: opencode:ses_21a181ab8ffe3nrttiVWHpM4LJ, opencode:ses_1ff344748ffeIooJBfFGRt6YG0, opencode:ses_1ffd53990ffeekLhAYVO14YLK9, opencode:ses_219f1f7a8ffeQVQYqola5M5a9n (+5)}

28. Make the ship tool guard cleanliness, version-match, and docs as a non-negotiable sequence with backup/restore

loop · tool: ship_to_league.py

Have the league submission tool refuse to ship from a dirty git tree unless an explicit override is passed; when overridden, record dirty-tree provenance (dirty-diff hash) into the ledger and notebook and quarantine the submission as non-canonical. Also require the internal version to match the current shipped/historical snapshot, record the validation result, and prompt for a matching notebook entry. The full alpha_cog ship sequence is fixed: create hypothesis+experiment files, flip the flag on AlphaPolicyFull, add the previous version to experiments/historical_policies.py, update the docstring and AGENTS.md (description + shipped-flags list), regenerate the notebook index, submit via experiments/ship_to_league.py, then commit. In a ship/orchestration pipeline check all external prerequisites UP FRONT (required binaries on PATH, docker daemon running, an authenticated league session reporting Authenticated), aborting early with the exact remediation command rather than failing midway. Before overwriting a live champion artifact, back it up to a .bak and register a restore() that reverts on any subsequent failure so a candidate failing a late gate does not leave a broken artifact in place. Make end-of-pipeline bookkeeping (version-log append, git commit) best-effort: a dirty tree or held lock should log a WARNING but never block the actual upload/submit. _{sources: alpha_cog/audits/research-audit-2026-05-04.md, alpha_cog/experiments/README.md, alpha_cog/experiments/notebook/protocol.md, opencode:ses_2042c1ff1ffefETImBU5n4D43k (+3)}

29. Thread a new alpha_cog tunable through four locations or it is dead

loop · ⚠ session-derived, unverified

Adding a tunable to the alpha_cog cogs-vs-clips policy requires four coordinated edits or the flag has no effect: the PolicyConfig dataclass field in alpha_cog/v0/config.py (and its summary() dict), the boost/decision logic in v0/stages/decide.py, the ClassVar on AlphaPolicy in v0/policy.py wired into _build_config(), and a test subclass in experiments/ab_policies.py. Defining a new A/B subclass that references a base (e.g. AlphaPolicyFull) before that base is imported fails at class-definition time — promote the base into the module's top-level imports (a redundant later import is harmless). _{sources: opencode:ses_1ff344748ffeIooJBfFGRt6YG0, opencode:ses_1ffd53990ffeekLhAYVO14YLK9}

30. alpha_cog experiments/ tooling conventions for new tools

loop · ⚠ session-derived, unverified

Match alpha_cog experiments/ conventions in new tools: Python stdlib argparse only (no typer/click); single-purpose tools (ab.py, trace.py, regression_check.py) use a flat parser, multi-command tools (ship_to_league.py, tournament.py) use add_subparsers(dest='command', required=True) dispatching via if args.command == 'ship': return cmd_ship(args); every tool ends with if __name__ == '__main__': sys.exit(main()) returning an int. Cross-package imports use a repo-root sys.path shim at file top (_REPO_ROOT = Path(file).resolve().parent.parent inserted into sys.path), so run via uv run python experiments/<tool>.py, not as installed entry points. _{sources: opencode:ses_2094d724cffebAsFmXFHGyRB7d}

31. tutorial make-policy is a near-verbatim copy; scripted classes need explicit class= paths

loop · negative result · ⚠ session-derived, unverified · tool: cogames

cogames tutorial make-policy --scripted -o <file> is not a generator: it shutil.copy2's starter_agent.py (carrying its side effects) then textually strips the short_names = ["starter"] line. When that's the class body's only statement, the result is an empty class header that raises IndentationError at import — minimal fix is adding pass, better is adding a unique short_name. A resulting scripted class (e.g. scripted_alpha.StarterPolicy) will NOT show in cogames policies (it's a class path, not a registered shorthand); run it via -p class=<module>.StarterPolicy, and the module name must match the generated filename. A policy that 'runs locally' is not the same as one that can be submitted: before calling something a starter/submittable policy, verify an end-to-end submission path exists — the cogames upload --policy "class=..." resolution must point at a real importable class (the documented default starter evidencebot_v2 had NO submittable wrapper at all). _{sources: opencode:ses_225895cdaffe6ZHf9B5LPZFJsZ, opencode:ses_223dc7136ffeCr4ZslnR426sVG, opencode:ses_219e032f9ffeqy8OzInbZDP27R}

32. Build cogames -p policy-spec flags by reference type; pass variants via -v/--variant

loop · ⚠ session-derived, unverified · tool: cogames

For cogames play -p mixed-team A/B specs the syntax differs by reference type: a registered shorthand (no dots, e.g. starter, random, noop) uses name:N (starter:2), while a full class path (has dots) uses class=module.Class,proportion=N. Select by detecting a dot, but prefer an explicit class= prefix when the caller supplied one before falling back to dot-detection. Pass mission-variant modifiers (e.g. angry_clips, adaptive_clips, caves) as -v <variant> appended to the command and expose them via a --variant flag rather than baking the variant into the mission name. _{sources: codex:019e037d-89e9-72b2-898b-b1f254de2faf}

33. Split player assets into baked runtime artifacts vs offline generators, and pin only deterministic ones

loop · negative result · ⚠ session-derived, unverified

Structure a game player as a baked/ directory of static runtime artifacts (walk/wall masks, map pixels, sprites, palette, font, waypoint graph, path data) separate from the offline tools that generate them; the runtime import path must depend ONLY on baked outputs so generators can carry heavyweight deps (pygame/numpy) that never enter the runtime dependency set. Add any new baked-asset extension to pyproject.toml package-data in the SAME commit (a list naming **/.json but not **/.npz silently omits the asset from the wheel, so local tests pass but the player breaks in Coworld). Before pinning a binary asset's SHA256 in a digest-drift invariant, re-run the generator and diff bytes: np.savez embeds zipfile mtimes and is non-deterministic, so the pin false-alarms. _{sources: auggie:02627f61-aac0-42a0-a290-3595e010abba, auggie:5d1ff8d5-2505-4dca-af45-56d3070363e2}

34. Lay out graders as self-contained dirs; group into families; count only real graders; upload.sh builds all roles

loop · negative result · ⚠ session-derived, unverified

Lay out each Coworld grader as a self-contained directory (Dockerfile, .dockerignore, README, build.sh, _grader.py) plus a root CATALOG.yaml entry and tests/ using SYNTHETIC episode bundles; a grader image is a tiny python:3.13-slim build run post-episode with flat, dependency-light scoring (~10 lines over a results dict). Avoid many near-identical surfaces by grouping coworlds into grader families by scoring shape, plus per-world graders where the shape differs (e.g. crewrift needs its own graders-crewrift, NOT the generic bitworld scorer - inspect the world's actual grader logic, not defaults). When auditing coverage, count only graders with real scoring logic, tests, AND a catalog entry; derive test expectations from the implementation's deterministic output, not hand-computed intuition. worlds/upload.sh builds EVERY role image in the manifest (game, player, reporter, grader, diagnoser, optimizer), so a build failure in ANY unrelated role blocks the whole upload; it refuses a dirty tree and expects sibling source repos per role - use a DISPOSABLE clone for throwaway cross-role fixes, pull every cross-repo checkout first, pass explicit *_CONTEXT paths on non-default layouts. Sequence so all offline work (builds, manifest edits, validation, local commits) finishes first and only the gated GHCR push plus PR remain at the end. _{sources: codex:019e66a2-b825-7181-9a3d-90435088068b, codex:019e66c3-638e-7e70-afdf-8f6df9299f6a, codex:019e746d-062c-78d3-a0ed-ef2fcb892baf, codex:019e74bf-7764-7fd2-82e5-8f66c5ff9e1d}

35. Keep a new agent additive and contract-respecting, scoped to its own self-contained submission bundle

loop · ⚠ session-derived, unverified

A new agent's policy entry point should follow the repo's existing contract EXACTLY (module-level AGENT_ID, DESCRIPTION, --url/--name args, a run() function) so submission/run tooling treats it uniformly; copy the baseline agent's shape rather than inventing one. Each per-agent directory is a self-contained submission bundle, so keep a change scoped strictly to the target block and the agent's own directory: do not fold in unrelated uncommitted hunks, do not leak edits into shared dirs or other agents' dirs (leakage makes the diff unreviewable, can clobber in-progress work, and breaks the submission boundary), and keep the submodule additive and contract-respecting (produce the existing shared dataclasses, consume the existing shared enums) without pulling in design docs or refactoring neighbors. When a repo hosts multiple game agents, give each its own subdirectory so packaging, tests, manifests, and Dockerfiles stay isolated. _{sources: codex:019e038f-b411-7a01-b409-b9c4a04270c6, codex:019e04f4-c141-7e62-9ac5-47c8bbcb2651, codex:019e05c4-f9e2-74d0-bcd3-53bbf33c209b, codex:019e090d-a88d-76f2-b1f5-cd3e4eda9381}

36. Scope an integration pass to operations, not strategy

loop · ⚠ session-derived, unverified

When the goal is getting a Coworld/cogames player onto live tournament servers, scope the work as operations/integration: fix anything blocking build/package/connect/complete-episode (Dockerfile, entrypoint, build.sh, manifest, perception/action crashes, env wiring like COGAMES_ENGINE_WS_URL, deps), and explicitly do NOT touch mode/strategy/intent-selection. 'Plays badly' (idles, wanders, loses) is out of scope for an integration pass. _{sources: claude-code:f196c6ef-01ee-461d-b240-e21e96b7c038}

37. State the full path when 'modifying coworld'; the platform/runner is read-only, the player's bridge subdir is yours

loop

Treat the Coworld platform packages (Metta-AI/metta packages/coworld) as READ-ONLY reference and never write there; runner.py is the protocol authority for the packaging/runner/CLI contract (read it only to learn the runtime contract). A player repo (e.g. crewborg) OWNS its own coworld/ bridge subdirectory (crewborg/coworld/policy_player.py) which it may edit. State the full path when discussing 'modifying coworld' — the bare phrase is ambiguous and triggered a human stop. The observe-protocol workflow pairs with the build-submit-policy workflow. _{sources: claude-code:0a2634a1-19b7-402e-841e-084ce80c015f, personal_labs/crewrift_lab/crewrift/crewborg/AGENTS.md, archive/cogames_playground/bulbacog/designs/COWORLD_MIGRATION.md, metta/agent-plugins/kitchensink/skills/ks.coworld-player-gotchas/SKILL}

38. Design the container to serve both the coworld and legacy BitWorld contracts

loop · negative result · ⚠ session-derived, unverified

Two runner contracts can target the same image, so build the entrypoint to serve both: the Coworld upload path expects a default command that reads an engine websocket URL from env (COGAMES_ENGINE_WS_URL), while the older BitWorld server discovers a coplayer_manifest.json and invokes 'image binary --address --port --name --slot --token'. Handle both raw BitWorld /player frames AND a JSON coworld.player.v1 adapter, and ship a coplayer_manifest.json pointing at the binary; local player discovery (players//.json) and league submission (a policy-class bundle: -p class=..., repeated -f includes, --season) are separate systems both needing wiring. A crewrift-style Nim player reads only COGAME_ env vars (COGAME_HOST, COGAME_PORT, COGAME_CONFIG_URI, COGAME_RESULTS_URI, COGAME_SAVE_REPLAY_URI, ...) and ignores CLI --address/--port. _{sources: codex:019dfece-78f8-7722-8301-f4e0fafc2cd9, codex:019e184d-2de6-7b22-b8c0-853edbf53265, claude-code:a5973ebb-5fd5-4158-8d61-0480948c75a9}

39. Prove the no-LLM fallback path and exercise the production wrapper shape

loop · negative result

Prove a Coworld player is playable in its fallback (no-LLM) path BEFORE attempting a league submission, because a player that only works with the LLM online is not submittable. Write a fixture-replay test with no API key that asserts a non-NOOP action within the first N frames (to pass the platform's action-validation gate), asserts mode transitions occur, and asserts no crash through the full fixture. Two runtime paths for the 'same' player silently diverge: when a player can launch as a compiled CLI binary OR a Python policy wrapper, verify which capabilities each path exercises — in one case LLM dispatch lived only in the Python wrapper, so the Nim CLI binary connected via --address but produced a non-LLM bot. Tournament submission instantiates your policy class inside the platform's own worker (the bot never opens its own sockets there), so a policy that only works because a local harness opens sockets still submits fine but can't be debugged that way; keep a local harness that exercises the EXACT same step_batch path the tournament uses, including the production concurrency model (e.g. the ThreadPoolExecutor in amongthem_policy.py). _{sources: opencode:ses_21b02c437ffeor4lV5Nb4Cec7y, opencode:ses_21b1e7074ffeYT9RjQFHNPlNKX, opencode:ses_21e7dd4d4ffewtISu38YXrsOgs, opencode:ses_21b899dadffeBiSiDW3XWFqEdz (+1)}

40. Build a multi-mode policy in dependency order, gate-first, with a resolve_policy seam

loop · ⚠ session-derived, unverified

Build a multi-mode policy in dependency order so each step is independently validatable, getting the action/navigation layer plus one mode past the submission validation gate before adding strategy depth: (1) navigation foundation (A* pathfinding plus action/button-mask generation everything depends on), (2) the default crewmate mode just good enough to pass the validation/dry-run gate, (3) a meeting/vote fallback, (4) reflex wiring, (5) role-specific offensive modes, (6) the modes reflexes route to (reporting, fleeing), then tests and docs. Give every player-run harness a generic -p/--policy class=... interface backed by a resolve_policy helper that resolves a fully-qualified class at runtime, with the project's default policy as the argparse default so the bare command still works (a harness hardcoding one class cannot run any other without editing source); document the default explicitly so omitting -p is not a hidden surprise. _{sources: opencode:ses_21b475c59ffeOFBSt0jbwsGras, opencode:ses_21b46167dffePyHuN6mEYBKwfS}

41. Verify a run pipeline end-to-end and write an import-only smoke test before logic

loop · ⚠ session-derived, unverified

Verify a run pipeline end-to-end before documenting it: compile the server, compile the player, start the server with a real config, connect the bots, and confirm a game actually STARTS; wrap runs in timeout N so a never-ending game doesn't hang the harness, and test both a minimal config (1 player, 0 imposters) and the full config (8 players, 2 imposters) so the recipe is proven across trivial and realistic cases. When scaffolding a new framework-based player, write an import-only smoke test (a single agent.py importing every public component — pipeline, perception, belief, buffers, modes, tasks, spatial/pathfinding, hooks, logging, types — plus a smoke_test() instantiating key classes) BEFORE writing logic, to catch packaging/import-path problems while cheap. Run the import smoke and targeted pytest from the EXACT invocation path/cwd the bundle ships under (the parent repo dir, or PYTHONPATH=<agent_dir> against an isolated venv), not from a convenient directory: bundles often have import surfaces that only resolve from a specific root, and generated tests using bare relative imports fail under repo-root collection — generate tests with explicit sys.path setup, namespace generated test filenames per game (test_protocol.py) to avoid 'import file mismatch' collisions, and clean stale pycache when regenerating. _{sources: opencode:ses_200cdca2fffeNC2dxiT8W9quAt, opencode:ses_1fbfc5c8fffesCp5Ilc2Hs3Q6X, codex:019e012e-626f-7081-91fb-fccc04e39010, codex:019e067b-643a-7100-abdb-8d6a0c79d242 (+1)}