ab-methodology

name	ab-methodology
description	Use for ab methodology recipes in scripted Coworld policy optimization.

A/B methodology & attribution — recipes (loop tier)

On-demand recipes (13). Trigger→action heuristics; pull the relevant one when its situation arises.

1. Run split-team A/B with a starter partner-proxy; judge per-split and target the weak-partner case

loop · tool: experiments/ab.py

For split-team validation use experiments/ab.py --split 6+2/4+4/2+6 --partner starter: the upstream 'starter' baseline is a competent-but-median partner proxy, whereas a 'random' policy that consumes all shared gear/stations is a poor one. Always break tournament results down by split before concluding a change is good — replay analysis showed 6+2 mean 29.1 > 2+6 23.8 > 4+4 16.8 (4+4 maxes partner-reliance and is where catastrophic failures cluster), and a change can improve 6+2 while regressing 2+6/4+4 so an aggregate delta can hide a sign flip. Score against STRONG partners is largely policy-independent (a 5-gen-old version scored within ~7 points of the newest when paired with leaders) while score against WEAK partners is heavily policy-dependent, so iterate on the weak-partner floor; tournament random partners can be worse than the local 'random' policy (tournament 4+4 mean 22.2 < local 4Full+4random 30.52), meaning real partners actively obstruct. COST: split-team breaks the shared-baseline optimization (each candidate needs its own baseline per split), so a 12-seed 3-split run is ~3x a standard 8+0 A/B. _{sources: opencode:ses_1fc9997c7ffe2ocHawVyzj5nrh, codex:019e037d-89e9-72b2-898b-b1f254de2faf, alpha_cog/AGENTS.md, archive/cogames_playground/alpha_cog/docs/cogs-v-clips-strategy.md (+4)}

2. Validate on an escalating ladder; the cheap discovery run is a smoke test, never a ship gate

loop · tool: experiments/ab.py

Use a staged pipeline that centers tournament signal: (1) import check uv run python -c "from experiments.ab_policies import <Subclass>; print('OK')"; (2) regression check that flags-off is byte-identical to baseline; (3) single-seed smoke run uv run cogames play -m machina_1 -p class=experiments.ab_policies.<Subclass> --seed 42 -s 500 -r none; (4) cheap discovery A/B (2000 steps, 12 seeds, -r none) as a SMOKE TEST only; (5) mixed-team A/B against a starter partner; (6) the heavy promotion gate of ~50 seeds at the full 10000-tick horizon plus a holdout; then tournament submission + replay analysis as the PRIMARY signal. A 6-10-seed 10k panel is a cheap inner gate; a 50-seed 10k run is the real promotion gate. Treating a 12-seed/2k win as ship-ready is the documented methodology drift to avoid — and reverting from mixed-team A/B back to cheap 8+0 self-play for later ships brought the predictive failures straight back. Even a positive 50-seed mean (+0.204, 11W/34T/5L) with a large new loss tail (-10.39, -8.83) is only 'pass with caution': trace the worst new regression before promoting. _{sources: codex:019df3f0-db41-7561-8184-8832002dba41, codex:019df40f-3672-73c0-8d5e-13abe5edd0f3, opencode:ses_1ffa96717ffeq1J1GeKsUpSp3k, opencode:ses_1ffd53990ffeekLhAYVO14YLK9 (+2)}

3. Never decide on one game or one seed; read the full per-seed distribution and respect noise

loop

Per-player score std dev (~40-53, or single-seed swings of +/-0.5-1.5 on a mean of ~1.9) is routinely LARGER than the mean, and within-game variance, role/seat asymmetry, and opponent dependence each swamp any single result — believe a result only after 5+ seeds. Use a quick set (3 seeds 42/43/44) for fast iteration, 10+ before committing, with an explicit regression threshold, and judge on the full per-seed distribution: win/tie/loss counts (e.g. 34W/8T/8L), per-seed spread, and worst-case regression, not just the mean. Score is concentrated (top ~10 of 50 seeds = 50% of total), so a positive mean can be driven by a few high-variance seeds or outlier rescues while materially regressing others (a stuck-recovery variant: seed 13 +12.13, seed 777 +6.03, but seed 99 -8.10, seed 512 -5.58). A mean-neutral or matched-aggregate verdict can hide a single shared root cause splitting wins and losses, so cluster the losers before concluding 'neutral' (in an A* regression the 6 losing seeds were all agent-dense chokepoint maps, the 6 winners less-congested hubs); when aggregates match but you need the mechanism, do per-tick per-agent trace diffing on a narrow per-seed target. At small episode counts trust mechanism metrics over the headline win-rate: at n=40/arm a +13pp crew-win delta sat at p0.22 while votes-at-crew (0.88->0.05), own-ejection (52%->2%) and kills moved at p<0.01; rate metrics need a few hundred appearances per side, so a borderline delta is not a win — collect more episodes or sharpen the metric. With only n=3 10k episodes a 2->1 imposter-kill drop is flat-to-regressing, and the insufficient sample is itself the finding. When the win axis is matchup-saturated (everyone wins ~100% as imposter, ~0-5% as crew vs a fixed pool), win rate cannot rank policies — fall back to per-role score, tasks, and kills per game. _{sources: codex:019e184d-2de6-7b22-b8c0-853edbf53265, codex:019e1579-a737-7b70-bf0e-172ff04307cf, codex:019e19a7-8605-7992-9b28-cb4d2ca9132e, claude-code:0e7b14ca-ffd5-4137-9456-47485d3c6f87 (+11)}

4. Add every behavior as a default-off A/B flag; never read a verdict off a bundle

loop

Add each new behavior as a default-off flag so the live policy (AlphaPolicyFull) is untouched: add a PolicyConfig field defaulting to False, add the matching enable_* attribute on AlphaPolicy, wire it through, and make the off path reproduce starter-equivalent (byte-identical) behavior (an LLM/control hook's off path returns None so deterministic logic is bit-identical; compile-guard an LLM layer behind e.g. -d:modTalksLlm or --no-llm so off==proven baseline). A/B every flag against BOTH baselines in one multi-candidate pass (baseline runs are shared so wall-clock is barely higher): first the bare all-flags-off substrate (lose here => net-negative, reject) then the current shipped Full+flag (must beat current-Full, not merely substrate). Never reject on the isolation run alone (an additive flag may only express value atop other winners, E009) and never read a verdict off a bundle: split bundled flags into single-flag A/Bs (hub-affordability-gating bundled with deficit-mining read +0.095 but isolated the gating dragged down mining's true +0.49; deficit-mining +0.34 plus vibe-coordination netted ~+0.02 because deficit-mining narrows all miners to one extractor, leaving vibe nothing to spread). Always also test the final combination. Land a winner with the two-commit pattern (commit 1 = flag implemented default-False, commit 2 = flip it on + docs) for a clean one-commit revert, and keep regressing flags implemented-but-disabled rather than deleting them. _{sources: opencode:ses_1fb283644ffeinY21ADUyoWOcs, codex:019df40f-3672-73c0-8d5e-13abe5edd0f3, claude-code:aae25940-ade3-41bc-b817-517dc86ccebf, opencode:ses_1ff344748ffeIooJBfFGRt6YG0 (+12)}

5. One target axis, one change at a time; pair quantitative deltas with qualitative trace review and a fresh head-to-head

loop

Frame each A/B around the single metric the change was meant to move (e.g. kills_mean, or shrinking imposter_no_kills_rate), state the expected direction up front, and change exactly one upstream thing per run or the delta is unattributable. A quantitative engine tells you WHETHER a metric moved but not WHY and misses patterns no column captures — pair it with a qualitative side-by-side of both batches' event timelines and the policy's own logs, steered by a chosen lens. To decide 'did v2 beat v1 right now', run a fresh, matched head-to-head A/B on the same seeds/field rather than eyeballing two separately-generated descriptive batch reports (those find signals but cannot decide whether a change helped). _{sources: personal_labs/crewrift_lab/.claude/skills/crewrift-ab/SKILL.md, personal_labs/crewrift_lab/AGENTS.md, personal_labs/crewrift_lab/.claude/skills/crewrift-report/SKILL.md, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/polic}

6. Use hosted experience requests as the primary eval; target each to its question, turn off backfill, and audit fairness

loop

Treat hosted Coworld experience (XP) requests (parallel batches on Softmax infra, currently free) as the PRIMARY evaluation instrument and use them liberally, but target each to the question: matched roles when the change was role-specific, the specific opponents the policy struggles against, heavy tracing on, num_episodes (1-100) high enough to smooth variance. The request body (V2CreateExperienceRequestRequest) defines the experiment — target (division/league or coworld_id), roster (requester+top_n opponents for non-owned fields, or caller-owned policy_version_ids for explicit rosters), per-slot role/seat overrides (game_config_overrides, rotate_seats). Seat count is fixed by the world manifest; empty seats get an opt-out 'backfill' of active high performers, and 'backfill' and explicit per-seat 'assignments' are MUTUALLY EXCLUSIVE — for a deterministic controlled A/B you must turn backfill OFF and cover every seat (one policy may be reused for all); you cannot half-specify and let backfill complete it. Round-robin opponent fairness is NOT automatic at small episode counts — a forced 50/50 role split and full opponent coverage can be jointly infeasible (with 16 other players, 7 opponent seats/game, only 10 episodes), so compute feasibility and surface the tradeoff; realized appearances are uneven (across 20 fixed-slot rotations opponents appeared 2 to 14 times), so print coverage and an appearance histogram instead of trusting the intended scheme, and rotate opponent slot assignments across requests (imposter evals fixed slots 0/1 as imposters but rotated the co-imposter partner across the top-7). After creating, treat 'created' != 'done': read it back (accounting for replica lag), verify it resolved as intended (episode_count, slot 0 = requester, expected opponents), monitor the xreq_ until every child episode is terminal, then DOWNLOAD via the episode-artifacts skill and compute win rate plus score mean/median/quartiles/std broken down by player and role. Harvest asynchronously. _{sources: player_labs/AGENTS.md, player_labs/best_practices.md, player_labs/.claude/skills/coworld-experience-requests/SKILL.md, player_labs/.claude/skills/coworld-episode-artifacts/SKILL.md (+4)}

7. Quantify the strategy from replays first, then validate fixes by behavioral metrics on a fixed, fully-seeded scenario

loop · ⚠ session-derived, unverified

Quantify a strategy from a replay corpus BEFORE coding it to turn 'whether' into 'where' (a button-runner-intercept study over ~1,890 episodes found button calls in 92.5% of games, mean 2.15/game, median call tick 1688 — the bottleneck was where, not whether). Game-bot replays can be bit-exact on the SIMULATOR side (input log + per-tick state hash) yet not reproducible on the POLICY side if the brain consumes wall-clock LLM calls and a time-seeded RNG — so explicitly seed ALL policy-side randomness and hold seed and agent spawn fixed across A/B phases (with positioning unchanged, tasks-visited going {0,2}->{11,19,21,22} is a clean before/after signal). Validate motion/behavior fixes with quantitative trace metrics, not 'looks better': measure direction-reversal rate (true wobble ~ a reversal every other frame, ~1-in-9 reversals are legitimate corner turns), noop rate against a threshold, and for a goal-switching feature confirm the explicit trigger ratio (new_dist < 0.5*old_dist fired) rather than assuming any goal change proves it. Match your comparison metric to what the trace actually records: a trace logging only branch transitions makes per-tick rate metrics meaningless, so compare the set of task indices visited and whether a branch fired. Hold the entire scenario constant (same seed, params, agent count, duration, port) and compare before/after BEHAVIORAL metrics (idle fraction, mode-transition counts, tasks completed), separating 'the policy changed' from 'the game went differently' — two runs of the 'same' scenario diverge because filler bots produce different states, so treat differing OUTCOMES as noise (tracking a clean 0/8->5/8->7/8 progression on identical seed-42 turned trace noise into per-fix signal). Some correct-by-construction fixes are not trace-visible in short runs (negative-evidence pruning needed ~1s of walking past unassigned tasks; confirmation cycles needed >60s; none visible in 20-30s) — inspect trace granularity before defining acceptance criteria and say so when a fix won't show in a short run. Before interpreting a trace pair, confirm the trace's reward delta reproduces the recorded loss (V8 19.82 vs candidate 9.43); if it differs materially, STOP — you are tracing the wrong configs. _{sources: claude-code:ef964049-4e16-4c32-b7f1-32871349d20a, opencode:ses_1fc949520ffe2UNDsOqbbL5T9a, opencode:ses_1fc91aa93ffeMtKpKXfQ6IA01y, claude-code:067e7439-cbd1-4b26-9b3f-d054bde3aa40 (+7)}

8. Periodically run a chronological round-robin tournament to catch the local-meta trap

loop

A/B only compares against today's floor, which misses the 'local-meta trap': a chain of ships each beating the previous floor can still leave the current policy worse than a two-versions-ago ancestor on the same seeds. Periodically — ideally after every ship — run a chronological round-robin tournament over the full roster (substrate floor plus every prior shipped version plus current) as an anti-regression gate the A/B harness alone cannot surface. A per-ancestor tournament also exposes persistent per-seed regressions a global gate glosses over (a K=1 'everyone to one resource' policy beats K=2 spread by ~1.37 on one map's geometry) — treat a reproducible per-seed regression as a load-bearing fingerprint to investigate, not noise to average away. _{sources: claude-code:425dc11e-1c8d-4d12-bf44-adf01b6fa796, alpha_cog/experiments/README.md, archive/cogames_playground/alpha_cog/experiments/notebook/protocol.md, archive/cogames_playground/alpha_cog/experiments/notebook/tournaments/}

9. Mine rivals and unread observation features for new axes; log every submission delta

loop · ⚠ session-derived, unverified

Treat a higher-scoring competitor policy as a mine of transferable ideas, not a black box to beat: enumerate the rival's concrete advantages (shared coordinator object, frontier-aware junction scoring, dynamic role switching) as candidate features to port, turning 'be better' into an actionable backlog. The unread observation features are another high-value axis — when a scripted policy reads only ~20 of 74 features, the unused ~60 enable phase/horizon-adaptive behavior keyed on an episode-completion-percentage feature and direct shared-memory deconfliction. BUT be wary of shared-state mechanisms that assume a full complement of YOUR agents: in a real tournament a single instance may sit among foreign agents, so shared-state coordination can collapse to a no-op — prefer observation-feature work that degrades gracefully with one controlled agent. Keep the player's submission/iteration history as a dated log in its own README recording the concrete model/config/behavior delta versus the prior submission (e.g. 'Bedrock + Sonnet 4.5', 'slot/color voting fix') to make regression hunting and cross-submission A/B reasoning tractable. _{sources: claude-code:7db8d397-c6dc-47b6-9b95-8f90675dd1b8, auggie:e251085a-b35d-4171-8dbd-d006b8bc2bd9, opencode:ses_200d7b1fbffehZb7LsbgTZtHzN, opencode:ses_2042c1ff1ffefETImBU5n4D43k}

10. Architect self-improvement as an offline observe->diagnose->act->verify->promote loop with a fat live path

loop

Structure a self-improving policy as an explicit OFFLINE loop run between games (never mid-game, to keep the live decision path stable): observe (episode_launch/status, eval_aggregate_metrics) -> diagnose (memory/episode search, episode_artifacts) -> act (policy_create_version/update_params/mutate, or a skill edit) -> verify against the gate (eval_enqueue, read gateDecision) -> promote (policy_promote only on gate=promote, plus memory_create) -> submit (policy_submit_to_league only a promoted version) -> re-read coworld_league_standings; on reject or weak_evidence loop back to diagnose, never ship. The original among-them instance fed a per-game memory dump to a more capable model (Opus) producing learnings -> post_game_analysis -> code_evolution edits via an auto-merged PR. Keep the live loop fat and programmatic (perceive/think/update-beliefs/think/act as Python micro-programs over <=128KB JSON working memory, think->update to a quiescence fixpoint before the action decision) with the LLM writing those programs offline. Model the optimizer itself as a state machine (created -> baseline_eval -> analysis -> proposal -> candidate_build -> candidate_eval -> comparison -> terminal). The minimum viable proof an optimizer works is population-of-1 greedy hill climbing (gaussian-perturb clamped params, keep only if it beats the parent, random restarts) validated first against a MOCK fitness function with a known ideal-params sweet spot before wiring the real server. Offer two modes -- the agent-driven loop (inject hypotheses each pass) vs the packaged loop (opt_create_run then repeated opt_advance for hands-off/overnight runs with memory and step escalation). _{sources: opencode:ses_223885796ffeVoLFHl6DGhmFrq, opencode:ses_223c540d9ffe8nF6kcU1RXf0wp, optimizers/docs/optimizer-primitives.md, optimizers/skills/README.md (+4)}

11. Treat the eval gate as objective + guardrail: ship only on 'promote', preview with gate_dry_run, sweep when plateaued

loop · tool: gate_dry_run

The eval gate is both objective and safety guardrail: evaluateGate(baseline, candidate, config) compares on matched seeds and returns promote (primary improved by >= minimumDelta AND no critical metric regressed AND enough episodes completed), reject (critical regression, more failures, or primary clearly flat), or weak_evidence (too few episodes, no baseline, inconclusive). Ship/promote ONLY on promote; treat reject and weak_evidence as never-a-yes (loop back to diagnosis, record a reject as an anti-pattern memory). To verify a candidate, eval_enqueue with the current champion as baselinePolicyId, poll eval_status to terminal:true, read detail.gateDecision/primaryDelta, and don't cherry-pick metrics (the gate already encodes primary + critical non-regression). Before spending episodes, preview cheaply with gate_dry_run (a suiteId plus a structured candidate {completed, failed, metrics}, baseline optional). Crewrift uses crewrift-smoke-v1 (fast screen) and crewrift-regression-v1 (promotion gate). When single changes plateau, sweep: policy_create_version/mutate -> eval_launch_campaign -> compare_runs to rank by the primary metric, promoting only on gate=promote, defining a lineup when multiple policies share an episode so results attribute per policy. Apply step escalation as the champion improves (raise maxTicks and episode counts); repeated weak_evidence means the eval lacks POWER, not that the change is bad -- raise episode counts. _{sources: coworld-source-repos.crewrift-upload/optimizers/skills/policy-eval-gat, coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/mcp-opt, coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/mcp-opt, optimizers/docs/optimizer-primitives.md (+3)}

12. Diagnose one bottleneck per pass, carry one hypothesis, prefer params, validate, and hand off a machine-readable report

loop

In the improve-policy loop observe the champion with 15-20 episodes, aggregate with eval_aggregate_metrics, and target the SINGLE weakest metric -- identify ONE bottleneck (the single constraint limiting fitness/win/survival) per analysis so effort concentrates. A proposal carries exactly one primary hypothesis (one sentence), expected metric movement (metric+direction+magnitude), risk, and an explicit params|code|both decision; prefer parameter changes (set code needed=true only when no safe param patch exists), expressed as a partial patch of only changed keys. Always validate with a rerun whose settings are explicit (1-5 episodes, variant, pass/fail gate) linking back to the exact version created. Make the episode report the durable hand-off: emit machine-readable JSON (post-game learnings as IF/THEN rules; recommendations with schema_version, expected_metric_movement, risk, evidence ticks/keys). Score each episode 0-10 on outcome AND process (0-3 fails core loop, 4-5 major-mistake-dominated, 6-7 sound strategy with fixable bugs, 8-9 strong, 10 optimal at current difficulty), judging improving/flat/regressing by a rolling average of the last 5 scores for the same policy+game, not one noisy episode. _{sources: optimizers/skills/improve-policy/SKILL.md, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/analy, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/orche, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/polic (+1)}

13. Optimize trainable skill text via bounded, dual-gated edits with a disjoint held-out set

loop

Treat trainable agent skills as versioned prompt fragments optimized SkillOpt-style, distinct from policy param/code optimization: keep the agent frozen and adapt behavior only through bounded, validation-gated edits. Propose at most L edits/step (default 4) using only four atomic ops (append, insert_after, replace, delete), encode procedures not instance values, prefer repairing existing sections, never target content inside the ... region (rejected by lib/skills/edits.ts; target text must match exactly). Gate with a hybrid two-stage dual gate: stage-1 skill_propose_edit runs an LLM-judge fast screen (compactness, description/body alignment, procedural form, protected-region integrity); stage-2 skill_resolve_proposal accepts a new revision only if it strictly improves BOTH a selection set AND a disjoint held-out set (guarding against overfitting) with no critical regression, else a human accepts. Rejections are buffered (skill_rejected_buffer) so the loop learns from near-misses. _{sources: coworld-source-repos.crewrift-upload/optimizers/README.md, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/skill, coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/bounded, coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/mcp-opt}