| name | crewrift-survey |
| description | Use to turn a SET of Crewrift episodes (an experience request, a policy's recent league games, a tournament batch) into a fast, polished HTML survey: a per-policy stats table, a policy×policy win heat map, and a short list of interesting episodes (with replay links). Triggers: 'survey these episodes', 'how is crewborg doing in this batch', 'who beats whom', 'give me the report on this XP request', 'which games should I watch'. Fast + lightweight (results.json + episode.json only). Pair with coworld-episode-artifacts (to pull the episodes); the DEEP per-episode dissection is a separate skill. |
Crewrift Survey — fast batch overview
Turn a directory of episode artifacts into a self-contained HTML report a human can read at a
glance: every policy's role-split stats, a policy×policy win heat map, and the handful of episodes
worth opening (each with a one-click replay). It reads only results.json + episode.json — no
replay parsing, no logs — so it's instant even over hundreds of episodes.
What it deliberately does NOT do: death / ejection / chat counts and per-episode behaviour. Those
aren't in the artifact JSONs (game_stats is empty; agent metrics are just reward) — they need the
replay. That's the separate deep survey (event-warehouse). Keep this one fast.
Announce: "Running a fast batch survey over these episodes → an HTML report; then I'll write the
reasons for the interesting episodes and link their replays."
Workflow
-
Get the episodes into one directory with coworld-episode-artifacts (you don't need
replays or logs for the survey):
F=players/crewborg/skills/coworld-episode-artifacts/scripts/fetch_artifacts.py
uv run python "$F" --policy crewborg -n 100 --no-replay --no-logs --out /tmp/eps
-
Run the survey (mint replay links for the flagged episodes):
S=players/crewborg/skills/crewrift-survey/scripts/survey.py
uv run python "$S" /tmp/eps --out /tmp/survey.html --mint-replays --title "crewborg — <what this batch is>"
It writes the HTML and a survey.interesting.json sidecar listing the flagged episodes (their
flags + minted replay viewer URLs).
-
Write the reasons — your job, the part the script can't do. The flagged episodes ship with a
generic placeholder sentence. Replace each with a real, specific, one-sentence reason a human can
act on — read the survey.interesting.json entry (and, if useful, glance at that episode's
results.json) and say why it's worth watching in plain language: which policy, which role, what
actually went wrong/right, the number that makes it stand out. Write them to a file:
{
"20260627T0729_b62c1bbe": "crewborg:v70 as crew finished all 8 tasks but the imposters still won — the clearest 'tasks done, game lost' case, watch how the votes went.",
"20260627T0742_8322cafa": "An operational disconnect (−100) for all three crewborg versions mid-game — an infra crash to rule out, not a behaviour fix."
}
Make each reason distinct and concrete — "a should-have-won game" repeated ten times is noise;
the whole point is interpretation a tag can't give.
-
Re-render with your reasons + present:
uv run python "$S" /tmp/eps --out /tmp/survey.html --mint-replays --reasons reasons.json --title "…"
open /tmp/survey.html
Relay the headline findings to the human (role-split outcome, who beats whom, the interesting
episodes) and point them at the report.
What's in the report
- ① Per-policy table (rows =
policy:vN, aggregated, role-split): Games (crew·imp), Win% overall
/ crew / imp, Score, Kills/g (imposter), Tasks/g (crew, of 8), Voted/g, Skip/g, NoVote/g
(abstentions, −10 each), Ops% (connect/disconnect-timeout rate). Win% is coloured sage (≥50%) /
terracotta (<30%); your policy is highlighted.
- ② Win heat map: cell = % of opposite-team games the row policy beat the column policy
(hover → raw count), sage→terracotta.
- ③ Interesting episodes: a focused, de-duplicated shortlist (rarer flags first; capped, with a
flag-count summary), each with your reason + a replay link.
The HTML follows the Ink & Print house style — report-style.md.
survey.py is a starting point: when a batch needs a different cut (an extra column, a second
matrix), adapt the HTML to fit, conform to that style, and look at the rendered page before
presenting.
How to extend the table
The survey reads exactly these fields — add a column by aggregating one of them in RoleAgg and
adding a <td> in render_html:
results.json (per slot) | already used for |
|---|
win (bool) | win rate (and the heat map) |
imposter / crew (1/0) | the authoritative role split — no inference |
scores (int) | Score (mean; you could add median/quartiles) |
kills (int) | Kills/g (imposter) |
tasks (int) | Tasks/g (crew, of 8; you could add completion %) |
vote_players / vote_skip | Voted/g, Skip/g |
vote_timeout | NoVote/g (the −10 abstention) |
connect_timeout / disconnect_timeout | Ops% (a crash, kept separate from behaviour) |
names | display only |
episode.json gives the slot→policy map (policy_results for league episodes, participants
for XP-request episodes — slot_policy_map handles both), the episode id, replay_url, and
tags.coworld_id (the last two feed the replay link). game_stats is empty and agent metrics
are just reward — anything richer (deaths, ejections, chats, positions) is not here; it needs
the replay → the deep survey.
Replay links
Each interesting episode links to the Observatory hosted replay viewer, minted at survey time:
POST /v2/coworlds/replays/session {coworld_id, replay_uri} → {viewer_url} (works for any episode,
league or XP). That's what --mint-replays does; without it the links read "not minted".
Discipline
- Decompose by role — crewmate and imposter are effectively different policies; the table splits
every rate, and the heat map is opponent-aware. Read them apart.
- Separate ops from behaviour —
Ops% (the −100 connect/disconnect timeouts) is a crash, not a
strategy flaw; it has its own column and its own flag.
- Reasons must be specific — the script flags; you explain. Distinct, concrete, human.
- This surveys one batch descriptively. "Did v2 beat v1?" is a matched fresh A/B
(
crewrift-ab); why a weakness happens is crewrift-diagnose.
See also
coworld-episode-artifacts — pulls the episodes this reads.coworld-experience-requests — creates the batches.best_practices.md — the measurement discipline (decompose by role,
ops vs behaviour, mind small n).
