| name | crewrift-report |
| description | Use to turn a SET of Crewrift episodes (an experience request, a policy's recent league games, a round/tournament batch) into a dense, high-signal report on a player policy's strengths and weaknesses. Triggers: 'analyze/report on these XP-request episodes', 'where is crewborg weak', 'find the interesting/bad/good games', 'what did crewborg do wrong as imposter', 'summarize this batch of replays/logs', post-mortem of an evaluation. Crewrift-specific (roles, votes, kills, expand_replay). Pair with coworld-episode-artifacts (to pull the episodes) and coworld-experience-requests (to create them). |
Crewrift Report
Turn the large, noisy pile of episode artifacts from an experience request (or a
batch of tournament episodes) into a dense, high-signal report that makes a
player policy's strengths and weaknesses easy to spot. This is the analysis engine of
the loop's Report step (see crewrift_lab/AGENTS.md).
It works by finding the "interesting" episodes — the outliers and failure/success
cases that actually carry signal — and nailing down what the policy did in them, so
the human can quickly see where to improve. Read
references/signals.md for the full taxonomy of what's
flagged and why; read crewrift_lab/docs/crewrift-gameplay.md
to interpret events as gameplay.
Three tiers (cheap → deep)
Most signal is cheap; only drill deep where it pays off.
- Tier 1 — structured (
scripts/report.py). Reads results.json + episode.json
across the batch (no replay parsing). Per-slot role flags, scores, tasks,
kills, vote behavior, and timeouts give role-decomposed distributions and flag most
"interesting" categories instantly. Scales to hundreds of episodes.
- Tier 2 — objective timeline (
scripts/profile_replay.py). Runs the
version-matched expand_replay on a flagged episode and extracts what Tier 1
can't: killed-by-imposter vs ejected-by-vote vs survived, the itemized score
breakdown, vote correctness (did it vote a real imposter? eject a real
crewmate?), and an event-feature vector.
- Tier 3 — the policy's own logs. The subjective why at the tick of interest
(crewborg:
crewrift/crewborg/docs/trace-logs.md;
others: plain-text stderr). Reach for this on the few juiciest episodes.
Workflow
-
Get the episodes into one directory with the coworld-episode-artifacts skill
(by experience request, --policy <name>, round, or division), e.g.
… fetch_artifacts.py --policy crewborg -n 100 --out /tmp/eps. (Add --version N
to focus a specific upload; omit it for the policy's recent games across versions.)
-
Tier 1 — the report:
scripts/report.py /tmp/eps --policy crewborg --json /tmp/report.json
Read the role-split distribution and the ranked interesting episodes. The
episode_dir on each line is the drill-in handle.
-
Tier 2 — profile the flagged episodes. Build the replay reader once
(crewrift_lab/tools/build_expand_replay.sh; Observatory replays match
CREWRIFT_REF), then:
scripts/profile_replay.py /tmp/eps/<episode_dir> --policy crewborg
Run it on the episodes Tier 1 flagged (the killed/ejected, vote-correctness, and
score-breakdown facts are here). Parallelize across the flagged set if there are
many.
-
Tier 3 — read the logs of the most informative episodes at the tick the
profile points to, for the behavioral why.
-
Synthesize + present. Present a dense report to the human in the chat (the
markdown is printed to stdout for you to read; the human sees it because you relay
it): the headline role-split outcome, then a ranked list of concrete
strengths/weaknesses tied to specific episodes + the deviating signal, and point at
the interesting replays — then hand the direction call back to the human.
To go deeper on why a weakness happens and offer candidate directions, run the
crewrift-diagnose skill (optional) — it turns these signals into
evidence-grounded, mechanistic improvement hypotheses.
Discipline (from best_practices.md)
- Always decompose by role — crewmate and imposter are effectively two different
policies; an aggregate hides one role being broken.
report.py splits every
distribution; keep that split in your synthesis.
- Separate operations from behavior — connect/disconnect timeouts (−100) are a
crash, not a strategy flaw; they get their own category. Don't let them pollute
behavioral analysis.
- Chase the "should-have-won" game —
crew_lost_nearly_won (tasks ≈ complete but
lost) is usually the most informative failure.
- Mind small n —
report.py warns under ~20 appearances; don't over-read a thin
batch. Pull more episodes for firm claims.
- To judge whether a change helped, use the
crewrift-ab skill — this report
surveys one batch descriptively; deciding "did v2 beat v1, right now" is a
targeted, fresh, matched head-to-head comparison, which is what crewrift-ab does
(compare.py + a qualitative investigation). Don't eyeball two separate reports.
"Unusual event profile"
profile_replay.py emits a per-episode feature vector (kills, bodies, meetings,
ticks, the policy's votes/ticks-alive/kills). An episode is unusual = an outlier
on some feature relative to the batch (robust z / rank), reported with the specific
deviating feature ("alive only 180 ticks vs median 3000", "0 chats vs median 4").
Aggregating these features across the batch for automatic outlier flagging is the
documented next extension; for now, scan the feature vectors of flagged episodes.
Files
scripts/report.py — Tier 1 structured report (+ --json).scripts/profile_replay.py — Tier 2 per-episode fact sheet from expand_replay (+ --json).references/signals.md — the category taxonomy, the artifact field reference, the
score model, and the expand_replay event-line formats.
