菜单
Use to decide what crewborg should log to validate a specific hypothesis, then instrument the policy so the next eval's artifacts carry the signal. Trigger on 'what data do I need', 'how do I instrument this', 'what should I log to test this', or 'add tracing to validate the hypothesis'.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | data-collection-design |
| description | Use to decide what crewborg should log to validate a specific hypothesis, then instrument the policy so the next eval's artifacts carry the signal. Trigger on 'what data do I need', 'how do I instrument this', 'what should I log to test this', or 'add tracing to validate the hypothesis'. |
A pre-registered hypothesis usually needs data crewborg does not yet emit. This skill decides what to log so the next eval's artifacts can confirm or kill the hypothesis, then instruments the policy. Two sister questions, both answered here:
server_tick) can validate against ground truth?Instrumentation lands together with the behavior change so the very next eval is diagnostic — never change behavior blind and hope the existing logs suffice.
Announce at start: "Designing the data collection for this hypothesis: which events/metrics/positions to log, in what shape, so the next eval's artifacts prove or disprove it — then wiring it in."
For the hypothesis's prediction and regression tripwire, list the minimum signals that distinguish it from the null. For each, pick the cheapest artifact home:
| You need to know… | Log it as… | Where |
|---|---|---|
| an outcome/state change (a kill, a vote, a phase flip) | a domain.* event with a JSON payload | traces table |
| a per-tick spatial/visibility fact (where, who-seen, which mode) | already in positions (self_x/y, room_id, mode, intent_kind, visible) — extend the row only if missing | positions table |
| a rate/duration/count (latency, dwell, cooldown age) | a counter/gauge/histogram | metrics table |
| a one-shot decision rationale | a payload field on the existing decision event (e.g. vote reason) | traces |
Prefer adding a field to an existing event over a new event; prefer a
derived query over logging a new kind at all (raw observations, derived
interpretations — design.md §5.2).
If the hypothesis is checked against the replay (most are), the new signal must be
locatable on the server_tick timeline so replay-reconstruction can join
it. Domain events carry tick; per-tick facts go in positions which already
records server_tick. Don't invent a parallel clock. Carry the spatial annotation
(self_x/self_y/room_id) the domain events already attach, so a pattern is both
when and where.
More logging = bigger artifacts and, on hosted runs, risk of hitting capped logs.
Turn on the narrowest thing that captures the signal (see artifact-capture
for the env matrix):
CREWBORG_METRICS=1.CREWBORG_TRACE=debug.CREWBORG_TRACE=viewer.CREWBORG_TRACE_GROUPS=... / CREWBORG_TRACE_INCLUDE=glob.Record which level the eval will run at — it must match across baseline and variant arms or the comparison is confounded.
crewborg's own game events are emitted in events.py
(CrewborgEventTracer.on_step_complete), which sees the finalized belief +
chosen intent + produced command each tick — the only place attempt-events
(keyed on the wire command) and post-mode conclusions (e.g. task_completed) are
observable. Add new domain.* emissions there; add positions columns in
artifact.py (bump SCHEMA_VERSION, update the embedded README) and populate
from the bridge. Keep strategy/ pure — read off finalized belief, don't log from
inside decision code.
Run a 1–3 episode smoke (artifact-capture) and confirm the new event/field/
metric actually appears with non-trivial values and joins to the replay. A
hypothesis silently un-instrumented produces a fake "no effect." Only then launch
the full eval (eval-set-design).
The instrumentation diff (events/fields/metrics + trace level), a note of the
server_tick join path, and confirmation from the smoke that the signal lands.
Hand to eval-set-design to run the comparison.
hypothesis-generation (the change + validation-data list).eval-set-design, then artifact-capture / replay-reconstruction
on the next pull.artifact.py (schema, record_position, trace levels),
events.py (the on_step_complete emission seam), design.md §5.2/§11.Use to identify and download Coworld episode artifacts — replays, results, and per-agent logs — for completed episodes, or STREAMED live from a still-running experience request (--watch: each episode downloads as it finishes, no waiting for the batch). Triggers: 'download the replay/logs for episode X', 'pull the last N episodes <player> played', 'fetch artifacts for this experience request / pool / round', 'stream artifacts while the request runs', 'get the episode data for post-mortem'. Game-agnostic (Crewrift, amongthem, etc.).
Use to create and monitor Coworld experience requests — hosted batches of episodes you define (target, roster, roles, count) for evaluating a policy against a live field. Triggers: 'run crewborg vs the top opponents', 'make an experience request', 'request N hosted games', 'A/B a policy against the league', 'measure the imposter', 'set up an evaluation battery'. DEFAULT FLOW after create: launch the STREAMING pipeline in the background (step 4) — artifacts download and analysis prep overlap the still-running episodes; don't wait for the batch to drain.
Use to run your own built player policy in a LOCAL Coworld episode and watch it — the Gate-1 smoke test ('did my change take, does it connect → play → exit cleanly?'). Triggers: 'smoke test the player locally', 'run a local game', 'run my policy locally and watch it', 'does the build still work', 'open the replay locally'. Game-agnostic; this is correctness only, NOT a competitive matchup (you can't run other users' policies locally).
Use to submit an ALREADY-UPLOADED policy version to a Coworld league and monitor whether it QUALIFIES, then competes (and becomes champion). Triggers: 'submit crewborg to the league', 'did it qualify', 'is it disqualified / champion', 'watch the qualifier', 'monitor standings after submitting'. SUBMIT is the public, effectively-irreversible, champion-making action — explicit human go-ahead only. (Building + uploading a version is the separate `build & upload` skill.)
Use to decide whether a crewborg change ACTUALLY helped — A/B the candidate against the baseline head-to-head, fresh, right now, against the same field. Triggers: 'did my change help', 'compare v71 vs v70', 'A/B test crewborg', 'is the candidate better', 'did fixing the imposter regress the crew'. Runs two MATCHED fresh experience requests, diffs role-split metrics with significance (compare.py), renders a comparison report, and guides a qualitative side-by-side. It's also the 'designed run' that crewrift-experiment hands off to.
Use to build and query the Crewrift event warehouse — a policy-indexed DuckDB/Parquet dataset of per-tick gameplay events (kills, follows, proximity, votes, tasks, chat) over many episodes — for deep, mechanistic, cross-episode questions about a policy's behaviour. Triggers: 'how often is crewborg near a crewmate without killing', 'who trails crewborg', 'where does it abandon tasks', 'does it vote real imposters', 'build the warehouse for this round', 'query the events', 'stream an experience request into a warehouse'. For a FRESH experience request, stream_eval.py builds the warehouse WHILE the episodes run (the default; builds are incremental). This is the deep-dig mainstay after crewrift-survey flags something.