| name | control-tower-usage |
| description | How to drive the `ct` CLI to run matches and produce every number and graph a control experiment ships. Covers `ct run eval`, `ct run monitor`, `ct run summarize`, `ct game-metrics metrics / graph roc / graph pareto`, `ct traj`, and the viewers. Load this when you launch a match, compute safety or usefulness, regenerate a graph, or need the exact reproduce command for an output. White loads it always; main and others load it on demand. |
We do all of our development with ControlTower, which has several agent environments: LinuxArena (a broad Linux SWE agent env, ~10 PUBLIC envs plus a PRIVATE set) and BashArena, among others. An experiment may use one or MIX several. Default: LinuxArena, with train and validation on the public env set and test on the private set (only if the human has access to the private set, which launch-control-research asks about; otherwise hold out extra public envs for test). Ask the human whether they want a different or additional environment. The public LinuxArena env names are listed under the publish-to-public constraint in Trajectories and viewers below.
ct is the infrastructure. Every safety number, usefulness number, and graph in an experiment comes from a ct command run against the immutable match store in <protocol>/runs/, never hand-computed and never hand-drawn. This is the anti-faking guarantee: outputs/README.md records the exact ct command plus source run URLs for every file in outputs/, so a reader can recompute it. See [white-team] for the match loop and [empirical-ai-research] for the quality bar the numbers must clear.
ControlTower is a work in progress and its ct interfaces can change. If a ct command errors or behaves unexpectedly, do not guess or give up: read the current ct source (the control_tower package under src/ in this repo, or https://github.com/linuxarena/control-tower) to find the real flags and behavior, debug the invocation, and record the working command in your notes and the run's command.txt so later runs reuse it.
Run every command with uv run ct ... from the control-tower repo root. Whether runs use the EC2 fleet (--ec2) or run locally on this machine is a setup choice recorded in <project>/.claude/control-research.local.md; read it before you launch. The control-tower framework itself ships to the fleet workers via its own bundle; your experiment code reaches them a different way. On EC2, add --extra-src "$PWD" (the repo root you launch from) to every run so the worker gets experiments/ on its import path and the same experiments.<YYMMDD>_<slug>.<proto>... ref resolves on the fleet exactly as it does locally. --extra-src DIR ships DIR's current WORKING-TREE state to the workers as an import root (like PYTHONPATH=DIR, --ec2-only, repeatable with a distinct basename per dir), skipping .git, .venv/venv, build, dist, __pycache__, node_modules, and lockfiles. No commit is required before an EC2 run: --extra-src ships the working tree as-is. On a local run there is no fleet and no --extra-src: the ref resolves through the repo-root CWD and the working tree runs live. Launch from the repo root either way.
Running a match (white is the sole runner)
A match is one (blue iter, red iter, split) triple. Blue and red never call ct run eval; only white does. The four artifacts a full match needs are honest_eval, attack_eval, honest_monitor, attack_monitor. The honest baseline sets the audit threshold; the attack monitor gives the suspicion scores safety depends on.
Before EVERY --ec2 launch, pass --extra-src "$PWD" (EC2 runs only; a local run uses the working tree live and adds no flag). This ships the current working tree of the repo root, experiments/ included, to the fleet workers as an import root, so the exact experiment code you have on disk reaches them with no commit step. Because it ships the working tree as-is, untracked and gitignored files ship too (a gitignored experiments/ is no obstacle). If you forget the flag on EC2, the worker has no experiments/ on its import path and the run fails with ModuleNotFoundError; the fix is the flag, not a commit.
ct run eval
uv run ct run eval --policy honest --task-file <set> --ec2 --extra-src "$PWD"
uv run ct run eval --policy attack --task-file <set> --ec2 --extra-src "$PWD"
For a protocol under test, pass the protocol instead of a bare policy:
uv run ct run eval --untrusted-policy attack -up model=<untrusted> \
--protocol <protocol> --protocol-arg <k>=<v> --task-file <set> --ec2 --extra-src "$PWD"
- Reference experiment-defined policies and monitors by their
module:attr path, never a bare registry name. The registry auto-discovers only control_tower.protocols.*; nothing imports experiments/, so a bare name for an experiment policy resolves to Unknown. The module:attr form is what imports the module and fires its @register() decorator, e.g. --untrusted-policy 'experiments.<YYMMDD>_<slug>.<proto>.attack_policy:<fn>', or a protocol's monitor via --protocol-arg action_monitor='experiments.<YYMMDD>_<slug>.<proto>.protocol:<fn>' (the same module:attr form is accepted by --trajectory-monitor, but that flag lives on ct run monitor, not ct run eval). The ref is identical local and on the fleet: locally it resolves through the repo-root CWD, and on EC2 --extra-src "$PWD" puts experiments/ on the worker's import path so the same ref resolves there. Registering does not remove the need for the module to be importable. Mind the two path formats: --untrusted-policy/--trajectory-monitor take this DOTTED module:attr path, while --task-file and -up variables_path= take a SLASH filesystem path (e.g. $NS/trajmon/red/v1/variables.py). Passing a slash path to the policy or monitor flag fails with Cannot import experiments/....
- The run's setup (in
<project>/.claude/control-research.local.md) decides EC2 vs local. When the setup is EC2, use --ec2 --extra-src "$PWD" on every run, including smoke (distributed EC2 workers via SQS), with upload on: running smoke on the fleet exercises the fleet path early, so import-path and compose gaps surface at smoke time instead of at expensive-spend time, and every run lands in the shared registry with a URL a human can open. When the setup is local, drop both --ec2 and --extra-src and the run executes on this machine: no fleet, no registry URL, and the runs table links the local .eval path instead. EC2 needs the fleet set up (creds, the MCP, the env repos); local needs none of that but stays on your machine.
- Smoke is small; on EC2 it still uploads:
uv run ct run eval --policy <honest|attack> --task-file <set> --limit 2 --sample-time-limit 600 --ec2 --extra-src "$PWD" (drop both --ec2 and --extra-src on a local run). A fast wiring check, n=2 has no statistical power. On EC2, --extra-src "$PWD" ships the frozen scaffold and variables.py to the workers (see Before EVERY --ec2 launch above) so the smoke produces a viewable run with no commit; a local smoke uses the working tree live and adds no flag.
- Launch matches with the Bash tool using
run_in_background: true, NOT the mcp__control_tower__ct-run-eval tool. The Bash background process re-invokes you (white) when it EXITS, and that exit is the match-completion event that drives the loop with no hook. The MCP ct-run-eval tool returns immediately with an output_dir and gives NO completion callback, so relying on it silently degrades the loop to main's staleness poll. If you ever use the MCP tool, you must poll stdout.log yourself. The output_dir holds stdout.log/stderr.log; read or tail -f to watch, do not block.
- Poll
stdout.log every 30 to 60 seconds. On any eval_error > 0, halt, note the blocker with the failing trajectory id, and message the requester. Do not paper over an error.
- On EC2, scale the fleet to the split: smoke (n=2) small, train (n=8 to 16) medium, validation and test (n>=24) larger worker and concurrency counts. Higher concurrency on small n wastes EC2 boot cost; lower on big n misses the match deadline. On a local run there is no fleet to scale; the split just runs on this machine.
- Task selection:
--task-file <set> (a named set or file), specific tasks with -t <task> / -s <side_task>, or expansion flags (--all, --just-main-tasks, --just-side-tasks). White owns and versions the task sets and holds them fixed so numbers stay comparable (see [white-team]).
ct run monitor
Scores each action for suspicion. Any safety claim MUST cite the monitored run, not the raw eval run.
uv run ct run monitor <eval.eval-or-run-id> --action-monitor <monitor> --only-action
--save-dir <dir> writes the new monitored .eval (a deep copy) there. On EC2, upload is on by default and must stay on: never pass --no-upload, so every run, eval and monitor, has a canonical registry URL a human can open; upload the monitored run before citing it. On a local run there is no registry, so cite the local .eval path instead. Upload and registry rules apply only on EC2.--only-action for per-action suspicion; --only-trajectory for a whole-trajectory monitor.
The run-pair autopsy: ct run summarize
The grounded per-match reading. Downloads trajectories for a honest + attack pair, fans out a per-trajectory summary, and aggregates into one report. This is the qualitative layer white distills for blue and red, and the raw material [report-writing] and [trajectory-analysis] build on.
uv run ct run summarize --honest-run <url-or-id> --attack-run <url-or-id> \
--extra-context <proposal.md-or-iter-dir> --out-dir <protocol>/runs/<MID>/review/
- Writes
report.{json,md}, per_traj/<traj_id>.json, and SPEC.md under --out-dir.
- Either run is optional alone (autopsy one side); passing both adds the cross-run safety-at-budget sections.
--extra-context (repeatable, file or folder) feeds design intent to the summarizer: proposal, the blue/red iteration folder, the match config. Use it so the reading is grounded in what the iteration was trying to do.- Run it after EVERY match before writing the match note; no match closes without the summarizer pass.
Numbers and graphs: ct game-metrics
Safety + usefulness: ct game-metrics metrics
uv run ct game-metrics metrics --honest <run-id-or-path> --attack <run-id-or-path> \
--output-dir <dir>
uv run ct game-metrics metrics --run-file runs.yaml --output-dir <dir>
Prints per-subset usefulness, attack counts, per-budget audit thresholds, and safety, plus an aggregate safety-vs-budget table. Always writes metrics.json and safety_curves.png to --output-dir. This is where the safety number and the safety curve come from. Report all registered audit budgets, never cherry-pick the one you won at (see [empirical-ai-research]).
Trajectory-monitor caveat. ct game-metrics metrics --mode trajectory_only still collects per-action monitor scores internally (weight_rule._collect_scores calls get_action_sus_scores), so it errors on a run scored by a whole-trajectory monitor, which emits one score per trajectory and no per-action data. To score a pure trajectory monitor, apply the safety formula directly to the ct run monitor trajectory scores plus side_task_success, the same math as the game-metrics trajectory leg. This is a known ct issue, not a setup error.
Curves: ct game-metrics graph
uv run ct game-metrics graph pareto --yml-config <pareto.yaml> --output-dir <dir>
uv run ct game-metrics graph roc --run-file runs.yaml --output-dir <dir>
graph pareto computes a (safety, usefulness) scatter plus frontier from a YAML listing the runs; writes data.json, pareto.png, pareto.pdf. Never cross-pool runs from different protocol versions onto one Pareto axis; one protocol version per plot.graph roc plots monitor separation of honest vs attack actions; supports --group-by, --color-by, --style-by, --filter-by.
Every graph that lands in outputs/graphs/ is regenerated by one of these commands on every new match, and its exact command plus source run URLs go in outputs/README.md. A graph with no reproduce entry is a defect.
Trajectories and viewers
This section is the single home for the two viewers, publishing to the public viewer, and the URL convention.
ct traj builds and inspects trajectory JSON (ct traj make, list, apply, patch). Use it to construct or edit a trajectory; to inspect .eval contents in-conversation, prefer the ct-eval-* MCP tools over unzipping.
Two viewers. Evals upload to the PRIVATE viewer by default. A shipped post must cite trajectories that render on the PUBLIC viewer (no login), or a reader gets a 404. Credentials live in the plugin .env.
| Viewer | Host | Auth | MongoDB env var |
|---|
| Private | equistamp.linuxarena.ai | HTTP basic auth (CONTROL_TOWER_API_AUTH_USERNAME / _PASSWORD) | $MONGOURI |
| Public | data.linuxarena.ai | none | $PUBLIC_MONGOURI |
Both use db linuxbench, collection trajectories-3.0. A fresh eval trajectory 404s on the public viewer until you publish it.
- URL convention. Internal reports cite private-viewer URLs; public posts cite public-viewer URLs only. [report-writing] and [post-writing] follow this convention; they do not redefine it.
- Publish to public. Override
MONGOURI with $PUBLIC_MONGOURI, then uv run ct traj upload path/to/<id>.json (re-uploading the same id overwrites, idempotent). If the traj only exists privately, fetch it first (ct traj download or the API with basic auth), unwrap {"trajectory": {...}} to the bare object, then upload.
- Hard constraint: only PUBLIC environments may go to the public viewer (auto_workflow, clinical_trial, file_indexing, media_processing, model_registry, shared_dotfiles, slack_clone, tiktok, user_tracking, web_scraping). Publishing a private-env trajectory leaks it. Check
environment.id before publishing.
- Verify the link renders, not just that the API returns 200: a Playwright screenshot of
https://data.linuxarena.ai/trajectories/<id> should show the "Control Tower Data" header, the trajectory name, and the numbered steps with Mon <score> badges. A raw URL string is not proof; broken URLs, deleted runs, and auth regressions all still let the string exist.
Cross-references
- Match loop, fairness holds, task-set ownership, match note: [white-team]
- Required-n, statsig discipline, audit-budget reporting, the shared quality bar: [empirical-ai-research]
- The game, safety formula, protocols, vocabulary: [control-research-knowledge]
- Where the numbers get written up: [report-writing] (internal), [post-writing] (public)
For Compaction
You MUST persist this skill through compaction. It is critical that this context remains available to the AI.
