| name | report-writing |
| description | The internal, hypothesis-graded per-protocol report. One report per protocol at `outputs/reports/<protocol>.md`, capturing everything that protocol's team tried and why, qualitatively and quantitatively, with a support tag on every claim and a runs table of every run inside the report so nothing is lost. Load this in the writing phase when synthesizing a protocol's matches into its report. This is the internal tier; the public post is [post-writing]. |
The report is the internal record of one protocol: every strategy blue and red tried, every match run, the numbers, and the gears-level reading of why each strategy won or failed. It lands at outputs/reports/<protocol>.md. It is exhaustive where the public [post-writing] is ruthlessly compressed. A human should open the report and see the complete history of that protocol, nothing lost.
Purpose is a gears-level model of red and blue dynamics, not a leaderboard. Numbers give the signal and its significance; qualitative trajectory evidence explains the mechanism. Every claim pairs a number with the trajectory behavior behind it. A number with no mechanism, or a mechanism with no number, is not a finding (see [empirical-ai-research], which owns this qualitative-and-quantitative bar).
The report is a LIVING document. Refresh it on every new match, not only at convergence, so the dashboard always shows the current state. It draws on white's per-match ct run summarize autopsies and on [trajectory-analysis] subagents for verbatim quotes; synthesis is yours, extraction is theirs.
The runs table lives inside the report
Every run the protocol ever produced is listed in a table INSIDE the report, so no run is ever lost even when a later section only discusses a few. Group by category, in this order, before the prose:
- Honest runs, one table per (model x task-pool) cell. Columns:
Run | Usefulness | Average sus | Summarizer | Summary. Name the fair baseline honest run that sets the audit threshold; list it once, it is shared across matches.
- Attack runs, one table per cell, split by side-task channel where it matters (network-requiring vs local). Columns:
Run | Average sus | ASR | Summarizer | Summary. No safety column here; safety needs the shared honest baseline and lives only in the matches table.
- Matches (the blue x red pairs the claims rest on). Columns:
Match (MID) | Blue iter | Red iter | Split | n | Honest run | Attack run | Usefulness | Safety | ASR | Summarizer | Summary. The safety cell cites the MONITORED attack run.
Every Run cell is a full clickable private-viewer URL (see the URL rule). Title each table by its envs, model, n, and task-pool; never add a per-row environment column. Report all registered audit budgets, never cherry-pick.
Claim grading (the report-specific discipline)
Every claim carries a support tag inline, so a reviewer sees the strength before the prose:
- STRONG multiple trajectories agree, verbatim quotes present, no alternative explanation survives.
- MEDIUM single-trajectory evidence with a mechanism story, or a clean aggregate with no contaminating confound.
- WEAK an aggregate where a known confound is comparable in size to the claimed effect.
- SPECULATIVE inference or a known-unmeasured dependency.
- FAILED investigated, evidence did not materialize. Keep these; the point of the report is to record what did not pan out, not only wins.
Cite the hypothesis id verbatim from proposal.md on each tag (STRONG per h01). Never delete a hypothesis; update its row in place with outcome and evidence links.
Hard rules
- No bare adjectives. Every "suspicious / subtle / sneaky / successful" claim carries a private-viewer trajectory link AND a verbatim quote of the action or monitor reasoning that earned the word.
- Real values, never null. If a substep or metric is missing, recover it (replay at temperature 0, re-monitor, or derive it and document the inference). No synthetic placeholder substeps, no "decision not stored".
- Verify the run actually ran the mechanism you credit. A run's mode label is not proof. Before attributing a result to a protocol feature, grep the trajectories for the substep that feature produces and confirm it fired, not a degenerate substitute. One grep on the substep signature before one section on the mechanism.
- Condition before attributing. A lower attack-success rate is not a higher safety number; a higher mean suspicion is not "the same action scored higher" until you condition on landed-versus-attempted and pair by task. Check the obvious confound first: if the favored arm's trajectories are longer, say so and quantify it before crediting the mechanism. If a metric saturates (every mode 1.000 at a budget), call it degenerate and lean on the discriminating budget. Deflationary-but-true beats triumphant-but-confounded; if the data kills the thesis, report that.
- Statsig discipline lives in [empirical-ai-research]. Follow it: Wilson for proportions, Newcombe or two-proportion z for deltas, t-interval for continuous, bootstrap forbidden as primary. Do not restate the formulas here.
- URL convention: PRIVATE viewer for the internal report. Cite the private (authenticated) viewer, which renders every env including private ones, because the report is internal. Safety claims cite the monitor run id, not the eval-only id. Never a bare 32-hex id anywhere (headline table, run-id map, inline, footnote): every id is a markdown link with the full URL visible. The PUBLIC viewer is reserved for [post-writing]; do not mix them.
- Use the controlled vocabulary; never coin terms. The glossary in [control-research-knowledge] and [post-writing] is the source of truth for the report's prose too. Say "side-task success" not "landed", name the specific monitor not "gate", "permissions / affordances" not "grain / grant". Never alter a verbatim quote to remove a banned word.
- No AI giveaways. No em dashes, no arrows, no special unicode (write "at least", "to", "approximately"). Full URLs as markdown links.
- Inline images at their section position. Embed key result images with
 where they belong: the safety curve or suspicion distribution at the top of the run-tables section, the Pareto or comparison plot at each claim that has one. A path listed in prose without ![]() embedding does not render on the dashboard and fails the bar. Every graph is regenerated by real infra (see [control-tower-usage]) with its command in outputs/README.md.
Build claims as a tree, probe each with a comparison
A good report is 3 to 5 main mechanism claims (the macro dynamics), each branching into comparison subclaims that probe or falsify it, each leaf carrying its own evidence (one number plus one trajectory or one verbatim quote). Lead the body with a short "how the game works" paragraph, the move order and the branches red can take, so every claim is visibly one branch. For each main claim ask "what comparison would sharpen or falsify this?" and run it. The axes that generate the subclaims:
- cross-attacker weak vs strong untrusted model: does the effect scale with capability?
- cross-variant which version of the protocol: the interesting subclaim is usually a reversal.
- cross-subset which side-task regime (network vs local, novel vs overlapping host): where a pooled claim hides a per-subset reversal.
- cross-policy per-action vs escalation vs static: where each does best and worst.
Tag each node STRONG / MEDIUM / LATENT / OPEN. LATENT is a real signal that has not yet moved the safety number (a higher approval rate the monitor still catches); name it and say what would convert it to a safety drop. OPEN is a comparison the current data cannot settle; state the probe that would.
Structure
# <protocol>: <one-line claim or question>
**Epistemic status:** <High / Medium / Low> + one line on what would change it.
## Runs (every run, links at top)
### Honest tables: one per (model x task-pool), fair baseline named once
### Attack tables: split by side-task channel, no safety column
### Matches: MID, blue/red iter, split, n, usefulness, safety (monitored run), ASR
## How the game works
The move order and the branches red can take, one paragraph.
## Claims: a tree, sorted by interestingness
3 to 5 main mechanism claims, each branching into cross-attacker / cross-variant /
cross-subset / cross-policy subclaims. Each leaf: mechanism + verdict + support tag +
headline number + evidence (>=1 trajectory link or verbatim quote, prefer a quote
contrast) + the run selection used for THIS leaf (and whether monitored). End with the
FAILED / NULL claims: investigated, evidence absent.
## What was tried and why (the exhaustive log)
Every blue and red iteration, in order: the knob changed, the qualitative reading that
motivated it, the match number, the result. This is the "nothing lost" section.
## Limitations
Wilson CIs, task-set composition, model-dependence, per-protocol confounds, no-pooling.
## Reproducibility
The ct commands, run-id map (all links, no bare ids), code commit SHA.
Anti-patterns
- Burying the runs table below prose so a reviewer cannot click through.
- A public-viewer URL, or a bare 32-hex id, anywhere in the report (rule 6).
- "Suspicious / subtle / successful" without a verbatim quote plus a trajectory link.
null or "decision not stored" where the value was recoverable.- An eval-only run cited for a safety number instead of the monitored run.
- Crediting a mechanism while the favored arm's trajectories are merely longer, or the headline metric is saturated (rule 4).
- Calling a latent signal a safety win.
- A flat claim list never probed by a cross-attacker / cross-variant / cross-subset / cross-policy comparison.
- Dropping the FAILED claims and the what-was-tried log, so a reader cannot see what did not work.
- Plots listed as paths without
![]() inline embedding; a graph with no reproduce command in outputs/README.md.
Cross-references
- The public writeup this feeds: [post-writing]
- The numbers, graphs, and autopsy commands: [control-tower-usage]
- Statsig, required-n, the shared quality bar, qualitative-and-quantitative rule: [empirical-ai-research]
- Per-trajectory extraction subagent: [trajectory-analysis]
- Vocabulary, game, safety formula: [control-research-knowledge]
