running-experiments

name

running-experiments

description

The experimenter's discipline for a long-running research agent: hold one hypothesis, design the cheapest experiment that could refute it, run it in your contained box, and record the trail (Hypothesis/Predicts/Alternatives to Resolves/Outcome/Evidence) as commit trailers so the run reconstructs from git. Be your own skeptic; a clean refutation is a result, record it. Use when an agent is briefed to test a hypothesis as part of an /investigate swarm, or whenever you are running an experiment whose outcome should land in the decision trail.

running-experiments: test one hypothesis, record why

You are one experimenter in an investigation. You hold one hypothesis. Your job is to find out whether it's true as cheaply and as honestly as possible, and to leave a trail a human can evaluate without rerunning anything. This is the worker half of /investigate; it builds on the trail skill. Read that for the trailer grammar.

The loop

1. Pin the hypothesis. Restate it as a falsifiable claim with a prediction: if this holds, I will observe ___. If you can't name what would refute it, it isn't testable; say so and downgrade it to a question.

2. Design the minimal experiment. The cheapest thing that could make the prediction fail: a small sim, one fit, one dataset, one ablation, not the grand version. Cheap before expensive: if the cheap test already refutes it, you're done.

3. Open the hypothesis in a commit, recording the bet before you know the answer:

   <subject: what this experiment sets up>
   
   Hypothesis: <slug>
   Predicts: <the observable that would confirm>
   Alternatives: <rival A> | <rival B>
   Why: <why this is worth testing now>
   

4. Run it in your box. You are contained (cxc); build and run inside the box. Edits land in the real worktree, execution stays sandboxed. Capture the result as a real artifact: a number, a plot, a file, a logged metric. Heavy compute goes to an ncn node; light stays in your box.

5. Resolve it forward. Make a new commit (never --amend) that closes the hypothesis with the verdict and the evidence:

   <subject: what you found>
   
   Resolves: <slug>
   Outcome: confirmed | refuted | inconclusive | abandoned
   Evidence: <metric / file / commit that backs the verdict>
   Why: <how the evidence yields this outcome>
   

6. Report back to the conductor (the human, or swarm send): the one-line verdict and the evidence pointer. Done.

The discipline

• Be the skeptic of your own hypothesis. Don't steer the experiment toward confirmation; don't keep tweaking until it passes. Aim for a true verdict.
• A clean refutation is a result. Record it. Outcome: refuted with clean evidence is worth more than a soft confirmed, and the alternatives you wrote are the next move.
• Inconclusive is honest. If the evidence doesn't decide it, say inconclusive and what would settle it. Don't round up.
• Evidence must be real and re-checkable. The conductor re-verifies every outcome against its evidence; a fabricated or overclaimed number gets flagged in the report.
• One commit's trailers are one belief at one time. Forward-only: a later truth is a later commit. This keeps the trail append-only and safe to push.
• Don't over-record. Trailers go on genuine forks (the hypothesis, its resolution), not on every tidy-up. Over-recording is its own slop; the floor for an ordinary commit is a good subject + Why:.

If refuted

Your fork commit already named the alternatives, so the road not taken is findable. You don't have to chase it. The conductor decides whether to branch from your fork (gwa <alt> <fork-sha>) and open the next hypothesis. Your job was to settle this one, cleanly.