hypothesis-experiment-loop

name

hypothesis-experiment-loop

description

Design and run scientific or engineering experiment loops that optimize toward falsifiable predictions, evaluate whether results support a hypothesis, and revise or replace the hypothesis from first-principles reasoning when evidence contradicts it. Use when the user asks to design experiments, test a hypothesis, iterate until results meet a target, diagnose why an expected result failed, optimize an experiment, build an ablation plan, update a theory from data, or decide whether the original hypothesis is wrong.

Hypothesis Experiment Loop

Use this skill to keep experimentation honest. The agent should make the hypothesis falsifiable, design discriminating experiments, optimize within the current theory, and revise the theory when the evidence says the premise is wrong.

Core Loop

1. State the hypothesis

   • Convert the user's idea into a precise hypothesis: mechanism, scope, expected direction, and measurable outcome.
   • Identify the first-principles reason it should be true.
   • State what observation would falsify it.

2. Derive predictions

   • Produce 2-5 predictions that must follow if the hypothesis is true.
   • Include at least one positive prediction, one failure-mode prediction, and one discriminating prediction against an alternative hypothesis.
   • Read references/first-principles-revision.md when deriving or revising the causal story.

3. Design the experiment

   • Specify variables, controls, baseline, treatment, sample/unit, measurement, metric, confounders, and stopping rule.
   • Prefer the cheapest experiment that can distinguish hypotheses.
   • Read references/experiment-patterns.md for experiment designs and validity checks.

4. Run or simulate the experiment

   • If tools/data are available, execute the smallest real experiment.
   • If execution is impossible, produce a protocol and a simulated expected-observation table, clearly marked as simulation.
   • Record raw observations separately from interpretation.

5. Optimize within the hypothesis

   • If results are directionally supportive but below target, tune controllable variables.
   • Change one factor at a time when causal attribution matters; use factorial/search designs when interactions matter.
   • Do not optimize on a broken metric or leaky validation setup.

6. Evaluate evidence

   • Classify results as supports, partially supports, inconclusive, contradicts, or invalid experiment.
   • Compare against baseline and alternative hypotheses, not just the desired outcome.
   • Separate measurement failure from hypothesis failure.

7. Revise or conclude

   • If supported and target met: state the result, scope, reproduction recipe, and residual uncertainty.
   • If partially supported: narrow scope or optimize another round.
   • If contradicted: revise the hypothesis from first principles, name the broken assumption, and design the next discriminating test.
   • If invalid: fix the experiment before updating the hypothesis.

Required Ledger

For substantial work, keep this compact ledger:

• Hypothesis
• First-principles rationale
• Predictions
• Alternative hypotheses
• Experiment design
• Controls and baselines
• Metrics
• Observed results
• Evidence classification
• Broken assumptions
• Next experiment
• Current conclusion

Use scripts/make_experiment_ledger.py to generate a starter ledger.

Optimization Rules

• Optimize toward the hypothesis only after verifying the metric and baseline.
• Prefer changes with a clear mechanistic reason.
• Track every change as an intervention, not a miscellaneous tweak.
• Stop optimizing when improvements are within noise or violate constraints.
• If three reasonable optimization attempts fail in the same way, switch to hypothesis revision.

Hypothesis Revision Rules

When evidence contradicts the hypothesis:

1. Identify which prediction failed.
2. Trace the failed prediction back to the assumption or mechanism that produced it.
3. Generate 2-3 revised hypotheses that explain both the old evidence and new contradiction.
4. Choose the revised hypothesis with the fewest ad hoc patches and the clearest next test.
5. Preserve the original negative result as evidence, not failure.

Output Modes

• Experiment plan: hypothesis, predictions, design, metrics, controls, and stopping rule.
• Optimization loop: trial ledger, changes, results, and next intervention.
• Hypothesis audit: first-principles rationale, assumptions, alternatives, and falsifiers.
• Result interpretation: evidence classification and whether to accept, narrow, revise, or reject the hypothesis.
• Ablation plan: minimal experiments to identify which mechanism matters.

Guardrails

• Do not treat desired outcomes as evidence.
• Do not revise the hypothesis because of one noisy result unless the experiment was decisive.
• Do not keep optimizing after evidence shows the causal premise is wrong.
• Do not silently move the target metric after seeing results.
• Do not use test data, hidden labels, or post-hoc subgroup selection to rescue a hypothesis.
• Do not claim causality from correlation unless the design supports it.