| name | co-experiment |
| description | Runs ML experiments reproducibly — single runs or autonomous BFS batches. Single mode: isolated venv, time-budgeted, failure-handled, logs to RESEARCH.md. BFS mode (opt-in): designs N hypotheses, runs each for a fixed budget, compares via a single verifiable metric, keeps improvements and git-resets failures — fully autonomous until done. Respects the RESEARCH.md supervision policy for notifications, approvals, and stop limits. Trigger phrases: "run experiment", "train model", "explore design space", "find best config", "autoresearch". |
Co-Experiment
Run experiments reproducibly. Log everything to RESEARCH.md.
If RESEARCH.md includes ## Supervision Policy, apply it for notifications, approvals, resource boundaries, and stop limits. If absent, preserve the defaults below.
Before Running
- Use an isolated virtual environment. If none exists, create one first.
Escape hatch: If the user has already activated a venv, skip this step.
- Validate the target script before execution:
python -m py_compile path/to/script.py
- Set a fixed time budget per run. Default: 5 minutes. Do not exceed without explicit user approval, even if the supervision preset is permissive.
Single Run
- Capture stdout and stderr to a log file.
- Watch for:
NaN/Inf in loss, OOM failures, silent hangs (>60s no output). Terminate and record on hang.
- Failure: patch once, retry once. On second failure — stop with reason
error_threshold, log to RESEARCH.md Context (script path, hash, error, patch, outcome), surface to user.
- Success: extract key metrics, append structured block to
RESEARCH.md Context (date, script hash, metrics, notes).
BFS Mode — opt-in
Activate when the user asks to "explore", "autoresearch", or "find the best config". Requires two preconditions:
- A single target file the agent is allowed to modify (e.g.,
train.py).
- A verifiable scalar metric to minimize or maximize (e.g.,
val_bpb, val_acc). Must be extractable from run output with a grep/awk one-liner.
Apply supervision policy: notify on experiment-start if configured; require approval before entering the loop if bfs-start is in Approve; in wild, proceed only within already granted resource and budget boundaries.
Loop (runs autonomously until budget or N experiments exhausted):
- Design the next hypothesis — one focused change to the target file (e.g., change learning rate schedule, add residual scaling, modify attention pattern). State the hypothesis in one line before modifying.
git commit -am "hypothesis: <one-line description>"- Run the script for the fixed time budget. Extract the metric:
grep "^<metric_key>:" run.log | awk '{print $2}'
- Compare to the current best:
- Improvement → log as
keep in results.tsv, update best.
- No improvement or failure →
git reset --hard HEAD~1, log as discard.
- Append a row to
results.tsv:
commit_hash metric_value status description
- Loop to step 8. Do not pause to ask the user between iterations unless a configured stop target or hard limit has been reached, or a forbidden resource / approval boundary blocks further progress.
End of batch: write a summary table to RESEARCH.md Context; surface best commit and top 3. Apply stop policy (target_reached) or wild continuation as configured, otherwise ask: "Best result is <hash> (<metric>=<value>). Continue exploring or proceed to writing?"
Constraints: only modify the target file; one focused change per hypothesis; log every run including discards; supervision presets never override target-file, metric, budget, or resource boundaries.
Example
Single: python train.py --lr 0.01, budget=5min → appends 2026-03-29 14:32 — val_acc=0.923, hash=abc1234 to RESEARCH.md Context.
BFS: target=train.py, metric=val_bpb (minimize), budget=5min/run, N=10 → runs 10 hypothesis variants autonomously, keeps 3 improvements, produces results.tsv + summary table in RESEARCH.md Context.
