| name | auto-research-execution |
| description | Implement and run the experiments specified in a Stage 2 experiment plan, producing logs, results.csv, and a run report that Stage 4 (writing) can consume. Generates a Sakana-AI-Scientist-style template repo (experiment.py + plot.py + configs/), trains/evaluates with mixed precision and proper checkpointing, captures every metric to WandB or local TensorBoard, and runs a self-healing debug loop (capped at 5 attempts) for failed runs. Use when an experiment plan exists and needs to be executed end-to-end. Do NOT use for ad-hoc one-off scripts (use a normal coding session) or to design experiments (route to auto-research-method). |
Skill 3 — Automated Engineering & Ablation Execution
You are a senior ML research engineer. The experiment plan is a contract — your job is to translate it into a reproducible, instrumented, debuggable experiment that produces clean results for Stage 4. The plan should not change in your hands; if it must, you escalate.
Built on the template-trio pattern from Sakana AI's AI-Scientist (templates/{nanoGPT,grokking,2d_diffusion}/), with self-healing & sandboxing patterns adapted from AutoResearchClaw's CodeAgent.
When to invoke
Trigger when:
- The orchestrator (
auto-research) hands off Stage 3 with experiment_plan.yaml and pseudocode.py ready.
- A user has a written experiment plan and says: "run this experiment", "implement and execute", "do the training and evaluation."
Do NOT trigger when:
- No plan exists (route to
auto-research-method).
- The user just wants you to debug their existing code (use a normal coding session).
- The experiment is purely conceptual / no code needed.
Stage outputs
In runs/<run_id>/stage3_execution/:
code/ # version-controlled experiment repo (init'd by this skill)
├── .git/
├── train.py / experiment.py
├── eval.py / plot.py
├── configs/
├── requirements.txt
└── README.md
logs/ # WandB or TensorBoard mirror + stdout/stderr per run
results.csv # one row per (config, seed) — contract v2 adds an `event_flags` column
results_summary.json # mean ± std per metric per config
checkpoints/ # may be gitignored if too large
run_report.md # what worked, what broke, deviations
hand_off.md # 1-paragraph note for Stage 4
The four critical files for Stage 4 are: results.csv, results_summary.json, run_report.md, hand_off.md.
Contract v2 additions:
results.csv carries an event_flags column — comma-separated subset
of oom, nan, restart, timeout, clean. Populate it from
training-monitor.md's monitor_events.jsonl aggregation; do not
drop these flags when summarising into results_summary.json.- When a stage finishes, write a JSON marker
runs/<id>/stage_3_done with
{stage, started_at, finished_at, gpu_hours_consumed_so_far}
(cumulative, not delta). The orchestrator and the Inspector both rely
on this; an empty marker file no longer satisfies the contract.
Workflow (6 phases)
Phase 1 — Read the plan, scaffold the repo
Read runs/<run_id>/stage2_method/experiment_plan.yaml, method.md, pseudocode.py, and hand_off.md. From the plan, decide which template to start from (see references/template-selection.md):
nanoGPT-style for LM training / fine-tuning experiments.hf-trainer-style for general fine-tuning with HuggingFace Transformers / TRL.vision-clf-style for image classification / detection / segmentation.rl-style for reinforcement learning (gym / pettingzoo).eval-only-style for inference-only studies (no training, just evaluating off-the-shelf models).from-scratch only if no template fits.
Initialize the repo from assets/templates/<chosen>/ into runs/<run_id>/stage3_execution/code/ and git init it. The first commit is "scaffold from
