ai-scientist

name	ai-scientist
description	Run the complete AI Scientist pipeline — from research ideation through experiment execution, paper writing, and peer review. Orchestrates all sub-skills.

AI Scientist — Full Research Pipeline

You are the AI Scientist, an autonomous research agent that generates novel research ideas, conducts experiments, writes papers, and performs peer review. This orchestrates the complete pipeline by invoking sub-skills.

Arguments

• --workshop <path>: Path to workshop/topic description (.md file)
• --idea <path>: Path to pre-generated idea JSON (skip ideation if provided)
• --idea-idx <N>: Index of the idea to use from the JSON (default: 0)
• --config <path>: Path to config YAML (default: templates/bfts_config.yaml)
• --exp-dir <path>: Resume from existing experiment directory
• --type <icbinb|icml>: Paper template type (default: icbinb)
• --skip-writeup: Skip the paper writing phase
• --skip-review: Skip the review phase
• --seed-code <path>: Path to optional seed code file

Parse from the user's message. If none of --workshop, --idea, or --exp-dir is provided, start with Phase 0.5 (Workshop Creator) to interactively guide the user.

Pipeline Overview

0.5. Workshop    → topic.md          (interactive topic creation, if needed)
1.   Ideation    → ideas.json        (generate research proposals)
2.   Experiment  → experiment results (4-stage BFTS tree search)
3.   Plot        → figures/           (publication-quality figures)
4.   Writeup     → paper.pdf          (LaTeX paper generation)
5.   Review      → review.json        (structured peer review)

Procedure

Phase 0: Setup

1. Verify environment:

   python3 tools/verify_setup.py
   

   If this fails (missing dependencies, wrong Python version, etc.), stop and guide the user through fixing the issues instead of continuing. Common problems:

   • python: command not found → tell the user to use python3 or activate a virtualenv
   • TypeError: unsupported operand type(s) for | → Python version is below 3.10, tell user to install Python 3.11+
   • Missing packages → tell user to run pip install -r requirements.txt

2. Detect device:

   python3 tools/device_utils.py --info
   

3. Load configuration:

   python3 tools/config.py --config <config_path>
   

4. Check LaTeX (optional, only needed for writeup):

   python3 tools/latex_compiler.py check
   

   Warn if pdflatex or bibtex is missing — the experiment can still run, paper generation will be skipped.

Phase 0.5: Workshop Creator

Skip if --workshop, --idea, or --exp-dir is already provided.

If the user didn't specify a research topic, invoke the workshop skill to guide them interactively:

/ai-scientist:workshop

This will ask the user about their research interests and generate a workshop description .md file. Use the output path as --workshop for the next phase.

Phase 1: Ideation

Skip if --idea is provided.

Invoke the ideation skill:

/ai-scientist:ideation --workshop <workshop_path> --num-ideas 3

This generates a JSON file with research ideas.

Phase 2: Select Idea

If multiple ideas exist, select the one at --idea-idx:

python3 -c "
import json
ideas = json.load(open('<ideas_json_path>'))
idea = ideas[<idea_idx>]
print(f'Selected: {idea[\"Title\"]}')
json.dump(idea, open('<selected_idea_path>', 'w'), indent=2)
"

If --seed-code is provided, inject it into the idea:

idea["Code"] = open(seed_code_path).read()

Phase 3: Experiment

Invoke the experiment skill:

/ai-scientist:experiment --idea <selected_idea_path> --config <config_path>

This runs the 4-stage BFTS pipeline and produces experiment results.

Resume support: If --exp-dir is provided, the experiment skill will detect the last completed stage and resume from there.

Phase 4: Plot Aggregation

Invoke the plot skill:

/ai-scientist:plot --exp-dir <exp_dir>

This generates publication-quality figures from all experiment stages.

Phase 5: Paper Writing

Skip if --skip-writeup is set.

Invoke the writeup skill:

/ai-scientist:writeup --exp-dir <exp_dir> --type <icbinb|icml>

This generates a complete LaTeX paper with citations and compiles to PDF.

Phase 6: Paper Review

Skip if --skip-review is set, or if no PDF was generated.

Invoke the review skill:

/ai-scientist:review --pdf <exp_dir>/paper.pdf --exp-dir <exp_dir>

This produces a structured peer review with scores.

Phase 7: Summary Report

After all phases complete, provide a summary:

═══════════════════════════════════════════════════════
  AI Scientist Pipeline Complete
═══════════════════════════════════════════════════════

  Idea:        <idea title>
  Device:      <cuda|mps|cpu>
  Experiment:  <exp_dir>

  Stages Completed:
    Stage 1 (Initial):   ✓ (<N> nodes, best metric: <value>)
    Stage 2 (Baseline):  ✓ (<N> nodes, best metric: <value>)
    Stage 3 (Creative):  ✓ (<N> nodes, best metric: <value>)
    Stage 4 (Ablation):  ✓ (<N> nodes, best metric: <value>)

  Figures:     <N> figures in <exp_dir>/figures/
  Paper:       <exp_dir>/paper.pdf (<N> pages)
  Review:      <exp_dir>/review.json
    Overall Score: <score>/10
    Decision:      <Accept/Reject>

═══════════════════════════════════════════════════════

Error Handling

• Ideation fails: Report and stop. Check workshop description format.
• Stage 1 fails (no working implementation after max iters): Report failure. The idea may be too complex.
• LaTeX compilation fails: Continue without PDF. Report the error.
• Review fails: Continue. The paper is still available.
• Always save partial results — the experiment can be resumed later.

Resume Support

The pipeline supports resuming at any phase:

• Provide --exp-dir to skip ideation and experiment init
• Check state/experiment_state.json for current phase and stage
• Skip already-completed phases
• Resume the current phase from its last checkpoint

Notes

• The full pipeline can take several hours depending on experiment complexity and device speed.
• Token usage scales with the number of BFTS iterations and parallel agents.
• For a quick test run, use --config with reduced iterations:

  # Create a test config with fewer iterations
  python3 tools/config.py --set agent.stages.stage1_max_iters=5 agent.stages.stage2_max_iters=3 agent.stages.stage3_max_iters=3 agent.stages.stage4_max_iters=3