auto-experiment

name	auto-experiment
description	自动化超参数/架构搜索循环。给定模型代码、数据集和优化目标，自动生成 train.py，经双 agent 审计门禁通过后，循环调用 DeepSeek 提出改进建议、训练、用 git 追踪每轮变更，保留最优配置和 checkpoint。

auto-experiment Skill

You are running the auto-experiment skill. Your job is to orchestrate an automated, git-tracked hyperparameter / architecture search loop using DeepSeek as the analysis engine.

The skill directory is located at: ~/.claude/skills/auto-experiment/ (Expand ~ to the actual home path when constructing bash commands.)

---

PHASE 1 — Collect Experiment Configuration

Ask the user for the following. Store all answers in memory for this session.

AskUserQuestion 使用规范（重要）： 每次需要用户提供具体内容（路径、代码、数值、列表）时，选项中必须包含一个"请在 Other 中输入…"的选项，让用户知道在哪里填写。格式固定为：

{ label: "请在 Other 中输入…", description: "（点击 Other 输入具体内容）" }

这个选项本身不会被选中——它的作用是引导用户点击 Other 并输入文本。对于纯选择题（无需自由输入）则不需要此选项。

---

1.0 任务描述（最先询问，最重要）

这是整个实验的上下文基础，必须第一个问，后续所有决策都依赖此信息。

用 AskUserQuestion 询问：

question: "请描述本次实验的目标任务"
options:
  - label: "请在 Other 中输入任务描述", description: "请说明：① 任务目标是什么（预测/分类/检测/生成…）② 模型是什么（名称+简要结构）③ 数据是什么（格式/来源/规模）④ 希望优化什么（loss/acc/F1…）"

用户需要描述清楚以下四点，如有遗漏则追问：

1. 任务目标：做什么？（如：预测未来24步的电力负荷 / 对心电图做异常检测 / 图像分类）
2. 模型：用什么模型？（如：DLinear / ResNet / 自定义Transformer）
3. 数据：数据是什么？（如：ETTh1.csv，7个通道，8640行 / ImageNet子集，10类）
4. 优化目标：希望最小化/最大化什么指标？（如：val MSE越低越好 / test F1越高越好）

将用户的完整描述存为 TASK_DESCRIPTION，在后续 DeepSeek prompt 的开头始终附上，让 DeepSeek 清楚实验背景。

---

1.1 Model code

用 AskUserQuestion 询问：

question: "请提供模型代码"
options:
  - label: "使用模板（默认 MLP）", description: "使用内置 Modelfile_template.py"
  - label: "提供文件路径", description: "请在 Other 中输入 Modelfile.py 的绝对路径"
  - label: "粘贴代码", description: "请在 Other 中粘贴完整的模型类代码"

• 选"文件路径" / Other 输入路径 → 读取文件
• 选"粘贴代码" / Other 输入代码 → 直接使用

1.2 Dataset setup

第一步，用 AskUserQuestion 询问是否有 test set：

question: "是否有独立的 test 数据集？"
options:
  - label: "有 test set"
  - label: "只有 train/val，无 test"

Case A — 有 test set：

Step A1，询问 test 数据处理代码：

question: "是否有自定义的 test 数据处理代码？"
options:
  - label: "有，提供文件路径", description: "请在 Other 中输入处理代码文件的绝对路径"
  - label: "有，粘贴代码片段", description: "请在 Other 中粘贴 TestDataset 相关代码"
  - label: "没有，按默认比例自动划分（0.7/0.1/0.2）"

Step A2，询问数据文件路径：

question: "数据文件路径（单个 CSV 或目录）"
options:
  - label: "请在 Other 中输入数据文件/目录的绝对路径", description: "（点击 Other 输入具体路径）"

Case B — 只有 train/val：

Step B1，询问数据文件结构：

question: "train 和 val 是同一个文件还是分开的？"
options:
  - label: "同一个文件，按 0.8/0.2 自动划分", description: "请在 Other 中输入数据文件的绝对路径"
  - label: "分开的文件", description: "选此项后我会依次询问两个路径"

若分开，则分别询问：

question: "train 数据文件路径"
options:
  - label: "请在 Other 中输入 train 文件的绝对路径", description: "（点击 Other 输入）"

question: "val 数据文件路径"
options:
  - label: "请在 Other 中输入 val 文件的绝对路径", description: "（点击 Other 输入）"

1.3 Data processing code

先询问任务类型，以便理解数据结构：

question: "这是什么类型的任务？（帮助我理解数据格式）"
options:
  - label: "时序预测（输入序列 → 未来序列）"
  - label: "分类（输入 → 类别标签）"
  - label: "异常检测（输入 → 正常/异常）"
  - label: "多模态（图像+文本 / 视频+音频 等）"
  - label: "请在 Other 中描述任务类型", description: "（点击 Other 输入任务描述）"

然后询问数据处理方式：

question: "数据处理代码"
options:
  - label: "我来提供完整的 Dataset 类代码", description: "请在 Other 中粘贴完整 Dataset 类（含 __init__/__len__/__getitem__）"
  - label: "我来提供 _load_data 和 collate_fn 代码", description: "选此项后我会依次询问两段代码"
  - label: "让你根据任务类型自动生成", description: "我会根据任务类型和数据路径推断数据处理逻辑"

• 若用户提供代码：直接使用，不做任何任务特定假设
• 若让自动生成：根据用户描述的任务类型和实际读取到的数据文件结构来推断，不硬编码任何格式

关键原则：不对数据格式、字段名、维度做任何先验假设。先读取数据文件的前几行/结构，再决定如何处理。

1.4 Fixed (frozen) parameters

IMPORTANT — 在询问可调参数之前先问冻结参数。

question: "哪些参数在实验中绝对不能被修改？（用于与其他模型公平对比）"
options:
  - label: "无冻结参数，所有参数都可调"
  - label: "有冻结参数", description: "请在 Other 中输入参数名，逗号分隔，例如：input_len, output_len, num_classes"

Store as FROZEN_PARAMS. DeepSeek 的建议中如含有这些 key 则静默删除。

1.5 Tunable parameters

question: "可调参数范围（DeepSeek 可以修改的参数）"
options:
  - label: "使用预设范围", description: "learning_rate:[1e-5,1e-3], batch_size:[8,128], dropout:[0,0.5]"
  - label: "请在 Other 中输入自定义范围", description: "格式：learning_rate: [1e-5, 1e-3]\nbatch_size: [8, 64]（每行一个）"

1.6 Random seed

question: "随机种子（实验可复现的关键，整个实验过程中固定不变）"
options:
  - label: "使用默认值 42"
  - label: "请在 Other 中输入自定义种子", description: "（整数，例如 123）"

Store as GLOBAL_SEED，写入 CONFIG 且永不修改。

1.7 Target metric

question: "目标优化指标"
options:
  - label: "val_loss（越低越好）"
  - label: "val_mae（越低越好）"
  - label: "val_acc（越高越好）"
  - label: "请在 Other 中输入自定义指标", description: "格式：指标名 方向，例如：my_metric lower"

若有 test set，test 指标名自动设为同名加 test_ 前缀（如 test_loss）。

1.8 Experiment mode

question: "实验模式"
options:
  - label: "hyperparams — 仅调超参数", description: "只修改 CONFIG 中的数值"
  - label: "architecture — 仅改模型架构", description: "只修改 Modelfile.py"
  - label: "both — 两者都可以改", description: "DeepSeek 可同时调参数和架构"

1.9 DeepSeek API Key

先检查 DEEPSEEK_API_KEY 环境变量。若已设置则直接使用，无需询问。 若未设置：

question: "请提供 DeepSeek API Key"
options:
  - label: "请在 Other 中输入 API Key", description: "将仅用于本次会话，不会持久化存储"

---

PHASE 2 — GPU Selection

Run:

python3 ~/.claude/skills/auto-experiment/scripts/gpu_info.py

Show the output to the user (or report "No GPU detected — CPU mode"). Then ask: "Which GPUs do you want to use?" (e.g. 0 or 0,1,2,3 or cpu). Store as SELECTED_GPUS.

---

PHASE 3 — Initialize Experiment Directory

Do not ask further questions — proceed automatically from here.

Create the experiment directory in the current working directory:

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
EXP_DIR="./auto_experiment_${TIMESTAMP}"
mkdir -p "${EXP_DIR}/checkpoints/best_val"
mkdir -p "${EXP_DIR}/checkpoints/best_test"   # only if test set exists
mkdir -p "${EXP_DIR}/best_configs"

Directory layout

auto_experiment_<timestamp>/
├── train.py                  # modified each round
├── Modelfile.py              # modified each round (if architecture mode)
├── accelerate_config.yaml    # fixed
├── run.log                   # overwritten each round (not git-tracked)
├── results.tsv               # appended each round (not git-tracked)
├── checkpoints/
│   └── best/                 # best checkpoint (primary metric)
│       ├── model.pt
│       └── config_snapshot.json
└── best_configs/
    ├── best_train.py         # snapshot of train.py at best round
    └── best_Modelfile.py     # snapshot of Modelfile.py at best round

Generate train.py

Write a complete, self-contained train.py. Do NOT copy the accelerate_template.py literally — write fresh code that fits the actual task. Key requirements:

1. Random seed: hardcode GLOBAL_SEED in CONFIG and call set_seed(CONFIG["seed"]) at the very start. Never change this across rounds.

2. CONFIG dict: include all user-provided initial hyperparameters. Mark frozen params with a comment # FROZEN:

   "some_fixed_param": value,  # FROZEN
   "learning_rate": 1e-3,
   

   The CONFIG dict is the single source of truth — all hyperparameters live here. Do not hardcode values elsewhere.

3. Task-agnostic design: the training loop structure depends entirely on the user's task:

   • Loss function: infer from task type (MSE for regression/forecasting, CrossEntropy for classification, etc.) — or ask user if ambiguous
   • Metric: compute exactly what the user specified in Phase 1.7
   • Model forward interface: use whatever the user's Modelfile.py defines — read it carefully before writing the training loop
   • Do NOT assume any specific input shape, output shape, or field names

4. Dataset handling — based on Phase 1.2 answers:

   • Single file with test (0.7/0.1/0.2): split by index ranges, compute normalization stats on train split only if applicable
   • Separate train/val files: load each directly
   • Single file train/val only: split 80/20 by index
   • Use whatever Dataset/collate_fn code the user provided; if auto-generating, derive it from the actual file structure

5. Metric output format — at end of training, print exactly:

   ---
   <val_metric_name>: <value>
   <test_metric_name>: <value>   # only if test set exists
   

   These lines are parsed by grep in Step 4.2. The metric name must exactly match what the user specified.

6. Checkpoint saving: save model.state_dict() only. Save alongside a config_snapshot.json:

   import json
   json.dump(CONFIG, open(os.path.join(best_dir, "config_snapshot.json"), "w"), indent=2)
   

   Save checkpoint when PRIMARY_METRIC improves (test if available, else val).

Generate Modelfile.py

• User provided path: copy as Modelfile.py.
• User pasted: write directly.
• No model: use template from ~/.claude/skills/auto-experiment/scripts/Modelfile_template.py.

Generate accelerate_config.yaml

CPU mode:

compute_environment: LOCAL_MACHINE
distributed_type: NO
num_processes: 1
gpu_ids: ""
mixed_precision: "no"
use_cpu: true

Single GPU:

compute_environment: LOCAL_MACHINE
distributed_type: NO
num_processes: 1
gpu_ids: "<GPU_ID>"
mixed_precision: fp16

Multi-GPU:

compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
num_processes: <count>
gpu_ids: "<SELECTED_GPUS>"
mixed_precision: fp16

---

PHASE 3.5 — Audit Gate (Subagent Review Loop)

This phase runs BEFORE git init and BEFORE any training. Do not skip it.

Step 3.5.0 — Ensure Audit Agents Are Present

The two required subagents are bundled with this skill at:

~/.claude/skills/auto-experiment/agents/training-template-auditor.md
~/.claude/skills/auto-experiment/agents/ml-validity-auditor.md

审计前检查两个 agent 文件是否已存在于当前项目的 .claude/agents/ 目录下（project-level agents 的标准位置）：

PROJECT_AGENTS_DIR="$(git rev-parse --show-toplevel 2>/dev/null || pwd)/.claude/agents"
SKILL_AGENTS_DIR="$HOME/.claude/skills/auto-experiment/agents"

for agent in training-template-auditor.md ml-validity-auditor.md; do
  if [ ! -f "${PROJECT_AGENTS_DIR}/${agent}" ]; then
    echo "Agent ${agent} 未在项目 .claude/agents/ 下找到，从 skill 包复制..."
    mkdir -p "${PROJECT_AGENTS_DIR}"
    cp "${SKILL_AGENTS_DIR}/${agent}" "${PROJECT_AGENTS_DIR}/${agent}"
    echo "  ✓ 已复制到 ${PROJECT_AGENTS_DIR}/${agent}"
  else
    echo "  ✓ ${agent} 已存在于项目 .claude/agents/ 下"
  fi
done

确认两个 agent 都存在于项目 .claude/agents/ 后，再进入 Step 3.5.1。

---

After writing train.py and Modelfile.py (if applicable), run a mandatory audit-and-fix loop using the two ML subagents. The loop continues until both subagents approve the code.

The subagents are auditors, not editors:

• they must not modify code
• they must not rewrite code
• they must only report issues that materially affect correctness, rigor, reproducibility, template compliance, or result validity
• they should ignore minor style, naming, formatting, or non-critical refactor suggestions

Overview

┌─────────────────────────────────────────────────────┐
│  PHASE 3.5 — Audit Gate                             │
│                                                     │
│  1. Launch both ML auditors in parallel             │
│  2. Collect all material issues                     │
│  3. If any issues → fix them → repeat               │
│  4. If both approve → proceed to git init           │
└─────────────────────────────────────────────────────┘

Step 3.5.1 — Launch Both Subagents in Parallel

Use the Agent tool to launch both subagents simultaneously (single message with two tool calls).

training-template-auditor prompt:

Audit the following ML training code for training-engineering correctness and accelerate-template compliance.

Focus only on material issues that affect:
1. accelerate template compliance
2. train/eval loop correctness
3. backward/optimizer/scheduler ordering
4. device placement and CPU/GPU compatibility
5. checkpoint save/resume correctness
6. seed and reproducibility setup
7. output metric format correctness

Task context: <TASK_DESCRIPTION>

Files to audit:
- <EXP_DIR>/train.py
- model definition files under <EXP_DIR> (if any)

Report only material issues.
For each issue provide:
- file
- line number
- severity (P0 or P1)
- description
- why it matters

Do not modify code.
Do not rewrite code.
Do not provide patches.

If no material issues exist, respond exactly:
APPROVED: no material issues found.

ml-validity-auditor prompt:

Audit the following ML training code for experiment validity and result trustworthiness.

Focus only on material issues that affect:
1. data leakage
2. train/val/test separation
3. metric correctness against the stated goal: <TARGET_METRIC>
4. threshold/model-selection independence from test evaluation
5. prediction-label alignment
6. data pipeline correctness end-to-end
7. whether frozen parameters (<FROZEN_PARAMS>) are respected

Task context: <TASK_DESCRIPTION>

Files to audit:
- <EXP_DIR>/train.py
- model definition files under <EXP_DIR> (if any)

Report only material issues.
For each issue provide:
- file
- line number
- severity (P0 or P1)
- description
- why it matters

Do not modify code.
Do not rewrite code.
Do not provide patches.

If no material issues exist, respond exactly:
APPROVED: no material issues found.

Step 3.5.2 — Collect and Deduplicate Issues

After both subagents return:

• Parse each response for material issues or APPROVED status.
• Deduplicate overlapping issues (same file + same line + same root cause).
• If both subagents respond with APPROVED: no material issues found. → proceed to Step 3.5.4.
• Otherwise → go to Step 3.5.3.

Step 3.5.3 — Fix All Reported Material Issues

For each reported issue:

• Apply the fix in the main agent using the Edit tool.
• Do NOT modify CONFIG values or frozen params during fixes.
• Do NOT change the required metric output format.
• Fix only material correctness / rigor / validity issues.

After all fixes are applied, go back to Step 3.5.1 and re-run both subagents.

There is no auto-bypass. Do not proceed to git init or training until both subagents explicitly approve. If the same unresolved issues persist across multiple rounds, stop and escalate the unresolved issues to the user instead of proceeding.

Step 3.5.4 — Proceed

Print: ✓ Audit gate passed — both subagents approved. Proceeding to git init.

---

Initialize git

cd "${EXP_DIR}"
git init
cat > .gitignore << 'EOF'
run.log
results.tsv
checkpoints/
best_configs/
__pycache__/
*.pyc
EOF
git add train.py Modelfile.py accelerate_config.yaml .gitignore
git commit -m "baseline: initial config seed=<GLOBAL_SEED>"

Initialize results.tsv

Create (NOT git-tracked):

• Has test set: commit\tval_metric\ttest_metric\tstatus\tdescription
• No test set: commit\tval_metric\tstatus\tdescription

---

PHASE 4 — Experiment Loop

Run automatically without asking the user. Repeat until user presses Ctrl+C. Track round_num starting at 1.

Primary metric for keep/discard and best-config saving:

• Has test set → PRIMARY_METRIC = test_metric
• No test set → PRIMARY_METRIC = val_metric

Maintain in memory:

• best_primary_metric (initialized to null)
• best_primary_commit

Step 4.1 — Run Training

Use the venv's accelerate if the system one is not available. Detect automatically:

ACCELERATE_BIN=$(which accelerate 2>/dev/null || find /opt /Users -name "accelerate" -path "*/bin/accelerate" 2>/dev/null | head -1)
$ACCELERATE_BIN launch --config_file accelerate_config.yaml train.py > run.log 2>&1

Print status line: --- Round <N> training started ---

Step 4.2 — Extract Metrics

Use the exact metric names the user specified in Phase 1.7:

grep "^<VAL_METRIC_NAME>:" run.log
grep "^<TEST_METRIC_NAME>:" run.log   # only if test set exists

• Values found → parse as float (take the last occurrence if multiple epochs printed it).
• Empty → crash. Read tail -n 50 run.log. Set metric = null, status = "crash".

Step 4.3 — Call DeepSeek

Use requests (not openai SDK) to avoid connection issues. Keep prompt compact:

import json, time, requests

API_KEY = "<DEEPSEEK_API_KEY>"
payload = {
    "model": "deepseek-chat",
    "messages": [{"role": "user", "content": PROMPT}],
    "response_format": {"type": "json_object"},
    "temperature": 0.3,
    "max_tokens": 1024,
}
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}"}

for attempt in range(3):
    try:
        resp = requests.post("https://api.deepseek.com/chat/completions",
                             json=payload, headers=headers, timeout=90)
        resp.raise_for_status()
        result = json.loads(resp.json()["choices"][0]["message"]["content"])
        print(json.dumps(result))
        break
    except Exception as e:
        print(f"Attempt {attempt+1} failed: {e}")
        if attempt < 2: time.sleep(5)

Prompt template (keep compact, no full file contents — extract only CONFIG section):

You are an ML optimizer. History of all rounds:
<one line per round: round, val_metric, test_metric, status, description>

Current CONFIG (non-frozen params):
<only the non-frozen lines from CONFIG dict>

Current Modelfile.py summary:
<brief description of architecture, not full code>

FROZEN params (never change): <list>
Tunable params and ranges: <list>
Experiment mode: <hyperparams|architecture|both>
Target: <metric> <lower|higher> is better. Best so far: <value>

Round <N> result: val=<val>, test=<test or N/A>

Return JSON:
{
  "analysis": "brief",
  "action": "modify_hyperparams|modify_architecture|both",
  "config_changes": {"key": value, ...},
  "modelfile_new": "full Modelfile.py if architecture changed, else null",
  "reason": "one-line commit message"
}
IMPORTANT: Never modify these frozen params: <FROZEN_PARAMS list>

Step 4.4 — Apply Changes

Validate frozen params: check that config_changes does NOT contain any frozen param key. If it does, remove those keys silently before applying.

Update CONFIG in train.py: for each key in config_changes, find and replace the corresponding line in the CONFIG dict using the Edit tool.

Update Modelfile.py if modelfile_new is not null: overwrite the file.

Step 4.5 — Git Commit

git add train.py Modelfile.py
git commit -m "exp-<round_num>: <reason>"
COMMIT_HASH=$(git rev-parse --short HEAD)

Step 4.6 — Keep/Discard and Save Best

Use PRIMARY_METRIC (test if available, else val) for ALL keep/discard and best-config decisions.

First round OR current_primary improved over best_primary_metric:

• best_primary_metric = current_primary
• best_primary_commit = COMMIT_HASH
• Copy train.py → best_configs/best_train.py
• Copy Modelfile.py→ best_configs/best_Modelfile.py
• Copy checkpoint to checkpoints/best/model.pt + config_snapshot.json
• Print: ✓ Round <N>: <primary_metric_name>=<value> improved — keeping

Not improved (or crash):

• git reset --hard HEAD~1
• Update this round's row status in results.tsv to discard
• Print: ✗ Round <N>: <primary_metric_name>=<current> not better than best <best> — reverting

Note on has-test case: val is still computed and logged every round (for reference), but it does NOT drive keep/discard. Only test_metric drives all decisions.

Step 4.7 — Record to results.tsv

<COMMIT_HASH>\t<val_metric>\t<test_metric or N/A>\t<keep|discard|crash>\t<reason>

Increment round_num, go back to Step 4.1.

---

PHASE 5 — Summary (on Ctrl+C or user stop)

═══════════════════════════════════════════════════
  auto-experiment Summary
═══════════════════════════════════════════════════
Experiment dir:    <EXP_DIR>
Total rounds:      <N>
Seed (fixed):      <GLOBAL_SEED>
Primary metric:    <test_metric if test exists, else val_metric>

Best result:  <primary_metric_name> = <best_primary_metric>
  Commit:     <best_primary_commit>
  Config:     best_configs/best_train.py
  Model:      best_configs/best_Modelfile.py
  Weights:    checkpoints/best/model.pt
  Config JSON: checkpoints/best/config_snapshot.json

Full results:
<contents of results.tsv>
═══════════════════════════════════════════════════

To reproduce:
  cd <EXP_DIR>
  cp best_configs/best_train.py train.py
  cp best_configs/best_Modelfile.py Modelfile.py
  accelerate launch --config_file accelerate_config.yaml train.py

---

Error Handling

• DeepSeek connection reset: use requests (not openai SDK). Keep prompt under ~2000 tokens. Retry 3×.
• Frozen param in config_changes: silently remove before applying.
• Training crash: pass last 50 log lines to DeepSeek for diagnosis. Mark crash in results.tsv.
• Invalid JSON from DeepSeek: retry with "Return ONLY valid JSON, no markdown code blocks" prepended.

---

Important Notes

• Never ask the user questions after Phase 2. From Phase 3 onward, proceed automatically.
• Seed is sacred: CONFIG["seed"] is always GLOBAL_SEED, never modified by DeepSeek.
• Frozen params: validated and stripped from DeepSeek suggestions before applying.
• Single best: only one best is tracked — test_metric if test set exists, val_metric otherwise. No dual tracking.
• Best config reproducibility: best_configs/best_train.py + best_configs/best_Modelfile.py + checkpoints/best/config_snapshot.json are sufficient to fully reproduce the best result. These are plain files (not git-tracked) updated in-place whenever primary metric improves.
• run.log is overwritten each round (not git-tracked).
• results.tsv is appended each round (not git-tracked).
• After git reset --hard HEAD~1, working files revert — next DeepSeek call sees the reverted files.
• DeepSeek prompt must not include full file contents for large files — extract only CONFIG dict lines and a brief architecture summary to keep requests small and avoid connection resets.