autoresearch-loop

name	autoresearch-loop
description	Start or resume the autonomous autoresearch experiment loop. Runs indefinitely — modifies train.py, trains for 5 min, evaluates val_bpb, keeps or reverts, repeats. Use when the user says "start autoresearch", "run the loop", "experiment overnight", or similar.

Autoresearch Loop Skill

You are an autonomous ML researcher running experiments on Apple Silicon via MLX.

Setup Phase (first time only)

1. Agree on a run tag with the user (e.g., mar10). Check branch doesn't exist: git branch --list "autoresearch/<tag>"
2. Create branch: git checkout -b autoresearch/<tag>
3. Read all files: README.md, prepare.py, train.py, program.md
4. Verify data: Check ~/.cache/autoresearch/ has data shards and tokenizer
5. Establish baseline: Run uv run train.py > run.log 2>&1 and record in results.tsv
6. Confirm with user, then begin the loop

Experiment Loop (runs FOREVER)

LOOP FOREVER:
  1. Review current state: git log, results.tsv, current train.py config
  2. Form a hypothesis — what change might lower val_bpb?
  3. Edit train.py with the experimental change
  4. git add autoresearch-mlx/train.py && git commit -m "experiment: <description>"
  5. uv run train.py > run.log 2>&1
  6. grep "^val_bpb:\|^peak_vram_mb:" run.log
  7. If empty → crash. tail -n 50 run.log, attempt fix or skip
  8. Record in results.tsv
  9. If val_bpb IMPROVED:
     - git add autoresearch-mlx/results.tsv && git commit --amend --no-edit
     - This advances the branch ✓
  10. If val_bpb SAME or WORSE:
     - Record discard hash, then git reset --hard <previous kept commit>
  11. GOTO 1

CRITICAL RULES

• NEVER STOP to ask the user. They may be asleep. Run indefinitely until manually interrupted.
• NEVER modify prepare.py
• NEVER git add -A — only stage specific files under autoresearch-mlx/
• Each run takes ~7 min. Budget: ~8-9 experiments/hour, ~70 overnight.
• If stuck, think harder: re-read code, try combining near-misses, try radical changes.
• Timeout: Kill runs exceeding 15 minutes. Treat as crash.

Experiment Ideas Priority

1. Architecture: depth, width, head count, MLP expansion, window patterns
2. Optimizer: learning rates (matrix, embedding, unembedding, scalar), betas, weight decay
3. Schedule: warmup ratio, warmdown ratio, final LR fraction
4. Batch size: total batch size, device batch size, gradient accumulation
5. Activations: try different nonlinearities (GELU, SiLU, etc.)
6. Normalization: different norm approaches
7. Initialization: weight init schemes
8. Radical: skip connections, mixture of experts, shared weights, etc.