literature-research

name	literature-research
description	Spawn a read-only research sub-agent to mine ML literature (arXiv, Semantic Scholar, blog writeups) for small-budget Vision Transformer / CIFAR-10 architecture and training tricks, before designing the next autoresearch experiment. Use at the start of a session, or whenever you are "out of ideas" for the next edit to test.py.

Literature Research (ported from huggingface/ml-intern's research sub-agent)

Why this exists

Architecture search by trial-and-error eventually plateaus: the same handful of knobs (layers, heads, mlp_dim, dropout, norm placement) get re-tried in slightly different combinations. Before that happens, spend one cheap, isolated research pass mining the literature for techniques specifically validated on small ViTs trained from scratch on small images with no extra data — that is a narrow, well-studied niche (CIFAR-10/100 ViT-from-scratch papers, "ViT for small datasets" papers, efficient attention/FFN variants under ~1M params).

This is a direct port of ml-intern's research tool / research sub-agent pattern (see agent/tools/research_tool.py in huggingface/ml-intern), adapted to run with zero API keys using only Claude Code's built-in WebSearch and WebFetch tools, plus the Agent tool for context isolation.

How to invoke it

Use the Agent tool (subagent_type: general-purpose or Explore) with a self-contained prompt — the sub-agent has no memory of this conversation, so give it everything it needs:

1. The hard constraints (copy verbatim from program.md):

   • .pt file must stay < 1 MB (1,048,576 bytes)
   • Only Part 5 (myFFN), Part 6 (myAttention), Part 7 (myTransformer), Part 9 (hyperparameters: hidden_dim/heads/head_dim/mlp_dim) are editable
   • myFFN must stay FFN(x) = max(0, xW1+b1)W2+b2 (two Linear + ReLU)
   • myAttention must stay real scaled dot-product attention softmax(QKᵀ/√d)V, no nn.MultiheadAttention
   • No data augmentation, no extra data, no random seed setting
   • 10 epochs, batch size 16, Adam lr=3e-4, fixed (cannot change)
   • Current architecture: 9-layer pre-LN transformer, hidden_dim=64, heads=16, head_dim=4, mlp_dim=72, attn_dropout=0.20, residual dropout=0.05

2. The research task — what to look for. Default template:

   Research task: Find architecture and training techniques for Vision
   Transformers trained FROM SCRATCH on CIFAR-10 (32x32 images, no extra data,
   no pretraining, no augmentation, only 10 epochs, batch size 16, Adam lr=3e-4)
   under a strict <1M parameter budget. Focus on: [specific angle, e.g.
   "tokenization/patch embedding alternatives", "attention variants for small
   models", "FFN/activation variants", "normalization placement
   (Pre-LN/Post-LN/Sandwich/ReZero)", "initialization schemes for fast
   convergence in <=10 epochs", "regularization for tiny datasets without
   augmentation"].
   
   For each technique found, report:
   - Source (paper title + arXiv ID, or blog/repo URL)
   - What it changes mechanistically (the actual math/code change)
   - What result it produced and on what benchmark/setup (be honest if the
     setup differs from ours — e.g. larger model, ImageNet, with augmentation)
   - Whether it is plausible to adapt to a 9-layer, hidden_dim=64, ~245K-param
     ViT trained for only 10 epochs with no augmentation
   - Estimated parameter cost (more heads/layers/mlp_dim = more params; norm
     placement / dropout / init changes = ~0 params)
   
   Output a ranked list of 5-8 candidate techniques, ranked by
   (a) plausibility of mechanistic benefit at this scale, (b) ~0 or low param
   cost preferred. Be skeptical: many tricks from large-model papers (e.g.
   stochastic depth, large-batch tricks, heavy augmentation) do not transfer to
   a 245K-param model trained for 10 epochs on raw CIFAR-10 — flag these
   explicitly as "likely doesn't transfer" rather than recommending them.
   

3. Tools the sub-agent should use (all available natively, no keys needed):

   • WebSearch — find papers/blogs (e.g. "vision transformer CIFAR-10 small dataset training from scratch", "ViT inductive bias small data", "compact transformer CIFAR-10 parameters")
   • WebFetch on:
     • https://arxiv.org/abs/<id> and https://arxiv.org/html/<id> — full paper text (works without any key)
     • https://api.semanticscholar.org/graph/v1/paper/search?query=... and .../paper/<id>/citations — citation graphs (unauthenticated requests work, just rate-limited; this mirrors ml-intern's hf_papers citation_graph operation but via the public S2 REST API directly)
     • GitHub repo READMEs for compact-ViT implementations (e.g. "compact-transformers", "vit-pytorch" small-dataset variants) for working code patterns

Output contract (recipe table)

Mirror ml-intern's research sub-agent output format — a ranked table, NOT prose:

Rank	Technique	Source	Mechanistic change	Param cost	Plausibility at this scale
1	...	arXiv:xxxx.xxxxx	...	+0 / +~Nk	high/medium/low + why

Followed by a short "Recommendations" section: which 1-2 techniques to try next, in what order, and — critically — the mechanistic justification required by program.md's keep/discard rules ("accuracy happened to be higher" is not sufficient; you need a reason why it should help).

Integrating into the autoresearch loop (program.md)

Insert as an optional step before step 2 ("Design the next experiment"):

1b. (Optional, when out of ideas or at session start): invoke the literature-research skill. Spawn a research sub-agent (Agent tool, general-purpose) with the template above, focused on one architectural angle not yet explored. Use its ranked recipe table to pick the next experiment's mechanistic justification.

Keep this occasional (e.g. every 5-10 experiments, or when a string of discards suggests the local search has plateaued) — it's a context-isolated, one-shot call, not a per-experiment requirement. The existing size_check.py / analyze.py / mechanistic-justification rules in program.md are unchanged and still govern the keep/discard decision.