literature-research

name	literature-research
description	Spawn a read-only research sub-agent to mine ML literature (arXiv, Semantic Scholar, blog writeups) for small-budget Vision Transformer / CIFAR-10 architecture and training tricks, before designing the next autoresearch experiment. On this branch (autoresearch/mandatory-lit), this is MANDATORY every iteration, run in citation-chain mode (see "Citation-Chain Mode" section below).

Literature Research (ported from huggingface/ml-intern's research sub-agent)

Why this exists

Architecture search by trial-and-error eventually plateaus: the same handful of knobs (layers, heads, mlp_dim, dropout, norm placement) get re-tried in slightly different combinations. Before that happens, spend one cheap, isolated research pass mining the literature for techniques specifically validated on small ViTs trained from scratch on small images with no extra data — that is a narrow, well-studied niche (CIFAR-10/100 ViT-from-scratch papers, "ViT for small datasets" papers, efficient attention/FFN variants under ~1M params).

This is a direct port of ml-intern's research tool / research sub-agent pattern (see agent/tools/research_tool.py in huggingface/ml-intern), adapted to run with zero API keys using only Claude Code's built-in WebSearch and WebFetch tools, plus the Agent tool for context isolation.

How to invoke it

Use the Agent tool (subagent_type: general-purpose or Explore) with a self-contained prompt — the sub-agent has no memory of this conversation, so give it everything it needs:

1. The hard constraints (copy verbatim from program.md):

   • .pt file must stay < 1 MB (1,048,576 bytes)
   • Only Part 5 (myFFN), Part 6 (myAttention), Part 7 (myTransformer), Part 9 (hyperparameters: hidden_dim/heads/head_dim/mlp_dim) are editable
   • myFFN must stay FFN(x) = max(0, xW1+b1)W2+b2 (two Linear + ReLU)
   • myAttention must stay real scaled dot-product attention softmax(QKᵀ/√d)V, no nn.MultiheadAttention
   • No data augmentation, no extra data, no random seed setting
   • 10 epochs, batch size 16, Adam lr=3e-4, fixed (cannot change)
   • Current architecture: 9-layer pre-LN transformer, hidden_dim=64, heads=16, head_dim=4, mlp_dim=72, attn_dropout=0.20, residual dropout=0.05

2. The research task — what to look for. Default template:

   Research task: Find architecture and training techniques for Vision
   Transformers trained FROM SCRATCH on CIFAR-10 (32x32 images, no extra data,
   no pretraining, no augmentation, only 10 epochs, batch size 16, Adam lr=3e-4)
   under a strict <1M parameter budget. Focus on: [specific angle, e.g.
   "tokenization/patch embedding alternatives", "attention variants for small
   models", "FFN/activation variants", "normalization placement
   (Pre-LN/Post-LN/Sandwich/ReZero)", "initialization schemes for fast
   convergence in <=10 epochs", "regularization for tiny datasets without
   augmentation"].
   
   For each technique found, report:
   - Source (paper title + arXiv ID, or blog/repo URL)
   - What it changes mechanistically (the actual math/code change)
   - What result it produced and on what benchmark/setup (be honest if the
     setup differs from ours — e.g. larger model, ImageNet, with augmentation)
   - Whether it is plausible to adapt to a 9-layer, hidden_dim=64, ~245K-param
     ViT trained for only 10 epochs with no augmentation
   - Estimated parameter cost (more heads/layers/mlp_dim = more params; norm
     placement / dropout / init changes = ~0 params)
   
   Output a ranked list of 5-8 candidate techniques, ranked by
   (a) plausibility of mechanistic benefit at this scale, (b) ~0 or low param
   cost preferred. Be skeptical: many tricks from large-model papers (e.g.
   stochastic depth, large-batch tricks, heavy augmentation) do not transfer to
   a 245K-param model trained for 10 epochs on raw CIFAR-10 — flag these
   explicitly as "likely doesn't transfer" rather than recommending them.
   

3. Tools the sub-agent should use (all available natively, no keys needed):

   • WebSearch — find papers/blogs (e.g. "vision transformer CIFAR-10 small dataset training from scratch", "ViT inductive bias small data", "compact transformer CIFAR-10 parameters")
   • WebFetch on:
     • https://arxiv.org/abs/<id> and https://arxiv.org/html/<id> — full paper text (works without any key)
     • https://api.semanticscholar.org/graph/v1/paper/search?query=... and .../paper/<id>/citations — citation graphs (unauthenticated requests work, just rate-limited; this mirrors ml-intern's hf_papers citation_graph operation but via the public S2 REST API directly)
     • GitHub repo READMEs for compact-ViT implementations (e.g. "compact-transformers", "vit-pytorch" small-dataset variants) for working code patterns

Output contract (recipe table)

Mirror ml-intern's research sub-agent output format — a ranked table, NOT prose:

Rank	Technique	Source	Mechanistic change	Param cost	Plausibility at this scale
1	...	arXiv:xxxx.xxxxx	...	+0 / +~Nk	high/medium/low + why

Followed by a short "Recommendations" section: which 1-2 techniques to try next, in what order, and — critically — the mechanistic justification required by program.md's keep/discard rules ("accuracy happened to be higher" is not sufficient; you need a reason why it should help).

Citation-Chain Mode (mandatory, this branch)

This branch (autoresearch/mandatory-lit) replaces the "optional, every 5-10 experiments, independent broad search" pattern above with a mandatory, cumulative, citation-chain pattern, as Step 0 of every loop iteration (see program.md). The motivation: the main branch's 13 independent broad searches across 120 experiments produced heavy redundancy (the same "large-model" tricks — fixed residual scaling, stochastic depth, LayerScale — were independently recommended 2-3x by different searches, and all failed at this scale). Citation-chaining instead walks outward from papers already known to be relevant, building a single connected graph over time, recorded in LITERATURE_LOG.md.

How it differs from the broad-search mode above

	Broad search (main branch, optional)	Citation-chain (this branch, mandatory)
Frequency	every 5-10 experiments	every iteration
Starting point	fresh WebSearch each time	a paper already in LITERATURE_LOG.md
Discovery mechanism	keyword search	follow that paper's references/citations
Memory across calls	none (each call independent)	LITERATURE_LOG.md — full graph, append-only
Redundancy risk	high (same tricks rediscovered)	lower (subagent sees prior log entries)

Step 0 procedure

1. Pick a frontier node. Read LITERATURE_LOG.md. Pick a row to expand: prefer the highest-relevance row whose citations haven't been explored yet (no child rows with found_via: cites:<this paper_id>). If the log is empty (first iteration), do ONE seed search using the broad-search template above, focused on a narrow angle (e.g. "ViT for small datasets / CIFAR-10 from scratch") — this becomes hop_depth=0.

2. Spawn the subagent (Agent tool, general-purpose). Self-contained prompt must include:

   • The hard constraints (same list as the broad-search template above)
   • The frontier paper's identity (arXiv ID/title) and a short excerpt of LITERATURE_LOG.md (last ~10 rows) so it knows what's already been explored/tried and can avoid re-recommending failed techniques
   • The task: "Fetch this paper's references/citations (via https://arxiv.org/abs/<id>, or https://api.semanticscholar.org/graph/v1/paper/<id>/citations and .../references). From the citation/reference list, pick 1-2 papers most relevant to a <250K-300K param ViT trained from scratch for 10 epochs on CIFAR-10 (32x32, no augmentation, no pretraining, batch 16, Adam lr=3e-4). For each: report title, arXiv ID/URL, what it changes mechanistically, and plausibility at this scale (be skeptical of large-model/long-schedule tricks — flag explicitly if 'likely doesn't transfer', citing LITERATURE_LOG.md rows with the same outcome if any)."

3. Hop depth limit & chain switching: cap each chain at hop_depth 3-4. If two consecutive hops both yield low-relevance papers (large-model / long-schedule / different-domain), stop that chain and start a new seed search next iteration instead of forcing a 5th hop down an unproductive path. Multiple chains may be open at once — LITERATURE_LOG.md is a forest, not a single path.

4. Append to LITERATURE_LOG.md: one row per new paper, with found_via: cites:<parent_paper_id>, hop_depth = parent.hop_depth + 1, honest relevance, and concrete techniques_extracted. Leave tried_at_exp/outcome blank until step 9b of the loop.

5. Feed into step 2 (design next experiment): the extracted technique(s) are the primary candidates. If all candidates from this iteration are low-relevance or duplicate a row already marked outcome: discard, step 2 may fall back to a grid-search-style tweak — but record in the commit message which LITERATURE_LOG.md row(s) were considered and why rejected.

The existing size_check.py / analyze.py / mechanistic-justification rules in program.md are unchanged and still govern the keep/discard decision — Step 0 only changes where the candidate ideas come from, not the keep/discard criteria.