hf-model-researcher

name

hf-model-researcher

description

Research a newly released Hugging Face model and produce a comprehensive deep-research report (model card + technical reports + novelty analysis vs predecessor + glossary + benchmark analysis), exported as markdown + PDF + metadata.json, optionally emailed. Invoke with a HuggingFace model ID like moonshotai/Kimi-K2-Thinking. Use when the user gives a HF model identifier or asks to deeply research a model release.

HF Model Research Agent

Purpose

Research newly released Hugging Face models to generate comprehensive, educational documentation that explains novel techniques from first principles.

Philosophy: DEEP Research, Not Surface Summaries

This is a DEEP research agent. Explain concepts as if the reader is unfamiliar with them. Trace innovations back to their original papers and first principles. Build understanding progressively. Use analogies.

Bad: "The model uses DSA for efficiency."

Good:

"The model introduces DeepSeek Sparse Attention (DSA), a novel attention mechanism that reduces computational complexity. Traditional attention computes relationships between every pair of tokens, scaling quadratically O(n²) with sequence length — processing 100K tokens requires 10 billion operations. DSA instead uses a two-stage approach: (1) a lightweight 'indexer' identifies the ~2,048 most relevant tokens for each query, then (2) full attention is computed only on those selected tokens. This reduces complexity to O(n×k) where k<<n. The technique builds on prior sparse attention work (Longformer, BigBird) but achieves finer granularity through learned token selection rather than fixed patterns. [See: arXiv:2512.02556, Section 3.2]"

Prose Style — write analysis, not a form

Deep research is only useful if it reads like analysis. You do NOT write research_report.md directly — you write a fact-dense research_notes.md and delegate the draft to the prose subagent (see §2.1, step 5). prose reads prose_style_prompt.md (next to this file) and follows it in its own clean context, then runs Slop Guard. The non-negotiables it enforces:

• No labeled-bullet template. Do NOT write "What's New" techniques as **Problem:** … **Mechanism:** … **Prior Art:** … **Tradeoffs:** … **Verification:** … bullets. Cover those five aspects as 2–4 flowing paragraphs with strong topic sentences.
• Say verification status once (a closing line per claim, or one "Claims-Evidence Mapping" table) — never repeat *Vendor claim, not independently reproduced.* after every subsection.
• Punctuation discipline. No sentence over ~35 words; one em-dash max per sentence (never two); parentheticals <8 words or pulled into their own sentence; don't stack bold+italics; cut headline language ("capstone," "headline innovation") and empty hedges ("arguably," "essentially").
• Vivid specifics in prose. Weave exact numbers into the sentence where they matter, not only into tables. MTP-2 loss spiked first, then everything is the register to aim for.

Model routing & parallelism (EFFICIENT BY DEFAULT)

This workflow is model-routed and parallelized to use your models well:

• You (the parent/orchestrator) should run on the STRONG model (e.g. lambda-glm/glm5.2). You do the judgment work: identifying the predecessor and arXiv IDs, novelty analysis, config diffing, and synthesizing the report. Select it with /model before starting.
• The bulk fetch+extract work runs in PARALLEL on the FAST model via the hf-retriever subagent (pinned to lambda-kimi/kimi-k2.7-code, with lambda-glm/glm5.2 fallback). Reading papers/configs/docs is mechanical and token-heavy — the cheap fast model does 5 clusters at once while you stay free to synthesize.
• The draft is written by the prose subagent (strong model, its own clean context). After synthesis you write a fact-dense research_notes.md and delegate the prose to prose — it reads prose_style_prompt.md and runs Slop Guard, so writing quality is enforced in an isolated context instead of competing with synthesis for your attention. See §2.1 step 5.
• To swap the fast model later (e.g. when you add a faster one), edit ONE line: the model: field in ~/.pi/agent/agents/hf-retriever.md.

The concrete fan-out plan is in §2.1, "Parallel Execution Plan" below. Use it for ultradeep/deep modes; for quick/standard you may fetch serially yourself.

Tooling (pi-native — IMPORTANT)

pi has no built-in WebSearch/WebFetch. Fetch raw files with curl and pages with the helpers below. (The original Claude Code workflow had a hook forcing curl for raw HuggingFace/GitHub URLs; in pi we simply use curl by default.)

Need	Command
HF raw README/config	curl -sL "https://huggingface.co/{model_id}/raw/main/README.md" (or /raw/main/config.json)
Web search	python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "<query>" [N]
arXiv search	python3 ~/.pi/agent/skills/deep-research/scripts/arxiv_search.py "<query>" [N]
Fetch URL → text file	python3 ~/.pi/agent/skills/deep-research/scripts/web_fetch.py "<url>" "<org>/<model>/<file>.txt"
Save facts (context hygiene)	python3 ~/.pi/agent/skills/deep-research/scripts/research_state.py save --session <slug> --key K --content "..." --sources N --tags a,b
Record citation	python3 ~/.pi/agent/skills/deep-research/scripts/citation_store.py add --session <slug> --url U --title "..." --tier 1
Validate deliverable	python3 ~/.pi/agent/skills/hf-model-researcher/scripts/validate_model_report.py ./{org}/{model}
Markdown → PDF	python3 ~/.pi/agent/skills/hf-model-researcher/scripts/md_to_pdf.py <in.md> <out.pdf> "Title"
Email PDF	python3 ~/.pi/agent/skills/hf-model-researcher/scripts/send_email.py <file.pdf> <to> "Subject"

Helper skills: /skill:fetch-model <id> [predecessor] and /skill:fetch-arxiv <id|query>.

---

Workflow

When given a Hugging Face model identifier:

1. Fetch Model Card & Technical Reports (Prerequisites)

Use curl (or the fetch-model skill) to fetch raw model data:

• https://huggingface.co/{model_id}/raw/main/README.md
• https://huggingface.co/{model_id}/raw/main/config.json

Also fetch predecessor model configs for comparison:

• https://huggingface.co/{org}/{previous_model}/raw/main/config.json

Extract key metadata: architecture type, organization, model family, parameter count, license.

1.5. Find Technical Reports (CRITICAL — DO THIS EARLY)

Before deep research, actively search for and fetch technical reports. These are the most authoritative sources and often contain details not in model cards.

Where to find technical reports:

1. Check the model card first — read the fetched readme_raw.md and grep for:

   • arXiv links (arxiv.org/abs/XXXX.XXXXX or arxiv.org/html/XXXX.XXXXX)
   • links to "technical report", "paper", "blog post", DOI references

2. Search arXiv directly (structured, reliable):

   python3 ~/.pi/agent/skills/deep-research/scripts/arxiv_search.py "{model_family} technical report" 5
   python3 ~/.pi/agent/skills/deep-research/scripts/arxiv_search.py "{org_name} {model_name}" 5
   

3. Check the organization's publications — GitHub repos often link to papers; company blogs announce papers. Use web_search.py or fetch candidate URLs.

4. Fetch using the /html/ endpoint:

   python3 ~/.pi/agent/skills/deep-research/scripts/web_fetch.py "https://arxiv.org/html/{arxiv_id}" "./{org}/{model}/paper_{arxiv_id}.txt"
   

   (If web_fetch reports no HTML version, fall back to /abs/.)

Why this matters: model cards are often sparse on methodology; technical reports contain the "why" behind design choices, benchmark methodology, and training data/process.

Architecture (how the three skills compose)

This skill is Part A — the harness. It does not implement its own research engine or its own prose writer; it orchestrates two other skills:

• Part A — hf-model-researcher (this skill, you on the strong model): identifies the model + predecessor + arXiv IDs, sets up the output dir and research-state/citation stores, invokes Part B for the research, compiles a fact-dense research_notes.md, then delegates the draft to the summarizer. Finally writes metadata.json and validates + exports.
• Part B — deep-research (~/.pi/agent/skills/deep-research/SKILL.md): the actual research engine. Runs the 8-phase pipeline (Scope → Plan → Retrieve → Triangulate → Synthesize → Critique → Refine → Package). Its Phase 3 RETRIEVE is model-routed via the hf-retriever subagent fan-out in §2.1 below — that fan-out IS deep-research's retrieval phase, specialized for HF model research (it is not a separate mechanism).
• Summarizer — prose subagent: takes the compiled research_notes.md and writes research_report.md as flowing analytical prose + Slop Guard (see §2.1 step 5). Clean context, no fetched-source clutter.

Data flow: hf-model-researcher → deep-research (briefs + research_state) → orchestrator compiles research_notes.md → prose writes research_report.md → orchestrator writes metadata.json + exports.

2. Deep Research Phase

IMPORTANT: Use ultradeep mode to maximize source gathering (20-45 min, 50+ sources).

Context Hygiene (CRITICAL — PREVENTS CONTEXT BLOAT):

SLUG="{org}-{model}"
python3 ~/.pi/agent/skills/deep-research/scripts/research_state.py init --session "$SLUG" --question "Research {model_name}"
python3 ~/.pi/agent/skills/deep-research/scripts/citation_store.py init --session "$SLUG"

After EACH fetch that returns useful info:

python3 ~/.pi/agent/skills/deep-research/scripts/citation_store.py add --session "$SLUG" --url "https://..." --title "..." --tier 1
python3 ~/.pi/agent/skills/deep-research/scripts/research_state.py save --session "$SLUG" --key "architecture" \
    --content "Key facts: 1) 16 layers 2) GQA ratio 4:1 3) ..." --sources 1 --tags "architecture"

The Pattern: fetch returns 5000 words → extract 10 key facts → save to research_state.py → context freed. Do NOT keep raw fetched content in conversation. Target <60K tokens context (the prior 25K target was unrealistic — the last two Nemotron-3-Ultra runs peaked at 181K and 218K because the orchestrator read full extracted tech reports; use grep -n windows, §2.1).

Load and execute the deep-research skill (read ~/.pi/agent/skills/deep-research/SKILL.md and run its 8-phase pipeline in ultradeep mode) with this structured research query:

Research the {model_name} model from {org_name}:

1. **What's Novel (PRIORITY FOCUS)** — what is NEW vs the predecessor? New
   attention/architecture, new training/data strategies, new inference
   optimizations, config.json diffs between versions, benchmark improvements
   with specific deltas.
2. **Technical Deep Dives** — for EACH novel technique: what problem it solves,
   how it works (mechanism), what prior work it builds on (cite original
   papers), tradeoffs.
3. **Prior Models & Evolution** — previous models from {org_name}, changelog,
   parameter-count comparisons, benchmark deltas.
4. **Target Audience & Use Cases** — modality, primary use cases, license
   implications, deployment requirements.
5. **Competitive Landscape** — similar models, benchmark comparisons,
   cost/performance tradeoffs.

2.1 Parallel Execution Plan (deep-research Phase 3 RETRIEVE, model-rated)

This fan-out IS deep-research's Phase 3 RETRIEVE, specialized for HF model research: instead of fetching every source serially on the strong model, fan out parallel hf-retriever subagents on the fast model, one per source cluster. Each returns a compressed brief on disk; you read the briefs (small context) and synthesize. This is the core efficiency win.

A brief must never be a primary source's only representation. Every Tier-1 paper the retrievers fetch must also be persisted to sources/ as full extracted text (paper_{arxiv_id}.txt), not only summarized in the brief. The brief is the synthesis aid; the persisted text is the ground truth the orchestrator greps during synthesis and the numeric-verify pass checks against. This is the single most important rule in this section — a flagship tech report that exists only as a 72-line brief is the pipeline's biggest knowledge leak.

Source-count bounds (per cluster): each retriever fetches a MIN 3, MAX 8 sources — the orchestrator-provided URLs/IDs count toward this. Across the clusters that yields 15–40+ sources, satisfying deep-research's "min 10 sources" floor and ultradeep's higher bar without unbounded fetching. If a cluster cannot reach 3 primary (Tier 1–2) sources, note it in the brief under "Unverified / Flagged" and continue — do not pad with low-tier sources.

Before fanning out, you (strong model) do the cheap judgment prep:

1. Fetch the model card + config + (if you can identify it) predecessor config via curl (or /skill:fetch-model). Save the model card to ./{org}/{model}/sources/hf-readme.md — it is a Tier-1 source and the model-card cluster (below) mines it.
2. From readme_raw.md, identify: org, model_family, the predecessor model id, all arXiv IDs, and any blog/GitHub/doc URLs. Decide the predecessor if the card doesn't state it (family + version numbering).
3. Create the output dir and the sources dir: mkdir -p ./{org}/{model}/sources.

Then fan out with one subagent PARALLEL call — tasks, each agent: "hf-retriever". The fast model is already pinned on the agent, so no per-task model field is needed. Give each task the contract fields the retriever expects (cluster, target, predecessor, arxiv_ids, known_urls, out, session), as a compact key=value | key=value string. The baseline clusters:

• tech-report — fetch the arXiv /html/ paper(s) and persist the full extracted text to sources/paper_{arxiv_id}.txt; extract architecture/training/benchmarks and NOVEL contributions into the brief.
• model-card — mine the fetched sources/hf-readme.md itself (deployment, hardware, reasoning modes, held-out-benchmark gates, test-time-scaling, model card benchmark tables). The model card is a Tier-1 source; do not let it sit unread after being fetched.
• predecessor — fetch predecessor config + its tech report (persist to sources/); extract what the predecessor ALREADY had (for novelty comparison).
• framework-docs — vLLM/SGLang/Transformers docs; deployment + inference optimizations.
• benchmarks — independent scores (swebench, artificialanalysis, lmsys).
• competitors — similar models from other orgs; benchmark comparisons.
• prior-work — background prior-art papers the new model builds on (Mamba-2, GQA, RoPE, MTP, GRPO, EAGLE, Medusa, RLHF, DPO, FFN Fusion, etc.). Given the list of prior-art techniques the orchestrator identifies from the tech report, resolve each via arxiv_search.py "<technique name>" (take the top hit), record the true ID + title + authors to the citation store, and fetch the abstract (full text not required — abstracts are sufficient to verify the citation points at the right paper). This cluster exists because prior-work papers are exactly the citations that get written from model memory when no cluster fetches them — and a hallucinated-but-real arXiv ID (one that resolves to a different paper) sails through the fetch-success citation gate. The prior-work cluster is the provenance that lets the citation-identity gate (validator) confirm each ID.

Scale extraction to source weight. The baseline fan-out gives a 40-page flagship tech report the same budget as the competitors cluster. For any tech report over ~20 pages, shard by section: fan out one tech-report retriever per major section (tech-report-pretraining, tech-report-posttraining, tech-report-quantization, tech-report-inference, tech-report-infra) instead of one for the whole paper. Each shard persists its section's text to sources/paper_{id}_{section}.txt. This is the single change that closes most of the knowledge gap on a big release.

Section-coverage completeness (do not skip sections). The Nemotron-3-Ultra audit found the depth gap vs the competing report came specifically from sections that were never assigned a retriever — quantization, inference, infrastructure, data, MOPD, MTP-boosting. So: after the tarball is extracted into sources/<id>_src/sections/*.tex, enumerate those section files and either (a) assign every major section to a tech-report-<section> shard (the shard list above is a menu, not a ceiling), or (b) write an explicit SECTIONS NOT SHARDED: <comma-separated list> line into research_notes.md naming every section you chose not to shard. There is no third option — a section file that exists on disk but that no brief references and notes never acknowledges is the exact depth gap the audit flagged, and validate_model_report.py now warns on it.

Example task string per cluster (use absolute paths for out and cwd):

cluster=tech-report | target=<org>/<model> | family=<family> | arxiv_ids=<id1,id2> | known_urls=<blog,github> | out=<abs>/{org}/{model}/brief_tech-report.md | sources_dir=<abs>/{org}/{model}/sources | session=<slug>-techreport

Set cwd to the absolute workspace path so briefs/citation files land in the right place. Pass sources_dir=<abs>/{org}/{model}/sources to every retriever so full text is persisted there. Each retriever uses its own --session so the shared research_state/ files don't race.

The PARALLEL call returns when all complete. If a retriever reports a fetch was blocked, re-launch just that one cluster. For bigger runs, add clusters (training-data, safety, license-deployment) — the retriever handles any cluster= label.

After the fan-out returns, you (strong model) synthesize:

1. read the briefs (small, fact-dense — context stays <60K). Never read an extracted paper end-to-end — use targeted grep -n '<pattern>' -A20 -B20 sources/paper_*.txt windows to pull only the lines you need. The last two Nemotron-3-Ultra runs peaked at 181K and 218K tokens precisely because the orchestrator read full extracted tech reports during synthesis; grep -n windows keep it bounded. Grep the persisted sources/ text for any fact a brief under-represents; the brief is a pointer, not the source.

2. Diff predecessor_config.json vs config.json yourself (the novelty hinge).

3. Merge each retriever's research_state/citations_<session>.json into one bibliography and renumber [N].

4. Write a fact-dense research_notes.md to disk (NOT prose — raw material), organized by report section: Executive Summary key points; Glossary terms + definitions + origins; each novel technique with its problem/mechanism/prior-art/tradeoffs/verification FACTS (not sentences); architecture details + the config-diff results; benchmark tables (numbers only); limitations; and the merged bibliography. This is the handoff doc for the prose agent — put the report structure (section list with one-line descriptions) at the top so prose knows what goes where. No memory-sourced citations (hard rule). No arXiv ID and no paper title may appear in research_notes.md, the Glossary, or the Bibliography unless it already exists in the run's citation store (citations_*.json) with a fetch record — i.e. a retriever actually resolved it. The Nemotron-3-Ultra NVFP4 report cited two arXiv IDs (2405.21052, 2404.05067) that resolve to real but wrong papers (RydbergGPT; a TDOA robotics paper) because they were written from model memory and happened to exist on arXiv. Every prior-art paper the report names must come from the prior-work retriever cluster (§2.1 fan-out) which records the true ID + title + authors. Glossary entries are claims too: every glossary row that cites a paper needs that paper in the citation store with a fetch record. If you would cite a paper the prior-work cluster did not fetch, re-launch that one cluster for it — do not write the ID from memory. This covers every attribution, not only arXiv IDs: an author name, a license-acronym expansion, or a benchmark's authorship must also trace to a fetched source in sources/ or the citation store — never written from memory. The NVFP4 run shipped a glossary that expanded OpenMDW two different unsourced ways and attributed Humanity's Last Exam to "Phong" instead of "Phan"; both were memory-sourced. A non-arXiv attribution needs its own fetched source (fetch the license page; arxiv_search.py the benchmark name) plus a citation-store / identity-cache entry, exactly as an arXiv ID does.

   Coverage contract at the notes boundary: raw tables travel verbatim (e.g. a MOPD ablation's raw SFT/RLVR/MOPD1/MOPD2/teacher scores, not just derived recovery percentages — raw is strictly more informative). Run the mechanical number diff after writing notes:

   python3 ~/.pi/agent/skills/hf-model-researcher/scripts/notes_number_diff.py ./{org}/{model}
   

   It lists every numeric token in brief_*.md that does not appear in research_notes.md. You must consciously accept every dropped number: for each, either fold it into notes, or append a line of the form DROPPED, why: <number> — <one-line reason it is not report-worthy> to research_notes.md. Re-run the diff until it reports clean (a DROPPED, why: line acknowledges a number, so re-runs don't re-flag it). Corollary: research_notes.md should be ≥2× the target report length. If notes are shorter than the report, the prose agent is forced to expand beyond its facts — and on a release that postdates the model's training data, every expansion is a hallucination vector. validate_model_report.py makes the <1× case an ERROR and the <2× case a warning.

5. Delegate the draft to the prose subagent — one SINGLE call, cwd set to the absolute workspace path: agent: "prose", task: "Read ./{org}/{model}/research_notes.md for the organized facts, and read <skills-dir>/hf-model-researcher/prose_style_prompt.md for the style rules. You may also read files in ./{org}/{model}/sources/ to verify or enrich a fact, but do not introduce facts absent from research_notes.md. Write ./{org}/{model}/research_report.md as flowing analytical prose following the report structure in research_notes.md — NOT a labeled-bullet template. Cover problem/mechanism/prior art/tradeoffs/ verification as 2–4 paragraphs per technique with strong topic sentences. Then run sg -j on the draft file, fix real issues once, and return the final prose. Do NOT write metadata.json." prose runs on the strong model in its own clean context (no fetched-source clutter), so its entire focus is writing well + Slop Guard.

6. Numeric-verify pass (Critique phase, grounded). After the draft is written, launch one reviewer subagent (fresh context, cwd = workspace) to verify every number in research_report.md against the persisted sources/ text and research_notes.md, and to explicitly reconcile cross-source disagreements (e.g. model card 70.7 vs report table 71.7 → write "the card and the paper disagree"; silence is a bug). The checker reads the draft + sources/, returns a list of any ungrounded or inconsistent numbers, and you fix them before export. The reviewer's enumerated scope also includes the Bibliography and the Glossary — verify every arXiv ID resolves to the paper its line names (title + authors match) and every glossary row that cites a paper has a source in the citation store. The two wrong IDs in the Nemotron-3-Ultra NVFP4 report survived specifically because citations were not in the numeric-verify task list; validate_model_report.py's citation-identity gate backstops this, but the reviewer must check first. This makes the deep-research Critique phase numeric-grounded rather than rhetorical.

7. Completeness-critic pass (inverse reviewer, fresh context). The numeric-verify pass checks numbers present in the draft; nothing asks what is missing. So immediately after it, launch a second reviewer subagent (fresh context, cwd = workspace) with the inverse mandate: read the briefs and skim the sources/<id>_src/sections/*.tex files the retrievers unpacked, then list the 10 most report-worthy findings absent from research_report.md, each with a source pointer (<file>:<line> or arXiv:<id> §X.Y). Bias toward the sections no dedicated brief covered (quantization, inference, infrastructure, data, MOPD, MTP-boosting). You (orchestrator) triage the list: fold the accepted items into research_notes.md (as facts) and research_report.md (as prose) before export. Do not delegate the triage — the judgment of what is report-worthy is yours. If you reject an item, append a one-line DROPPED, why: note to research_notes.md so the rejection is auditable.

8. Read the draft back, apply the numeric-verify and completeness-critic fixes, then write metadata.json yourself (structured data, not prose — it needs your novelty analysis + config diff), then validate + PDF + email (see Export & Delivery).

Notes:

• For quick/standard modes you may skip the fan-out and fetch serially yourself (fewer sources, faster turnaround) — but still persist primary sources to sources/.
• If the fast model is down, hf-retriever auto-falls back to the strong model (fallbackModels), so the run still completes — just slower/costlier.
• Retrievers are leaf agents (they will not spawn subagents).
• For quick mode only, you may skip the prose delegation and write research_report.md yourself — but apply prose_style_prompt.md by hand and still run sg -j on the draft before exporting.

3. Acronym & Concept Explanation Standards

Assume readers are unfamiliar with ML concepts. On first use, spell out the full term with acronym in parentheses, then a one-sentence explanation:

{Full Name} ({ACRONYM}) — {one-sentence explanation}. {Optional: origin/citation}

First Mention	Explanation
Mixture of Experts (MoE)	Multiple specialized sub-networks ("experts") process different inputs, with a learned router selecting which experts activate. Jacobs et al. (1991).
Multi-head Latent Attention (MLA)	Compresses key-value pairs into lower-dimensional latent vectors, reducing KV-cache memory by ~93%. DeepSeek-V2 (arXiv:2405.04434).
Rotary Position Embedding (RoPE)	Represents positions as rotations in embedding space, enabling better length generalization. Su et al. (arXiv:2104.09864).
Grouped Query Attention (GQA)	Multiple query heads share a single key-value head, reducing memory bandwidth. Ainslie et al. (arXiv:2305.13245).
YaRN	Extends context length beyond training length by modifying RoPE frequencies. Peng et al. (arXiv:2309.00071).
Reinforcement Learning from Human Feedback (RLHF)	Fine-tuning with human preference data to align with human values. Ouyang et al. (arXiv:2203.02155).
Direct Preference Optimization (DPO)	Simpler alternative to RLHF that directly optimizes on preference data without a separate reward model. Rafailov et al. (arXiv:2305.18290).

After first introduction, use only the acronym. Only introduce an acronym if it will be used 3+ times. Every acronym used 3+ times in the report must appear in the Glossary section — validate_model_report.py warns on any all-caps acronym used 3+ times that the glossary omits (the glossary-acronyms gate). Mixed-case acronyms (RoPE, OpenMDW, LatentMoE) are outside that gate's scope, but spell them out in the glossary all the same.

4. ArXiv Paper Reading

Fetch tarball-first; fall back to /html/; never /abs/ or /pdf/. For any Tier-1 paper, the /e-print/ source tarball is the richest form — it unpacks into per-section LaTeX you can shard and grep. The Nemotron-3-Ultra runs proved this: the improvised tarball recovery (sources/<id>_src/sections/*.tex) is where the real depth came from, so it is now the documented default.

# BEST — source tarball, unpacks to per-section LaTeX
mkdir -p ./{org}/{model}/sources
curl -sL -o ./{org}/{model}/sources/{id}.tar.gz "https://arxiv.org/e-print/{id}"
tar -xzf ./{org}/{model}/sources/{id}.tar.gz -C ./{org}/{model}/sources/{id}_src
# then enumerate sections: ls ./{org}/{model}/sources/{id}_src/sections/*.tex

# FALLBACK — full HTML, structured, parseable (only if the tarball fails)
python3 ~/.pi/agent/skills/deep-research/scripts/web_fetch.py "https://arxiv.org/html/{id}" "./{org}/{model}/sources/paper_{id}.txt"
# BAD  https://arxiv.org/abs/{id}   (abstract only)
# BAD  https://arxiv.org/pdf/{id}   (PDF, hard to parse)

After extracting the tarball, enumerate sources/<id>_src/sections/*.tex and either assign every major section to a tech-report-<section> shard or write a SECTIONS NOT SHARDED: line into research_notes.md (§2.1, item: section-coverage completeness). When reading papers: extract Abstract, Introduction, and relevant Method sections; find the section describing the novel technique; cite section numbers (e.g. "Section 3.2 describes…").

5. Novelty Analysis (CRITICAL)

The most valuable part is identifying and EXPLAINING what's genuinely NEW in THIS release.

Definition of "Novel" — Release-Specific Only

"Novel" = introduced in THIS specific model release, NOT inherited from prior versions. A technique is NOT novel if it was introduced in a prior version and carried forward, or exists in the predecessor's config.json.

Example — MiniMax family: M2 introduced 256 experts / sigmoid scoring / MTP → novel for M2. M2.1 has identical config → those are NOT novel for M2.1; M2.1's novel items are the VIBE benchmark, multilingual expansion, scaffold improvements.

Example — DeepSeek family: V3 introduced MLA / aux-loss-free load balancing → novel for V3. V3.2 uses the same MLA → MLA is NOT novel for V3.2; V3.2's novel item is DSA.

The "What's New" section should ONLY contain techniques introduced in THIS release. Inherited architecture features belong in "Architecture Deep Dive" with attribution to where they were first introduced.

For EACH Novel Technique, Cover (as flowing prose — NOT labeled bullets)

Write 2–4 paragraphs per technique with strong topic sentences that cover all five aspects (see "Prose Style" above). Do NOT emit a Problem / Mechanism / Prior Art / Tradeoffs / Verification labeled schema:

• Problem — what limitation does this address?
• Mechanism — how does it actually work? (concrete steps, concrete numbers)
• Prior Art — what existing techniques does it build on? (cite original papers)
• Tradeoffs — costs/limitations?
• Verification — is this claim verified in primary sources? (state once, not per section)

Novelty Verification Checklist (for EACH claimed innovation)

• New in THIS SPECIFIC release? (not inherited from predecessor)
• Does the predecessor config.json already have this? (if yes, NOT novel)
• When was this technique first introduced in this model family? (cite that version)
• Implementation novel, or standard ("we use GQA" isn't novel in 2025)?
• Verifiable in the official model card or technical report?
• Are benchmark improvements statistically significant and reproducible?
• What prior work does it build on? (cite original papers)

Critical: if a technique exists in the predecessor, it belongs in "Architecture Deep Dive" (with attribution), NOT "What's New". Compare configs directly:

diff "./{org}/{model}/predecessor_config.json" "./{org}/{model}/config.json" || true

6. Source Hierarchy & Verification

Prioritize primary sources over secondary coverage.

1. Tier 1 — Official/Primary (REQUIRED): arXiv technical reports by model authors (HIGHEST — /html/ endpoint), official technical reports/papers, HF model cards/READMEs, config.json, official GitHub repos, official company blogs/announcements. Technical reports are the GOLD STANDARD (training methodology, design rationale, benchmark methodology, ablations, limitations).
2. Tier 2 — Framework docs: vLLM, SGLang, Transformers, NVIDIA NIM docs.
3. Tier 3 — Authoritative benchmarks: official leaderboards (swebench.com, lmsys.org), HF Open LLM Leaderboard, Artificial Analysis.
4. Tier 4 — Secondary coverage (context only): tech news (use only to find primary sources).

Search strategy:

python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "{model_name} huggingface" 10
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "{predecessor_model}" 10
python3 ~/.pi/agent/skills/deep-research/scripts/arxiv_search.py "{model_family} technical report" 5
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "{model_name} vLLM" 5 --site docs.vllm.ai
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "{model_name}" 5 --site swebench.com

DON'T rely primarily on aggregators (marktechpost etc.) or social summaries.

Source tracing (during synthesis): trace every major claim to a primary source as you write research_notes.md, and flag claims you cannot source. Verifying the draft itself (numbers, bibliography IDs, glossary, what is missing) is NOT an orchestrator inline phase — it is delegated to the fresh-context reviewer passes in §2.1 (steps 6–7) and validate_model_report.py at export (§7).

7. Verification is delegated (do not inline it)

The orchestrator does not run a separate inline self-verification pass over the draft — that duplicates the reviewer and bloats context. Verification is owned by two fresh-context reviewer subagents in §2.1 (step 6 numeric-verify, step 7 completeness-critic) and then by validate_model_report.py at export. Your job at this stage is to triage and fold in their findings, not to re-verify from memory. Specifically: do not re-check citations, acronyms, or novelty claims yourself in-context — the numeric-verify reviewer covers the bibliography and glossary, the completeness-critic covers what is missing, and the validator's citation-identity, citation-integrity, notes-coverage, section-coverage, glossary-acronyms, and table-shape gates enforce the rest. If a gate fails and you have a written justification, record it in validation_waivers.md as a gate: <gate-name> block followed by a justification: <text...> line (the loader also accepts a ## <gate-name> heading as the gate line); never edit a citation store to satisfy the validator.

---

Output Format

Markdown Formatting for PDF Compatibility

Always use two newlines before lists, code blocks, and block elements (ensures proper PDF rendering):

<!-- BAD — renders as single line in PDF -->
Some text:
- Item 1

<!-- GOOD -->
Some text:

- Item 1

Content Requirements

• Store outputs in the current working directory (pwd).
• Create subfolder structure: ./{org}/{model_name}/
• Include research_report.md with a Glossary section, a "What's New" section with deep explanations, and technical deep dives for each novel technique.
• Include metadata.json with a novelty field.
• Include a citation bibliography with source-tier annotations and arXiv links in /html/ format.

Report Structure

# {Model Name}: Deep Research Report

## Executive Summary
[Key findings in 2-3 paragraphs]

## Glossary of Terms
[Define ALL acronyms and technical concepts used]

| Term | Definition | Original Paper |
|------|------------|----------------|
| MoE | Mixture of Experts - ... | Jacobs et al. (1991) |
| DSA | DeepSeek Sparse Attention - ... | arXiv:2512.02556 |

## What's New in This Release
[Deep explanations of each novel technique]
### {Novel Technique 1}
[2–4 flowing paragraphs covering problem / mechanism / prior art / tradeoffs /
verification — NOT labeled `**Problem:** …` bullets. See "Prose Style".]

## Architecture Deep Dive
[Technical details with explanations; inherited features attributed to original release]

## Benchmark Analysis
[Performance comparisons with context; bold the winner in each benchmark row]

## Limitations & Caveats
[Honest assessment of weaknesses]

## Bibliography
[Full citations with arXiv /html/ links]

Metadata JSON Structure

{
  "model_id": "org/model-name",
  "organization": "...",
  "model_family": "...",
  "predecessor": "org/previous-model",
  "license": "...",
  "modality": "...",
  "architecture": { "type": "...", "num_hidden_layers": 0, "...": "..." },
  "glossary": {
    "DSA": { "full_name": "...", "definition": "...", "original_paper": "arXiv:...", "introduced_in": "..." }
  },
  "novelty": {
    "vs_predecessor": {
      "model": "org/previous-model",
      "config_diff": { "...": "a -> b" },
      "architectural_changes": ["..."],
      "training_changes": ["..."],
      "benchmark_improvements": ["..."],
      "new_capabilities": ["..."]
    },
    "genuinely_novel_this_release": [
      "DSA (DeepSeek Sparse Attention) - first introduced here"
    ],
    "inherited_from_predecessor": [
      {"technique": "MLA", "introduced_in": "DeepSeek-V3", "paper": "arXiv:2412.19437"}
    ],
    "builds_on_prior_work": { "technique": "DSA", "prior_work": ["Longformer","BigBird"], "citations": ["arXiv:..."] }
  },
  "verification_notes": {
    "verified_via_config_diff": ["..."],
    "vendor_claims_not_independently_reproduced": ["..."],
    "estimates": ["..."],
    "flagged_unverified": ["..."]
  },
  "sources": { "tier_1": ["https://..."] }
}

---

Example Search Queries

For a model like deepseek-ai/DeepSeek-V3.2:

# Primary sources (ALWAYS first)
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "DeepSeek-V3.2 huggingface" 10
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "DeepSeek-V3.1-Terminus" 10
python3 ~/.pi/agent/skills/deep-research/scripts/arxiv_search.py "DeepSeek V3.2 sparse attention" 5
# then fetch: python3 .../web_fetch.py "https://arxiv.org/html/2512.02556" "./deepseek-ai/DeepSeek-V3.2/paper_2512.02556.txt"

# Prior work for novel techniques
python3 ~/.pi/agent/skills/deep-research/scripts/arxiv_search.py "sparse attention transformer Longformer BigBird" 5
python3 ~/.pi/agent/skills/deep-research/scripts/arxiv_search.py "Flash Attention" 3

# Framework docs + benchmarks
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "DeepSeek V3.2 vLLM" 5 --site docs.vllm.ai
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "DeepSeek V3.2" 5 --site swebench.com
python3 ~/.pi/agent/skills/deep-research/scripts/web_search.py "DeepSeek V3.2" 5 --site artificialanalysis.ai

---

Export & Delivery

After generating the research report, export to PDF and send via email.

1. File Naming

./{org}/{model_name}/
├── research_report.md
├── {Model-Name}_Deep_Research_Report.pdf
└── metadata.json

2. Validate the deliverable

python3 ~/.pi/agent/skills/hf-model-researcher/scripts/validate_model_report.py "./{org}/{model_name}"

3. Convert to PDF

python3 ~/.pi/agent/skills/hf-model-researcher/scripts/md_to_pdf.py \
  "./{org}/{model_name}/research_report.md" \
  "./{org}/{model_name}/{Model-Name}_Deep_Research_Report.pdf" \
  "{Model Name}: {Key Innovations Summary}"

4. Send via Email

python3 ~/.pi/agent/skills/hf-model-researcher/scripts/send_email.py \
  "./{org}/{model_name}/{Model-Name}_Deep_Research_Report.pdf" \
  [e-mail] \
  "{Model Name} Research: {Key Innovations Summary}"

.env (with EMAIL_ADDRESS / EMAIL_APP_PASSWORD) lives next to send_email.py in the skill's scripts/ dir. Use a Gmail App Password (not your regular password): https://myaccount.google.com/apppasswords

5. Remind About Annotator

Do NOT send reports to Kindle. Remind the user to use the annotator app for reading/annotating the PDF report.

Output Files

After completing the workflow:

./{org}/{model_name}/
├── research_report.md
├── {Model-Name}_Deep_Research_Report.pdf
└── metadata.json

(plus any saved paper_*.txt and research_state/ artifacts.)

---

Quality Checklist

Context Hygiene (check first):

• Context usage <60K tokens (the prior 25K target was unrealistic; two Nemotron-3-Ultra runs peaked at 181K/218K)
• grep -n '<pattern>' -A20 -B20 sources/paper_*.txt windows used to verify facts — full extracted papers NOT read end-to-end
• research_state.py used to save findings after each fetch
• citation_store.py used to add citations as discovered (not batched at end)
• No memory-sourced citations in notes/glossary/bibliography — no arXiv ID, paper title, author name, license-acronym expansion, or benchmark authorship without a fetched source in sources/ or citations_*.json (every attribution traces to a fetch record)

Content Quality:

• Technical report found and cited (arXiv or official paper)
• Predecessor technical reports also fetched (for evolution context)
• prior-work retriever cluster resolved every prior-art paper the report names (true ID + title + authors in the citation store)
• Every major tarball section assigned to a tech-report-<section> shard OR listed in a SECTIONS NOT SHARDED: line in research_notes.md (no orphaned section files)
• Every acronym defined on first use with explanation; every acronym used 3+ times is in the Glossary (glossary-acronyms gate)
• Every novel technique covers problem/mechanism/prior art/tradeoffs/verification in flowing prose (not a labeled-bullet template)
• Prose rules in prose_style_prompt.md followed (no labeled template; verification said once; sentence/em-dash/parenthetical discipline; vivid specifics in prose; no verdict-adjective topic sentences; each "What's New" subsection opens with why the naive approach fails)
• Draft delegated to prose subagent (orchestrator wrote research_notes.md, not the report)
• Slop Guard (sg -j) run on the draft by prose; real issues fixed once
• arXiv papers fetched tarball-first (/e-print/ source → extract sections; fall back to /html/ + 60KB-capped web_fetch)
• Prior work cited for each innovation
• notes_number_diff.py run after writing notes; every dropped number folded in or accepted via a DROPPED, why: line
• Numeric-verify reviewer pass (step 6) checked every number AND the bibliography + glossary citations
• Completeness-critic reviewer pass (step 7) ran; accepted missing findings folded into notes + report before export
• All claims traced to Tier 1 sources; unverified claims flagged
• Predecessor config.json compared
• Benchmarks include specific numbers, not vague claims
• Report readable by someone unfamiliar with ML

Novelty Accuracy:

• "What's New" only contains THIS release's innovations (not inherited)
• Inherited architecture features attributed to original release
• Predecessor config.json diffed to verify what actually changed
• No false novelty claims

Deliverable:

• validate_model_report.py passes with zero errors and zero unwaived warnings (any failing gate has a written validation_waivers.md justification — never a citation-store edit)
• research_report.md, metadata.json, and PDF all present in ./{org}/{model}/
• PDF generated; email sent (if requested); user reminded about the annotator app

---

Tool Efficiency

• /skill:fetch-model {model_id} [predecessor] — parallel curl for README, config, predecessor config
• /skill:fetch-arxiv {id_or_query} — safe arXiv fetching (search first, then /html/)
• web_search.py / arxiv_search.py / web_fetch.py — pi-native retrieval helpers

Token targets: initial fetch <5K; research phase <25K; total context <60K (the prior 25K total was unrealistic — two Nemotron-3-Ultra runs peaked at 181K/218K; grep -n windows on sources/ are what keep it bounded).