research-paper-writing

# Research Paper Writing Pipeline End-to-end pipeline for producing publication-ready ML/AI research papers targeting **NeurIPS, ICML, ICLR, ACL, AAAI, and COLM**. This skill covers the full research lifecycle: experiment design, execution, monitoring, analysis, paper writing, review, revision, and submission. This is **not a linear pipeline** — it is an iterative loop. Results trigger new experiments. Reviews trigger new analysis. The agent must handle these feedback loops. <!-- ascii-guard-ignore --> ``` ┌─────────────────────────────────────────────────────────────┐ │ RESEARCH PAPER PIPELINE │ │ │ │ Phase 0: Project Setup ──► Phase 1: Literature Review │ │ │ │ │ │ ▼ ▼ │ │ Phase 2: Experiment Phase 5: Paper Drafting ◄──┐ │ │ Design │ │ │ │ │ ▼ │ │ │ ▼ Phase 6: Self-Review │ │ │ Phase 3: Execution & & Revision ──────────┘ │ │ Monitoring │ │ │ │ ▼ │ │ ▼ Phase 7: Submission │ │ Phase 4: Analysis ─────► (feeds back to Phase 2 or 5) │ │ │ └─────────────────────────────────────────────────────────────┘ ``` <!-- ascii-guard-ignore-end --> --- ## When To Use This Skill Use this skill when: - **Starting a new research paper** from an existing codebase or idea - **Designing and running experiments** to support paper claims - **Writing or revising** any section of a research paper - **Preparing for submission** to a specific conference or workshop - **Responding to reviews** with additional experiments or revisions - **Converting** a paper between conference formats - **Writing non-empirical papers** — theory, survey, benchmark, or position papers (see [Paper Types Beyond Empirical ML](#paper-types-beyond-empirical-ml)) - **Designing human evaluations** for NLP, HCI, or alignment research - **Preparing post-acceptance deliverables** — posters, talks, code releases ## Core Philosophy 1. **Be proactive.** Deliver complete drafts, not questions. Scientists are busy — produce something concrete they can react to, then iterate. 2. **Never hallucinate citations.** AI-generated citations have ~40% error rate. Always fetch programmatically. Mark unverifiable citations as `[CITATION NEEDED]`. 3. **Paper is a story, not a collection of experiments.** Every paper needs one clear contribution stated in a single sentence. If you can't do that, the paper isn't ready. 4. **Experiments serve claims.** Every experiment must explicitly state which claim it supports. Never run experiments that don't connect to the paper's narrative. 5. **Commit early, commit often.** Every completed experiment batch, every paper draft update — commit with descriptive messages. Git log is the experiment history. ### Proactivity and Collaboration **Default: Be proactive. Draft first, ask with the draft.** | Confidence Level | Action | |-----------------|--------| | **High** (clear repo, obvious contribution) | Write full draft, deliver, iterate on feedback | | **Medium** (some ambiguity) | Write draft with flagged uncertainties, continue | | **Low** (major unknowns) | Ask 1-2 targeted questions via `clarify`, then draft | | Section | Draft Autonomously? | Flag With Draft | |---------|-------------------|-----------------| | Abstract | Yes | "Framed contribution as X — adjust if needed" | | Introduction | Yes | "Emphasized problem Y — correct if wrong" | | Methods | Yes | "Included details A, B, C — add missing pieces" | | Experiments | Yes | "Highlighted results 1, 2, 3 — reorder if needed" | | Related Work | Yes | "Cited papers X, Y, Z — add any I missed" | **Block for input only when**: target venue unclear, multiple contradictory framings, results seem incomplete, explicit request to review first. --- ## Phase 0: Project Setup **Goal**: Establish the workspace, understand existing work, identify the contribution. ### Step 0.1: Explore the Repository ```bash # Understand project structure ls -la find . -name "*.py" | head -30 find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "result\|conclusion\|finding" ``` Look for: - `README.md` — project overview and claims - `results/`, `outputs/`, `experiments/` — existing findings - `configs/` — experimental settings - `.bib` files — existing citations - Draft documents or notes ### Step 0.2: Organize the Workspace Establish a consistent workspace structure: ``` workspace/ paper/ # LaTeX source, figures, compiled PDFs experiments/ # Experiment runner scripts code/ # Core method implementation results/ # Raw experiment results (auto-generated) tasks/ # Task/benchmark definitions human_eval/ # Human evaluation materials (if needed) ``` ### Step 0.3: Set Up Version Control ```bash git init # if not already git remote add origin <repo-url> git checkout -b paper-draft # or main ``` **Git discipline**: Every completed experiment batch gets committed with a descriptive message. Example: ``` Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task) Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier ``` ### Step 0.4: Identify the Contribution Before writing anything, articulate: - **The What**: What is the single thing this paper contributes? - **The Why**: What evidence supports it? - **The So What**: Why should readers care? > Propose to the scientist: "Based on my understanding, the main contribution is: [one sentence]. The key results show [Y]. Is this the framing you want?" ### Step 0.5: Create a TODO List Use the `todo` tool to create a structured project plan: ``` Research Paper TODO: - [ ] Define one-sentence contribution - [ ] Literature review (related work + baselines) - [ ] Design core experiments - [ ] Run experiments - [ ] Analyze results - [ ] Write first draft - [ ] Self-review (simulate reviewers) - [ ] Revise based on review - [ ] Submission prep ``` Update this throughout the project. It serves as the persistent state across sessions. ### Step 0.6: Estimate Compute Budget Before running experiments, estimate total cost and time: ``` Compute Budget Checklist: - [ ] API costs: (model price per token) × (estimated tokens per run) × (number of runs) - [ ] GPU hours: (time per experiment) × (number of experiments) × (number of seeds) - [ ] Human evaluation costs: (annotators) × (hours) × (hourly rate) - [ ] Total budget ceiling and contingency (add 30-50% for reruns) ``` Track actual spend as experiments run: ```python # Simple cost tracker pattern import json, os from datetime import datetime COST_LOG = "results/cost_log.jsonl" def log_cost(experiment: str, model: str, input_tokens: int, output_tokens: int, cost_usd: float): entry = { "timestamp": datetime.now().isoformat(), "experiment": experiment, "model": model, "input_tokens": input_tokens, "output_tokens": output_tokens, "cost_usd": cost_usd, } with open(COST_LOG, "a") as f: f.write(json.dumps(entry) + "\n") ``` **When budget is tight**: Run pilot experiments (1-2 seeds, subset of tasks) before committing to full sweeps. Use cheaper models for debugging pipelines, then switch to target models for final runs. ### Step 0.7: Multi-Author Coordination Most papers have 3-10 authors. Establish workflows early: | Workflow | Tool | When to Use | |----------|------|-------------| | **Overleaf** | Browser-based | Multiple authors editing simultaneously, no git experience | | **Git + LaTeX** | `git` with `.gitignore` for aux files | Technical teams, need branch-based review | | **Overleaf + Git sync** | Overleaf premium | Best of both — live collab with version history | **Section ownership**: Assign each section to one primary author. Others comment but don't edit directly. Prevents merge conflicts and style inconsistency. ``` Author Coordination Checklist: - [ ] Agree on section ownership (who writes what) - [ ] Set up shared workspace (Overleaf or git repo) - [ ] Establish notation conventions (before anyone writes) - [ ] Schedule internal review rounds (not just at the end) - [ ] Designate one person for final formatting pass - [ ] Agree on figure style (colors, fonts, sizes) before creating figures ``` **LaTeX conventions to agree on early**: - `\method{}` macro for consistent method naming - Citation style: `\citet{}` vs `\citep{}` usage - Math notation: lowercase bold for vectors, uppercase bold for matrices, etc. - British vs American spelling --- ## Phase 1: Literature Review **Goal**: Find related work, identify baselines, gather citations. ### Step 1.1: Identify Seed Papers Start from papers already referenced in the codebase: ```bash # Via terminal: grep -r "arxiv\|doi\|cite" --include="*.md" --include="*.bib" --include="*.py" find . -name "*.bib" ``` ### Step 1.2: Search for Related Work **Load the `arxiv` skill** for structured paper discovery: `skill_view("arxiv")`. It provides arXiv REST API search, Semantic Scholar citation graphs, author profiles, and BibTeX generation. Use `web_search` for broad discovery, `web_extract` for fetching specific papers: ``` # Via web_search: web_search("[main technique] + [application domain] site:arxiv.org") web_search("[baseline method] comparison ICML NeurIPS 2024") # Via web_extract (for specific papers): web_extract("https://arxiv.org/abs/2303.17651") ``` Additional search queries to try: ``` Search queries: - "[main technique] + [application domain]" - "[baseline method] comparison" - "[problem name] state-of-the-art" - Author names from existing citations ``` **Recommended**: Install **Exa MCP** for real-time academic search: ```bash claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp" ``` ### Step 1.2b: Deepen the Search (Breadth-First, Then Depth) A flat search (one round of queries) typically misses important related work. Use an iterative **breadth-then-depth** pattern inspired by deep research pipelines: ``` Iterative Literature Search: Round 1 (Breadth): 4-6 parallel queries covering different angles - "[method] + [domain]" - "[problem name] state-of-the-art 2024 2025" - "[baseline method] comparison" - "[alternative approach] vs [your approach]" → Collect papers, extract key concepts and terminology Round 2 (Depth): Generate follow-up queries from Round 1 learnings - New terminology discovered in Round 1 papers - Papers cited by the most relevant Round 1 results - Contradictory findings that need investigation → Collect papers, identify remaining gaps Round 3 (Targeted): Fill specific gaps - Missing baselines identified in Rounds 1-2 - Concurrent work (last 6 months, same problem) - Key negative results or failed approaches → Stop when new queries return mostly papers you've already seen ``` **When to stop**: If a round returns >80% papers already in your collection, the search is saturated. Typically 2-3 rounds suffice. For survey papers, expect 4-5 rounds. **For agent-based workflows**: Delegate each round's queries in parallel via `delegate_task`. Collect results, deduplicate, then generate the next round's queries from the combined learnings. ### Step 1.3: Verify Every Citation **NEVER generate BibTeX from memory. ALWAYS fetch programmatically.** For each citation, follow the mandatory 5-step process: ``` Citation Verification (MANDATORY per citation): 1. SEARCH → Query Semantic Scholar or Exa MCP with specific keywords 2. VERIFY → Confirm paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef) 3. RETRIEVE → Get BibTeX via DOI content negotiation (programmatically, not from memory) 4. VALIDATE → Confirm the claim you're citing actually appears in the paper 5. ADD → Add verified BibTeX to bibliography If ANY step fails → mark as [CITATION NEEDED], inform scientist ``` ```python # Fetch BibTeX via DOI import requests def doi_to_bibtex(doi: str) -> str: response = requests.get( f"https://doi.org/{doi}", headers={"Accept": "application/x-bibtex"} ) response.raise_for_status() return response.text ``` If you cannot verify a citation: ```latex \cite{PLACEHOLDER_author2024_verify_this} % TODO: Verify this citation exists ``` **Always tell the scientist**: "I've marked [X] citations as placeholders that need verification." See [references/citation-workflow.md](references/citation-workflow.md) for complete API documentation and the full `CitationManager` class. ### Step 1.4: Organize Related Work Group papers by methodology, not paper-by-paper: **Good**: "One line of work uses X's assumption [refs] whereas we use Y's assumption because..." **Bad**: "Smith et al. introduced X. Jones et al. introduced Y. We combine both." --- ## Phase 2: Experiment Design **Goal**: Design experiments that directly support paper claims. Every experiment must answer a specific question. ### Step 2.1: Map Claims to Experiments Create an explicit mapping: | Claim | Experiment | Expected Evidence | |-------|-----------|-------------------| | "Our method outperforms baselines" | Main comparison (Table 1) | Win rate, statistical significance | | "Effect is larger for weaker models" | Model scaling study | Monotonic improvement curve | | "Convergence requires scope constraints" | Constrained vs unconstrained | Convergence rate comparison | **Rule**: If an experiment doesn't map to a claim, don't run it. ### Step 2.2: Design Baselines Strong baselines are what separates accepted papers from rejected ones. Reviewers will ask: "Did they compare against X?" Standard baseline categories: - **Naive baseline**: Simplest possible approach - **Strong baseline**: Best known existing method - **Ablation baselines**: Your method minus one component - **Compute-matched baselines**: Same compute budget, different allocation ### Step 2.3: Define Evaluation Protocol Before running anything, specify: - **Metrics**: What you're measuring, direction symbols (higher/lower better) - **Aggregation**: How results are combined across runs/tasks - **Statistical tests**: What tests will establish significance - **Sample sizes**: How many runs/problems/tasks ### Step 2.4: Write Experiment Scripts Follow these patterns from successful research pipelines: **Incremental saving** — save results after each step for crash recovery: ```python # Save after each problem/task result_path = f"results/{task}/{strategy}/result.json" if os.path.exists(result_path): continue # Skip already-completed work # ... run experiment ... with open(result_path, 'w') as f: