{"community":{"tags":["Research","Paper Writing","Experiments","ML","AI","NeurIPS","ICML","ICLR","ACL","AAAI","COLM","LaTeX","Citations","Statistical Analysis"],"category":"research","related_skills":["arxiv","ml-paper-writing","subagent-driven-development","plan"],"requires_toolsets":["terminal","files"]},"upstream_import":{"original_name":"research-paper-writing","source":"community catalog active profile ~/.argent/skills"},"argent":{"os":["linux","darwin"]}}
Research Paper Writing Pipeline
End-to-end pipeline for producing publication-ready ML/AI research papers targeting NeurIPS, ICML, ICLR, ACL, AAAI, and COLM. This skill covers the full research lifecycle: experiment design, execution, monitoring, analysis, paper writing, review, revision, and submission.
This is not a linear pipeline — it is an iterative loop. Results trigger new experiments. Reviews trigger new analysis. The agent must handle these feedback loops.
Be proactive. Deliver complete drafts, not questions. Scientists are busy — produce something concrete they can react to, then iterate.
AI-generated citations have ~40% error rate. Always fetch programmatically. Mark unverifiable citations as .
이 SKILL.md는 매우 커서 SkillsMP가 여기에는 첫 섹션만 미리 보여줍니다.GitHub에서 보기
Never hallucinate citations.
[CITATION NEEDED]
Paper is a story, not a collection of experiments. Every paper needs one clear contribution stated in a single sentence. If you can't do that, the paper isn't ready.
Experiments serve claims. Every experiment must explicitly state which claim it supports. Never run experiments that don't connect to the paper's narrative.
Commit early, commit often. Every completed experiment batch, every paper draft update — commit with descriptive messages. Git log is the experiment history.
Proactivity and Collaboration
Default: Be proactive. Draft first, ask with the draft.
Confidence Level
Action
High (clear repo, obvious contribution)
Write full draft, deliver, iterate on feedback
Medium (some ambiguity)
Write draft with flagged uncertainties, continue
Low (major unknowns)
Ask 1-2 targeted questions via clarify, then draft
Section
Draft Autonomously?
Flag With Draft
Abstract
Yes
"Framed contribution as X — adjust if needed"
Introduction
Yes
"Emphasized problem Y — correct if wrong"
Methods
Yes
"Included details A, B, C — add missing pieces"
Experiments
Yes
"Highlighted results 1, 2, 3 — reorder if needed"
Related Work
Yes
"Cited papers X, Y, Z — add any I missed"
Block for input only when: target venue unclear, multiple contradictory framings, results seem incomplete, explicit request to review first.
Phase 0: Project Setup
Goal: Establish the workspace, understand existing work, identify the contribution.
git init # if not already
git remote add origin <repo-url>
git checkout -b paper-draft # or main
Git discipline: Every completed experiment batch gets committed with a descriptive message. Example:
Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task)
Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier
Step 0.4: Identify the Contribution
Before writing anything, articulate:
The What: What is the single thing this paper contributes?
The Why: What evidence supports it?
The So What: Why should readers care?
Propose to the scientist: "Based on my understanding, the main contribution is: [one sentence]. The key results show [Y]. Is this the framing you want?"
Step 0.5: Create a TODO List
Use the todo tool to create a structured project plan:
Research Paper TODO:
- [ ] Define one-sentence contribution
- [ ] Literature review (related work + baselines)
- [ ] Design core experiments
- [ ] Run experiments
- [ ] Analyze results
- [ ] Write first draft
- [ ] Self-review (simulate reviewers)
- [ ] Revise based on review
- [ ] Submission prep
Update this throughout the project. It serves as the persistent state across sessions.
Step 0.6: Estimate Compute Budget
Before running experiments, estimate total cost and time:
Compute Budget Checklist:
- [ ] API costs: (model price per token) × (estimated tokens per run) × (number of runs)
- [ ] GPU hours: (time per experiment) × (number of experiments) × (number of seeds)
- [ ] Human evaluation costs: (annotators) × (hours) × (hourly rate)
- [ ] Total budget ceiling and contingency (add 30-50% for reruns)
When budget is tight: Run pilot experiments (1-2 seeds, subset of tasks) before committing to full sweeps. Use cheaper models for debugging pipelines, then switch to target models for final runs.
Step 0.7: Multi-Author Coordination
Most papers have 3-10 authors. Establish workflows early:
Workflow
Tool
When to Use
Overleaf
Browser-based
Multiple authors editing simultaneously, no git experience
Git + LaTeX
git with .gitignore for aux files
Technical teams, need branch-based review
Overleaf + Git sync
Overleaf premium
Best of both — live collab with version history
Section ownership: Assign each section to one primary author. Others comment but don't edit directly. Prevents merge conflicts and style inconsistency.
Author Coordination Checklist:
- [ ] Agree on section ownership (who writes what)
- [ ] Set up shared workspace (Overleaf or git repo)
- [ ] Establish notation conventions (before anyone writes)
- [ ] Schedule internal review rounds (not just at the end)
- [ ] Designate one person for final formatting pass
- [ ] Agree on figure style (colors, fonts, sizes) before creating figures
LaTeX conventions to agree on early:
\method{} macro for consistent method naming
Citation style: \citet{} vs \citep{} usage
Math notation: lowercase bold for vectors, uppercase bold for matrices, etc.
British vs American spelling
Phase 1: Literature Review
Goal: Find related work, identify baselines, gather citations.
Step 1.1: Identify Seed Papers
Start from papers already referenced in the codebase:
Load the arxiv skill for structured paper discovery: skill_view("arxiv"). It provides arXiv REST API search, Semantic Scholar citation graphs, author profiles, and BibTeX generation.
Use web_search for broad discovery, web_extract for fetching specific papers:
# Via web_search:
web_search("[main technique] + [application domain] site:arxiv.org")
web_search("[baseline method] comparison ICML NeurIPS 2024")
# Via web_extract (for specific papers):
web_extract("https://arxiv.org/abs/2303.17651")
Recommended: Install Exa MCP for real-time academic search:
claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp"
Step 1.2b: Deepen the Search (Breadth-First, Then Depth)
A flat search (one round of queries) typically misses important related work. Use an iterative breadth-then-depth pattern inspired by deep research pipelines:
Iterative Literature Search:
Round 1 (Breadth): 4-6 parallel queries covering different angles
- "[method] + [domain]"
- "[problem name] state-of-the-art 2024 2025"
- "[baseline method] comparison"
- "[alternative approach] vs [your approach]"
→ Collect papers, extract key concepts and terminology
Round 2 (Depth): Generate follow-up queries from Round 1 learnings
- New terminology discovered in Round 1 papers
- Papers cited by the most relevant Round 1 results
- Contradictory findings that need investigation
→ Collect papers, identify remaining gaps
Round 3 (Targeted): Fill specific gaps
- Missing baselines identified in Rounds 1-2
- Concurrent work (last 6 months, same problem)
- Key negative results or failed approaches
→ Stop when new queries return mostly papers you've already seen
When to stop: If a round returns >80% papers already in your collection, the search is saturated. Typically 2-3 rounds suffice. For survey papers, expect 4-5 rounds.
For agent-based workflows: Delegate each round's queries in parallel via delegate_task. Collect results, deduplicate, then generate the next round's queries from the combined learnings.
Step 1.3: Verify Every Citation
NEVER generate BibTeX from memory. ALWAYS fetch programmatically.
For each citation, follow the mandatory 5-step process:
Citation Verification (MANDATORY per citation):
1. SEARCH → Query Semantic Scholar or Exa MCP with specific keywords
2. VERIFY → Confirm paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef)
3. RETRIEVE → Get BibTeX via DOI content negotiation (programmatically, not from memory)
4. VALIDATE → Confirm the claim you're citing actually appears in the paper
5. ADD → Add verified BibTeX to bibliography
If ANY step fails → mark as [CITATION NEEDED], inform scientist
Good: "One line of work uses X's assumption [refs] whereas we use Y's assumption because..."
Bad: "Smith et al. introduced X. Jones et al. introduced Y. We combine both."
Phase 2: Experiment Design
Goal: Design experiments that directly support paper claims. Every experiment must answer a specific question.
Step 2.1: Map Claims to Experiments
Create an explicit mapping:
Claim
Experiment
Expected Evidence
"Our method outperforms baselines"
Main comparison (Table 1)
Win rate, statistical significance
"Effect is larger for weaker models"
Model scaling study
Monotonic improvement curve
"Convergence requires scope constraints"
Constrained vs unconstrained
Convergence rate comparison
Rule: If an experiment doesn't map to a claim, don't run it.
Step 2.2: Design Baselines
Strong baselines are what separates accepted papers from rejected ones. Reviewers will ask: "Did they compare against X?"
Standard baseline categories:
Naive baseline: Simplest possible approach
Strong baseline: Best known existing method
Ablation baselines: Your method minus one component
Compute-matched baselines: Same compute budget, different allocation
Step 2.3: Define Evaluation Protocol
Before running anything, specify:
Metrics: What you're measuring, direction symbols (higher/lower better)
Aggregation: How results are combined across runs/tasks
Statistical tests: What tests will establish significance
Sample sizes: How many runs/problems/tasks
Step 2.4: Write Experiment Scripts
Follow these patterns from successful research pipelines:
Incremental saving — save results after each step for crash recovery:
# Save after each problem/task
result_path = f"results/{task}/{strategy}/result.json"
if os.path.exists(result_path):
continue # Skip already-completed work
# ... run experiment ...
with open(result_path, 'w') as f:
json.dump(result, f, indent=2)
Artifact preservation — save all intermediate outputs:
results/<experiment>/
<task>/
<strategy>/
final_output.md # Final result
history.json # Full trajectory
pass_01/ # Per-iteration artifacts
version_a.md
version_b.md
critic.md
Separation of concerns — keep generation, evaluation, and visualization separate:
Many NLP, HCI, and alignment papers require human evaluation as primary or complementary evidence. Design this before running automated experiments — human eval often has longer lead times (IRB approval, annotator recruitment).
When human evaluation is needed:
Automated metrics don't capture what you care about (fluency, helpfulness, safety)
Your contribution is about human-facing qualities (readability, preference, trust)
Reviewers at NLP venues (ACL, EMNLP) expect it for generation tasks
Key design decisions:
Decision
Options
Guidance
Annotator type
Expert, crowdworker, end-user
Match to what your claims require
Scale
Likert (1-5), pairwise comparison, ranking
Pairwise is more reliable than Likert for LLM outputs
Sample size
Per annotator and total items
Power analysis or minimum 100 items, 3+ annotators
Agreement metric
Cohen's kappa, Krippendorff's alpha, ICC
Krippendorff's alpha for >2 annotators; report raw agreement too
Platform
Prolific, MTurk, internal team
Prolific for quality; MTurk for scale; internal for domain expertise
Annotation guideline checklist:
- [ ] Clear task description with examples (good AND bad)
- [ ] Decision criteria for ambiguous cases
- [ ] At least 2 worked examples per category
- [ ] Attention checks / gold standard items (10-15% of total)
- [ ] Qualification task or screening round
- [ ] Estimated time per item and fair compensation (>= local minimum wage)
- [ ] IRB/ethics review if required by your institution
Reporting requirements (reviewers check all of these):
Number of annotators and their qualifications
Inter-annotator agreement with specific metric and value
Annotation interface description or screenshot (appendix)
Total annotation time
See references/human-evaluation.md for complete guide including statistical tests for human eval data, crowdsourcing quality control patterns, and IRB guidance.
Phase 3: Experiment Execution & Monitoring
Goal: Run experiments reliably, monitor progress, recover from failures.
Step 3.1: Launch Experiments
Use nohup for long-running experiments:
nohup python run_experiment.py --config config.yaml > logs/experiment_01.log 2>&1 &
echo $! # Record the PID
Parallel execution: Run independent experiments simultaneously, but be aware of API rate limits. 4+ concurrent experiments on the same API will slow each down.
Step 3.2: Set Up Monitoring (Cron Pattern)
For long-running experiments, set up periodic status checks. The cron prompt should follow this template:
Monitor Prompt Template:
1. Check if process is still running: ps aux | grep <pattern>
2. Read last 30 lines of log: tail -30 <logfile>
3. Check for completed results: ls <result_dir>
4. If results exist, read and report: cat <result_file>
5. If all done, commit: git add -A && git commit -m "<descriptive message>" && git push
6. Report in structured format (tables with key metrics)
7. Answer the key analytical question for this experiment
Silent mode: If nothing has changed since the last check, respond with [SILENT] to suppress notification to the user. Only report when there's news.
Step 3.3: Handle Failures
Common failure modes and recovery:
Failure
Detection
Recovery
API rate limit / credit exhaustion
402/429 errors in logs
Wait, then re-run (scripts skip completed work)
Process crash
PID gone, incomplete results
Re-run from last checkpoint
Timeout on hard problems
Process stuck, no log progress
Kill and skip, note in results
Wrong model ID
Errors referencing model name
Fix ID and re-run
Key: Scripts should always check for existing results and skip completed work. This makes re-runs safe and efficient.
Step 3.4: Commit Completed Results
After each experiment batch completes:
git add -A
git commit -m "Add <experiment name>: <key finding in 1 line>"
git push
Step 3.5: Maintain an Experiment Journal
Git commits track what happened, but not the exploration tree — the decisions about what to try next based on what you learned. Maintain a structured experiment journal that captures this tree:
// experiment_journal.jsonl — append one entry per experiment attempt
{
"id": "exp_003",
"parent": "exp_001",
"timestamp": "2025-05-10T14:30:00Z",
"hypothesis": "Adding scope constraints will fix convergence failure from exp_001",
"plan": "Re-run autoreason with max_tokens=2000 and fixed structure template",
"config": { "model": "haiku", "strategy": "autoreason", "max_tokens": 2000 },
"status": "completed",
"result_path": "results/exp_003/",
"key_metrics": { "win_rate": 0.85, "convergence_rounds": 3 },
"analysis": "Scope constraints fixed convergence. Win rate jumped from 0.42 to 0.85.",
"next_steps": ["Try same constraints on Sonnet", "Test without structure template"],
"figures": ["figures/exp003_convergence.pdf"]
}
Why a journal, not just git? Git tracks file changes. The journal tracks the reasoning: why you tried X, what you learned, and what that implies for the next experiment. When writing the paper, this tree is invaluable for the Methods section ("we observed X, which motivated Y") and for honest failure reporting.
Selecting the best path: When the journal shows a branching tree (exp_001 → exp_002a, exp_002b, exp_003), identify the path that best supports the paper's claims. Document dead-end branches in the appendix as ablations or negative results.
Snapshot code per experiment: Copy the experiment script after each run:
This enables exact reproduction even after subsequent code changes.
Phase 4: Result Analysis
Goal: Extract findings, compute statistics, identify the story.
Step 4.1: Aggregate Results
Write analysis scripts that:
Load all result files from a batch
Compute per-task and aggregate metrics
Generate summary tables
# Standard analysis pattern
import json, os
from pathlib import Path
results = {}
for result_file in Path("results/").rglob("result.json"):
data = json.loads(result_file.read_text())
strategy = result_file.parent.name
task = result_file.parent.parent.name
results.setdefault(strategy, {})[task] = data
# Compute aggregate metrics
for strategy, tasks in results.items():
scores = [t["score"] for t in tasks.values()]
print(f"{strategy}: mean={np.mean(scores):.1f}, std={np.std(scores):.1f}")
Step 4.2: Statistical Significance
Always compute:
Error bars: Standard deviation or standard error, specify which
Confidence intervals: 95% CI for key results
Pairwise tests: McNemar's test for comparing two methods
Effect sizes: Cohen's d or h for practical significance
What is the main finding? State it in one sentence.
What surprised you? Unexpected results often make the best papers.
What failed? Failed experiments can be the most informative. Honest reporting of failures strengthens the paper.
What follow-up experiments are needed? Results often raise new questions.
Handling Negative or Null Results
When your hypothesis was wrong or results are inconclusive, you have three options:
Situation
Action
Venue Fit
Hypothesis wrong but why is informative
Frame paper around the analysis of why
NeurIPS, ICML (if analysis is rigorous)
Method doesn't beat baselines but reveals something new
Reframe contribution as understanding/analysis
ICLR (values understanding), workshop papers
Clean negative result on popular claim
Write it up — the field needs to know
NeurIPS Datasets & Benchmarks, TMLR, workshops
Results inconclusive, no clear story
Pivot — run different experiments or reframe
Don't force a paper that isn't there
How to write a negative results paper:
Lead with what the community believes and why it matters to test it
Describe your rigorous methodology (must be airtight — reviewers will scrutinize harder)
Present the null result clearly with statistical evidence
Analyze why the expected result didn't materialize
Discuss implications for the field
Venues that explicitly welcome negative results: NeurIPS (Datasets & Benchmarks track), TMLR, ML Reproducibility Challenge, workshops at major conferences. Some workshops specifically call for negative results.
Step 4.4: Create Figures and Tables
Figures:
Use vector graphics (PDF) for all plots: plt.savefig('fig.pdf')
Colorblind-safe palettes (Okabe-Ito or Paul Tol)
Self-contained captions — reader should understand without main text
No title inside figure — the caption serves this function
Step 4.6: Write the Experiment Log (Bridge to Writeup)
Before moving to paper writing, create a structured experiment log that bridges results to prose. This is the single most important connective tissue between experiments and the writeup — without it, the writing agent has to re-derive the story from raw result files.
Create experiment_log.md with the following structure:
# Experiment Log
## Contribution (one sentence)
[The paper's main claim]
## Experiments Run
### Experiment 1: [Name]
- **Claim tested**: [Which paper claim this supports]
- **Setup**: [Model, dataset, config, number of runs]
- **Key result**: [One sentence with the number]
- **Result files**: results/exp1/final_info.json
- **Figures generated**: figures/exp1_comparison.pdf
- **Surprising findings**: [Anything unexpected]
### Experiment 2: [Name]
...
## Figures
| Filename | Description | Which section it belongs in |
| --------------------------- | ---------------------------------------------- | --------------------------- |
| figures/main_comparison.pdf | Bar chart comparing all methods on benchmark X | Results, Figure 2 |
| figures/ablation.pdf | Ablation removing components A, B, C | Results, Figure 3 |
...
## Failed Experiments (document for honesty)
- [What was tried, why it failed, what it tells us]
## Open Questions
- [Anything the results raised that the paper should address]
Why this matters: When drafting, the agent (or a delegated sub-agent) can load experiment_log.md alongside the LaTeX template and produce a first draft grounded in actual results. Without this bridge, the writing agent must parse raw JSON/CSV files and infer the story — a common source of hallucinated or misreported numbers.
Git discipline: Commit this log alongside the results it describes.
Iterative Refinement: Strategy Selection
Any output in this pipeline — paper drafts, experiment scripts, analysis — can be iteratively refined. The autoreason research provides empirical evidence for when each refinement strategy works and when it fails. Use this section to choose the right approach.
Quick Decision Table
Your Situation
Strategy
Why
Mid-tier model + constrained task
Autoreason
Sweet spot. Generation-evaluation gap is widest. Baselines actively destroy weak model outputs.
Mid-tier model + open task
Autoreason with scope constraints added
Add fixed facts, structure, or deliverable to bound the improvement space.
Frontier model + constrained task
Autoreason
Wins 2/3 constrained tasks even at frontier.
Frontier model + unconstrained task
Critique-and-revise or single pass
Autoreason comes last. Model self-evaluates well enough.
Concrete technical task (system design)
Critique-and-revise
Direct find-and-fix loop is more efficient.
Template-filling task (one correct structure)
Single pass or conservative
Minimal decision space. Iteration adds no value.
Code with test cases
Autoreason (code variant)
Structured analysis of why it failed before fixing. Recovery rate 62% vs 43%.
Very weak model (Llama 8B class)
Single pass
Model too weak for diverse candidates. Invest in generation quality.
The Generation-Evaluation Gap
Core insight: Autoreason's value depends on the gap between a model's generation capability and its self-evaluation capability.
Model Tier │ Generation │ Self-Eval │ Gap │ Autoreason Value
──────────────────┼────────────┼───────────┼────────┼─────────────────
Weak (Llama 8B) │ Poor │ Poor │ Small │ None — can't generate diverse candidates
Mid (Haiku 3.5) │ Decent │ Poor │ LARGE │ MAXIMUM — 42/42 perfect Borda
Mid (Gemini Flash)│ Decent │ Moderate │ Large │ High — wins 2/3
Strong (Sonnet 4) │ Good │ Decent │ Medium │ Moderate — wins 3/5
Frontier (S4.6) │ Excellent │ Good │ Small │ Only with constraints
This gap is structural, not temporary. As costs drop, today's frontier becomes tomorrow's mid-tier. The sweet spot moves but never disappears.
Autoreason Loop (Summary)
Each pass produces three candidates from fresh, isolated agents:
Critic → finds problems in incumbent A (no fixes)
Author B → revises A based on critique
Synthesizer → merges A and B (randomized labels)
Judge Panel → 3 blind CoT judges rank A, B, AB via Borda count
Convergence → A wins k=2 consecutive passes → done
Key parameters:
k=2 convergence (k=1 premature, k=3 too expensive, no quality gain)
CoT judges always (3x faster convergence)
Temperature 0.8 authors, 0.3 judges
Conservative tiebreak: incumbent wins ties
Every role is a fresh agent with no shared context
Applying to Paper Drafts
When refining the paper itself through autoreason:
Provide ground truth to the critic: actual experimental data, result JSONs, statistical outputs. Without this, models hallucinate fabricated ablation studies and fake confidence intervals.
Use 3 working judges minimum: A broken judge parser doesn't add noise — it prevents equilibrium entirely.
Scope constrain the revision: "Address these specific weaknesses" not "improve the paper."
Failure Modes
Failure
Detection
Fix
No convergence (A never wins)
A wins <15% over 20+ passes
Add scope constraints to the task
Synthesis drift
Word counts grow unboundedly
Constrain structure and deliverable
Degradation below single pass
Baselines score higher than iterated output
Switch to single pass; model may be too weak
Overfitting (code)
High public-test pass, low private-test pass
Use structured analysis, not just test feedback
Broken judges
Parsing failures reduce panel below 3
Fix parser before continuing
See references/autoreason-methodology.md for complete prompts, Borda scoring details, model selection guide, scope constraint design patterns, and compute budget reference.
Phase 5: Paper Drafting
Goal: Write a complete, publication-ready paper.
Context Management for Large Projects
A paper project with 50+ experiment files, multiple result directories, and extensive literature notes can easily exceed the agent's context window. Manage this proactively:
What to load into context per drafting task:
Drafting Task
Load Into Context
Do NOT Load
Writing Introduction
experiment_log.md, contribution statement, 5-10 most relevant paper abstracts
Raw result JSONs, full experiment scripts, all literature notes
experiment_log.md is the primary context bridge — it summarizes everything needed for writing without loading raw data files (see Step 4.6)
Load one section's context at a time when delegating. A sub-agent drafting Methods doesn't need the literature review notes.
Summarize, don't include raw files. For a 200-line result JSON, load a 10-line summary table. For a 50-page related paper, load the 5-sentence abstract + your 2-line note about its relevance.
For very large projects: Create a context/ directory with pre-compressed summaries:
context/
contribution.md # 1 sentence
experiment_summary.md # Key results table (from experiment_log.md)
literature_map.md # Organized citation notes
figure_inventory.md # List of figures with descriptions
The Narrative Principle
The single most critical insight: Your paper is not a collection of experiments — it's a story with one clear contribution supported by evidence.
Every successful ML paper centers on what Neel Nanda calls "the narrative": a short, rigorous, evidence-based technical story with a takeaway readers care about.
Three Pillars (must be crystal clear by end of introduction):
Pillar
Description
Test
The What
1-3 specific novel claims
Can you state them in one sentence?
The Why
Rigorous empirical evidence
Do experiments distinguish your hypothesis from alternatives?
The So What
Why readers should care
Does this connect to a recognized community problem?
If you cannot state your contribution in one sentence, you don't yet have a paper.
The Sources Behind This Guidance
This skill synthesizes writing philosophy from researchers who have published extensively at top venues. The writing philosophy layer was originally compiled by Orchestra Research as the ml-paper-writing skill.
Source
Key Contribution
Link
Neel Nanda (Google DeepMind)
The Narrative Principle, What/Why/So What framework
Why? Most reviewers form judgments before reaching your methods. Readers encounter your paper as: title → abstract → introduction → figures → maybe the rest.
Writing Workflow
Paper Writing Checklist:
- [ ] Step 1: Define the one-sentence contribution
- [ ] Step 2: Draft Figure 1 (core idea or most compelling result)
- [ ] Step 3: Draft abstract (5-sentence formula)
- [ ] Step 4: Draft introduction (1-1.5 pages max)
- [ ] Step 5: Draft methods
- [ ] Step 6: Draft experiments & results
- [ ] Step 7: Draft related work
- [ ] Step 8: Draft conclusion & discussion
- [ ] Step 9: Draft limitations (REQUIRED by all venues)
- [ ] Step 10: Plan appendix (proofs, extra experiments, details)
- [ ] Step 11: Complete paper checklist
- [ ] Step 12: Final review
Two-Pass Refinement Pattern
When drafting with an AI agent, use a two-pass approach (proven effective in SakanaAI's AI-Scientist pipeline):
Pass 1 — Write + immediate refine per section:
For each section, write a complete draft, then immediately refine it in the same context. This catches local issues (clarity, flow, completeness) while the section is fresh.
Pass 2 — Global refinement with full-paper context:
After all sections are drafted, revisit each section with awareness of the complete paper. This catches cross-section issues: redundancy, inconsistent terminology, narrative flow, and gaps where one section promises something another doesn't deliver.
Second-pass refinement prompt (per section):
"Review the [SECTION] in the context of the complete paper.
- Does it fit with the rest of the paper? Are there redundancies with other sections?
- Is terminology consistent with Introduction and Methods?
- Can anything be cut without weakening the message?
- Does the narrative flow from the previous section and into the next?
Make minimal, targeted edits. Do not rewrite from scratch."
LaTeX Error Checklist
Append this checklist to every refinement prompt. These are the most common errors when LLMs write LaTeX:
LaTeX Quality Checklist (verify after every edit):
- [ ] No unenclosed math symbols ($ signs balanced)
- [ ] Only reference figures/tables that exist (\ref matches \label)
- [ ] No fabricated citations (\cite matches entries in .bib)
- [ ] Every \begin{env} has matching \end{env} (especially figure, table, algorithm)
- [ ] No HTML contamination (</end{figure}> instead of \end{figure})
- [ ] No unescaped underscores outside math mode (use \_ in text)
- [ ] No duplicate \label definitions
- [ ] No duplicate section headers
- [ ] Numbers in text match actual experimental results
- [ ] All figures have captions and labels
- [ ] No overly long lines that cause overfull hbox warnings
Step 5.0: Title
The title is the single most-read element of the paper. It determines whether anyone clicks through to the abstract.
Good titles:
State the contribution or finding: "Autoreason: When Iterative LLM Refinement Works and Why It Fails"
Highlight a surprising result: "Scaling Data-Constrained Language Models" (implies you can)
Name the method + what it does: "DPO: Direct Preference Optimization of Language Models"
Bad titles:
Too generic: "An Approach to Improving Language Model Outputs"
Too long: anything over ~15 words
Jargon-only: "Asymptotic Convergence of Iterative Stochastic Policy Refinement" (who is this for?)
Rules:
Include your method name if you have one (for citability)
Include 1-2 keywords reviewers will search for
Avoid colons unless both halves carry meaning
Test: would a reviewer know the domain and contribution from the title alone?
Step 5.1: Abstract (5-Sentence Formula)
From Sebastian Farquhar (DeepMind):
1. What you achieved: "We introduce...", "We prove...", "We demonstrate..."
2. Why this is hard and important
3. How you do it (with specialist keywords for discoverability)
4. What evidence you have
5. Your most remarkable number/result
Delete generic openings like "Large language models have achieved remarkable success..."
Step 5.2: Figure 1
Figure 1 is the second thing most readers look at (after abstract). Draft it before writing the introduction — it forces you to clarify the core idea.
Figure 1 Type
When to Use
Example
Method diagram
New architecture or pipeline
TikZ flowchart showing your system
Results teaser
One compelling result tells the whole story
Bar chart: "Ours vs baselines" with clear gap
Problem illustration
The problem is unintuitive
Before/after showing failure mode you fix
Conceptual diagram
Abstract contribution needs visual grounding
2x2 matrix of method properties
Rules: Figure 1 must be understandable without reading any text. The caption alone should communicate the core idea. Use color purposefully — don't just decorate.
Step 5.3: Introduction (1-1.5 pages max)
Must include:
Clear problem statement
Brief approach overview
2-4 bullet contribution list (max 1-2 lines each in two-column format)
Methods should start by page 2-3
Step 5.4: Methods
Enable reimplementation:
Conceptual outline or pseudocode
All hyperparameters listed
Architectural details sufficient for reproduction
Present final design decisions; ablations go in experiments
Step 5.5: Experiments & Results
For each experiment, explicitly state:
What claim it supports
How it connects to main contribution
What to observe: "the blue line shows X, which demonstrates Y"
Requirements:
Error bars with methodology (std dev vs std error)
Hyperparameter search ranges
Compute infrastructure (GPU type, total hours)
Seed-setting methods
Step 5.6: Related Work
Organize methodologically, not paper-by-paper. Cite generously — reviewers likely authored relevant papers.
Step 5.7: Limitations (REQUIRED)
All major conferences require this. Honesty helps:
Reviewers are instructed not to penalize honest limitation acknowledgment
Pre-empt criticisms by identifying weaknesses first
Full hyperparameter tables, training details, hardware specs, random seeds
Dataset Documentation
Data collection process, annotation guidelines, licensing, preprocessing
Prompts & Templates
Exact prompts used (for LLM-based methods), evaluation templates
Human Evaluation
Annotation interface screenshots, instructions given to annotators, IRB details
Additional Figures
Per-task breakdowns, trajectory visualizations, failure case examples
Rules:
The main paper must be self-contained — reviewers are not required to read appendices
Never put critical evidence only in the appendix
Cross-reference: "Full results in Table 5 (Appendix B)" not just "see appendix"
Use \appendix command, then \section{A: Proofs} etc.
Page Budget Management
When over the page limit:
Cut Strategy
Saves
Risk
Move proofs to appendix
0.5-2 pages
Low — standard practice
Condense related work
0.5-1 page
Medium — may miss key citations
Combine tables with subfigures
0.25-0.5 page
Low — often improves readability
Use \vspace{-Xpt} sparingly
0.1-0.3 page
Low if subtle, high if obvious
Remove qualitative examples
0.5-1 page
Medium — reviewers like examples
Reduce figure sizes
0.25-0.5 page
High — figures must remain readable
Do NOT: reduce font size, change margins, remove required sections (limitations, broader impact), or use \small/\footnotesize for main text.
Step 5.10: Ethics & Broader Impact Statement
Most venues now require or strongly encourage an ethics/broader impact statement. This is not boilerplate — reviewers read it and can flag ethics concerns that trigger desk rejection.
What to include:
Component
Content
Required By
Positive societal impact
How your work benefits society
NeurIPS, ICML
Potential negative impact
Misuse risks, dual-use concerns, failure modes
NeurIPS, ICML
Fairness & bias
Does your method/data have known biases?
All venues (implicitly)
Environmental impact
Compute carbon footprint for large-scale training
ICML, increasingly NeurIPS
Privacy
Does your work use or enable processing of personal data?
ACL, NeurIPS
LLM disclosure
Was AI used in writing or experiments?
ICLR (mandatory), ACL
Writing the statement:
\section*{Broader Impact Statement}
% NeurIPS/ICML: after conclusion, does not count toward page limit
% 1. Positive applications (1-2 sentences)
This work enables [specific application] which may benefit [specific group].
% 2. Risks and mitigations (1-3 sentences, be specific)
[Method/model] could potentially be misused for [specific risk]. We mitigate
this by [specific mitigation, e.g., releasing only model weights above size X,
including safety filters, documenting failure modes].
% 3. Limitations of impact claims (1 sentence)
Our evaluation is limited to [specific domain]; broader deployment would
require [specific additional work].
Common mistakes:
Writing "we foresee no negative impacts" (almost never true — reviewers distrust this)
Being vague: "this could be misused" without specifying how
Ignoring compute costs for large-scale work
Forgetting to disclose LLM use at venues that require it
# Estimate using ML CO2 Impact tool methodology
gpu_hours = 1000 # total GPU hours
gpu_tdp_watts = 400 # e.g., A100 = 400W
pue = 1.1 # Power Usage Effectiveness (data center overhead)
carbon_intensity = 0.429 # kg CO2/kWh (US average; varies by region)
energy_kwh = (gpu_hours * gpu_tdp_watts * pue) / 1000
carbon_kg = energy_kwh * carbon_intensity
print(f"Energy: {energy_kwh:.0f} kWh, Carbon: {carbon_kg:.0f} kg CO2eq")
Step 5.11: Datasheets & Model Cards (If Applicable)
If your paper introduces a new dataset or releases a model, include structured documentation. Reviewers increasingly expect this, and NeurIPS Datasets & Benchmarks track requires it.
Datasheets for Datasets (Gebru et al., 2021) — include in appendix:
Dataset Documentation (Appendix):
- Motivation: Why was this dataset created? What task does it support?
- Composition: What are the instances? How many? What data types?
- Collection: How was data collected? What was the source?
- Preprocessing: What cleaning/filtering was applied?
- Distribution: How is the dataset distributed? Under what license?
- Maintenance: Who maintains it? How to report issues?
- Ethical considerations: Contains personal data? Consent obtained?
Potential for harm? Known biases?
Model Cards (Mitchell et al., 2019) — include in appendix for model releases:
Model Card (Appendix):
- Model details: Architecture, training data, training procedure
- Intended use: Primary use cases, out-of-scope uses
- Metrics: Evaluation metrics and results on benchmarks
- Ethical considerations: Known biases, fairness evaluations
- Limitations: Known failure modes, domains where model underperforms
Always copy the entire template directory first, then write within it.
Template Setup Checklist:
- [ ] Step 1: Copy entire template directory to new project
- [ ] Step 2: Verify template compiles as-is (before any changes)
- [ ] Step 3: Read the template's example content to understand structure
- [ ] Step 4: Replace example content section by section
- [ ] Step 5: Use template macros (check preamble for \newcommand definitions)
- [ ] Step 6: Clean up template artifacts only at the end
Step 1: Copy the Full Template
cp -r templates/neurips2025/ ~/papers/my-paper/
cd ~/papers/my-paper/
ls -la # Should see: main.tex, neurips.sty, Makefile, etc.
Copy the ENTIRE directory, not just the .tex file. Templates include style files (.sty), bibliography styles (.bst), example content, and Makefiles.
Step 2: Verify Template Compiles First
Before making ANY changes:
latexmk -pdf main.tex
# Or manual: pdflatex main.tex && bibtex main && pdflatex main.tex && pdflatex main.tex
If the unmodified template doesn't compile, fix that first (usually missing TeX packages — install via tlmgr install <package>).
Step 3: Keep Template Content as Reference
Don't immediately delete example content. Comment it out and use as formatting reference:
% Template example (keep for reference):
% \begin{figure}[t]
% \centering
% \includegraphics[width=0.8\linewidth]{example-image}
% \caption{Template shows caption style}
% \end{figure}
% Your actual figure:
\begin{figure}[t]
\centering
\includegraphics[width=0.8\linewidth]{your-figure.pdf}
\caption{Your caption following the same style.}
\end{figure}
Step 4: Replace Content Section by Section
Work through systematically: title/authors → abstract → introduction → methods → experiments → related work → conclusion → references → appendix. Compile after each section.
Include direction symbols ($\uparrow$ higher better, $\downarrow$ lower better)
Right-align numerical columns
Consistent decimal precision
Figures:
Vector graphics (PDF, EPS) for all plots and diagrams — plt.savefig('fig.pdf')
Raster (PNG 600 DPI) only for photographs
Colorblind-safe palettes (Okabe-Ito or Paul Tol)
Verify grayscale readability (8% of men have color vision deficiency)
No title inside figure — the caption serves this function
Self-contained captions — reader should understand without main text
Conference Resubmission
For converting between venues, see Phase 7 (Submission Preparation) — it covers the full conversion workflow, page-change table, and post-rejection guidance.
Professional LaTeX Preamble
Add these packages to any paper for professional quality. They are compatible with all major conference style files:
% --- Professional Packages (add after conference style file) ---
% Typography
\usepackage{microtype} % Microtypographic improvements (protrusion, expansion)
% Makes text noticeably more polished — always include
% Tables
\usepackage{booktabs} % Professional table rules (\toprule, \midrule, \bottomrule)
\usepackage{siunitx} % Consistent number formatting, decimal alignment
% Usage: \num{12345} → 12,345; \SI{3.5}{GHz} → 3.5 GHz
% Table alignment: S column type for decimal-aligned numbers
% Figures
\usepackage{graphicx} % Include graphics (\includegraphics)
\usepackage{subcaption} % Subfigures with (a), (b), (c) labels
% Usage: \begin{subfigure}{0.48\textwidth} ... \end{subfigure}
% Diagrams and Algorithms
\usepackage{tikz} % Programmable vector diagrams
\usetikzlibrary{arrows.meta, positioning, shapes.geometric, calc, fit, backgrounds}
\usepackage[ruled,vlined]{algorithm2e} % Professional pseudocode
% Alternative: \usepackage{algorithmicx} if template bundles it
% Cross-references
\usepackage{cleveref} % Smart references: \cref{fig:x} → "Figure 1"
% MUST be loaded AFTER hyperref
% Handles: figures, tables, sections, equations, algorithms
% Math (usually included by conference .sty, but verify)
\usepackage{amsmath,amssymb} % AMS math environments and symbols
\usepackage{mathtools} % Extends amsmath (dcases, coloneqq, etc.)
% Colors (for figures and diagrams)
\usepackage{xcolor} % Color management
% Okabe-Ito colorblind-safe palette:
\definecolor{okblue}{HTML}{0072B2}
\definecolor{okorange}{HTML}{E69F00}
\definecolor{okgreen}{HTML}{009E73}
\definecolor{okred}{HTML}{D55E00}
\definecolor{okpurple}{HTML}{CC79A7}
\definecolor{okcyan}{HTML}{56B4E9}
\definecolor{okyellow}{HTML}{F0E442}
Notes:
microtype is the single highest-impact package for visual quality. It adjusts character spacing at a sub-pixel level. Always include it.
siunitx handles decimal alignment in tables via the S column type — eliminates manual spacing.
cleveref must be loaded afterhyperref. Most conference .sty files load hyperref, so put cleveref last.
Check if the conference template already loads any of these (especially algorithm, amsmath, graphicx). Don't double-load.
siunitx Table Alignment
siunitx makes number-heavy tables significantly more readable:
The S column type auto-aligns on the decimal point. Headers in {} escape the alignment.
Subfigures
Standard pattern for side-by-side figures:
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{fig_results_a.pdf}
\caption{Results on Dataset A.}
\label{fig:results-a}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{fig_results_b.pdf}
\caption{Results on Dataset B.}
\label{fig:results-b}
\end{subfigure}
\caption{Comparison of our method across two datasets. (a) shows the scaling
behavior and (b) shows the ablation results. Both use 5 random seeds.}
\label{fig:results}
\end{figure}
Use \cref{fig:results} → "Figure 1", \cref{fig:results-a} → "Figure 1a".
This produces a PDF with deletions in red strikethrough and additions in blue — standard format for rebuttal supplements.
SciencePlots for matplotlib
Install and use for publication-quality plots:
pip install SciencePlots
import matplotlib.pyplot as plt
import scienceplots # registers styles
# Use science style (IEEE-like, clean)
with plt.style.context(['science', 'no-latex']):
fig, ax = plt.subplots(figsize=(3.5, 2.5)) # Single-column width
ax.plot(x, y, label='Ours', color='#0072B2')
ax.plot(x, y2, label='Baseline', color='#D55E00', linestyle='--')
ax.set_xlabel('Training Steps')
ax.set_ylabel('Accuracy')
ax.legend()
fig.savefig('paper/fig_results.pdf', bbox_inches='tight')
# Available styles: 'science', 'ieee', 'nature', 'science+ieee'
# Add 'no-latex' if LaTeX is not installed on the machine generating plots
Standard figure sizes (two-column format):
Single column: figsize=(3.5, 2.5) — fits in one column
Double column: figsize=(7.0, 3.0) — spans both columns
Square: figsize=(3.5, 3.5) — for heatmaps, confusion matrices
Phase 6: Self-Review & Revision
Goal: Simulate the review process before submission. Catch weaknesses early.
Step 6.1: Simulate Reviews (Ensemble Pattern)
Generate reviews from multiple perspectives. The key insight from automated research pipelines (notably SakanaAI's AI-Scientist): ensemble reviewing with a meta-reviewer produces far more calibrated feedback than a single review pass.
Step 1: Generate N independent reviews (N=3-5)
Use different models or temperature settings. Each reviewer sees only the paper, not other reviews. Default to negative bias — LLMs have well-documented positivity bias in evaluation.
You are an expert reviewer for [VENUE]. You are critical and thorough.
If a paper has weaknesses or you are unsure about a claim, flag it clearly
and reflect that in your scores. Do not give the benefit of the doubt.
Review this paper according to the official reviewer guidelines. Evaluate:
1. Soundness (are claims well-supported? are baselines fair and strong?)
2. Clarity (is the paper well-written? could an expert reproduce it?)
3. Significance (does this matter to the community?)
4. Originality (new insights, not just incremental combination?)
Provide your review as structured JSON:
{
"summary": "2-3 sentence summary",
"strengths": ["strength 1", "strength 2", ...],
"weaknesses": ["weakness 1 (most critical)", "weakness 2", ...],
"questions": ["question for authors 1", ...],
"missing_references": ["paper that should be cited", ...],
"soundness": 1-4,
"presentation": 1-4,
"contribution": 1-4,
"overall": 1-10,
"confidence": 1-5
}
Step 2: Meta-review (Area Chair aggregation)
Feed all N reviews to a meta-reviewer:
You are an Area Chair at [VENUE]. You have received [N] independent reviews
of a paper. Your job is to:
1. Identify consensus strengths and weaknesses across reviewers
2. Resolve disagreements by examining the paper directly
3. Produce a meta-review that represents the aggregate judgment
4. Use AVERAGED numerical scores across all reviews
Be conservative: if reviewers disagree on whether a weakness is serious,
treat it as serious until the authors address it.
Reviews:
[review_1]
[review_2]
...
Step 3: Reflection loop (optional, 2-3 rounds)
Each reviewer can refine their review after seeing the meta-review. Use an early termination sentinel: if the reviewer responds "I am done" (no changes), stop iterating.
Model selection for reviewing: Reviewing is best done with the strongest available model, even if you wrote the paper with a cheaper one. The reviewer model should be chosen independently from the writing model.
Few-shot calibration: If available, include 1-2 real published reviews from the target venue as examples. This dramatically improves score calibration. See references/reviewer-guidelines.md for example reviews.
Step 6.1b: Visual Review Pass (VLM)
Text-only review misses an entire class of problems: figure quality, layout issues, visual consistency. If you have access to a vision-capable model, run a separate visual review on the compiled PDF:
You are reviewing the visual presentation of this research paper PDF.
Check for:
1. Figure quality: Are plots readable? Labels legible? Colors distinguishable?
2. Figure-caption alignment: Does each caption accurately describe its figure?
3. Layout issues: Orphaned section headers, awkward page breaks, figures far from their references
4. Table formatting: Aligned columns, consistent decimal precision, bold for best results
5. Visual consistency: Same color scheme across all figures, consistent font sizes
6. Grayscale readability: Would the figures be understandable if printed in B&W?
For each issue, specify the page number and exact location.
This catches problems that text-based review cannot: a plot with illegible axis labels, a figure placed 3 pages from its first reference, inconsistent color palettes between Figure 2 and Figure 5, or a table that's clearly wider than the column width.
Step 6.1c: Claim Verification Pass
After simulated reviews, run a separate verification pass. This catches factual errors that reviewers might miss:
Claim Verification Protocol:
1. Extract every factual claim from the paper (numbers, comparisons, trends)
2. For each claim, trace it to the specific experiment/result that supports it
3. Verify the number in the paper matches the actual result file
4. Flag any claim without a traceable source as [VERIFY]
For agent-based workflows: delegate verification to a fresh sub-agent that receives only the paper text and the raw result files. The fresh context prevents confirmation bias — the verifier doesn't "remember" what the results were supposed to be.
Step 6.2: Prioritize Feedback
After collecting reviews, categorize:
Priority
Action
Critical (technical flaw, missing baseline)
Must fix. May require new experiments → back to Phase 2
High (clarity issue, missing ablation)
Should fix in this revision
Medium (minor writing issues, extra experiments)
Fix if time allows
Low (style preferences, tangential suggestions)
Note for future work
Step 6.3: Revision Cycle
For each critical/high issue:
Identify the specific section(s) affected
Draft the fix
Verify the fix doesn't break other claims
Update the paper
Re-check against the reviewer's concern
Step 6.4: Rebuttal Writing
When responding to actual reviews (post-submission), rebuttals are a distinct skill from revision:
Format: Point-by-point. For each reviewer concern:
> R1-W1: "The paper lacks comparison with Method X."
We thank the reviewer for this suggestion. We have added a comparison with
Method X in Table 3 (revised). Our method outperforms X by 3.2pp on [metric]
(p<0.05). We note that X requires 2x our compute budget.
Rules:
Address every concern — reviewers notice if you skip one
Lead with the strongest responses
Be concise and direct — reviewers read dozens of rebuttals
Include new results if you ran experiments during the rebuttal period
Never be defensive or dismissive, even of weak criticisms
Use latexdiff to generate a marked-up PDF showing changes (see Professional LaTeX Tooling section)
Thank reviewers for specific, actionable feedback (not generic praise)
What NOT to do: "We respectfully disagree" without evidence. "This is out of scope" without explanation. Ignoring a weakness by only responding to strengths.
Step 6.5: Paper Evolution Tracking
Save snapshots at key milestones:
paper/
paper.tex # Current working version
paper_v1_first_draft.tex # First complete draft
paper_v2_post_review.tex # After simulated review
paper_v3_pre_submission.tex # Final before submission
paper_v4_camera_ready.tex # Post-acceptance final
Phase 7: Submission Preparation
Goal: Final checks, formatting, and submission.
Step 7.1: Conference Checklist
Every venue has mandatory checklists. Complete them carefully — incomplete checklists can result in desk rejection.
Double-blind review means reviewers cannot know who wrote the paper. Check ALL of these:
Anonymization Checklist:
- [ ] No author names or affiliations anywhere in the PDF
- [ ] No acknowledgments section (add after acceptance)
- [ ] Self-citations written in third person: "Smith et al. [1] showed..." not "We previously showed [1]..."
- [ ] No GitHub/GitLab URLs pointing to your personal repos
- [ ] Use Anonymous GitHub (https://anonymous.4open.science/) for code links
- [ ] No institutional logos or identifiers in figures
- [ ] No file metadata containing author names (check PDF properties)
- [ ] No "our previous work" or "in our earlier paper" phrasing
- [ ] Dataset names don't reveal institution (rename if needed)
- [ ] Supplementary materials don't contain identifying information
Common mistakes: Git commit messages visible in supplementary code, watermarked figures from institutional tools, acknowledgments left in from a previous draft, arXiv preprint posted before anonymity period.
Step 7.3: Formatting Verification
Pre-Submission Format Check:
- [ ] Page limit respected (excluding references and appendix)
- [ ] All figures are vector (PDF) or high-res raster (600 DPI PNG)
- [ ] All figures readable in grayscale
- [ ] All tables use booktabs
- [ ] References compile correctly (no "?" in citations)
- [ ] No overfull hboxes in critical areas
