vibe-research

name	vibe-research
description	Use when starting an academic research session, planning literature review, designing experiments, or managing a multi-stage research workflow - parallel search, ground-truth verification, research knowledge base

Vibe Research

Overview

Engineering discipline applied to academic research. The three failure modes that kill research quality: serial search (missing related work), interpretation without verification (trusting Claude's summary over the paper), and undocumented dead ends (repeating failed experiments).

Core principle: Claude is a synthesis tool, not an authority. Always verify claims against primary sources.

When to Use

• Starting any research session (literature review, experiment planning, writing)
• Any stage where Claude summarizes papers, results, or findings
• Before committing to a research direction or experimental setup

Do NOT skip for "quick literature checks" — unverified summaries and serial searches create compounding errors.

Workflow

1. Lock the Research Question First

Before any search or experiment, write research_plan.md:

- Research question (specific, testable)
- Hypotheses (what you expect and why)
- Evaluation metrics (defined before seeing results)
- Baselines (what you're comparing against)
- Scope boundaries (what this study does NOT claim)

Read it before every major decision. For long sessions, also keep findings.md for key paper notes and decisions — prevents goal drift across iterations.

Why this matters: Metrics defined after seeing results = p-hacking. Baselines chosen after seeing your method = cherry-picking. Lock both before you look.

2. Parallel Literature Search

Run independent searches simultaneously, not serially:

Agent 1: Keyword cluster A (e.g., "gender bias medical QA LLM")
Agent 2: Keyword cluster B (e.g., "fairness healthcare NLP benchmark")
Agent 3: Citation network of seed paper
Agent 4: Related datasets / benchmarks

Different sources in parallel: arXiv, Semantic Scholar, ACL Anthology, OpenAlex. Synthesize after all return — never wait for one before starting another.

3. Primary Source Verification

Never trust Claude's interpretation. Always go to the source.

Claude summarizes paper → read the actual numbers in the paper
Claude reports experiment result → check the metric in W&B / log file
Claude says "significant" → check p-value and CI yourself
Claude lists citations → verify each one exists and says what Claude claims

Citation verification is mandatory. No exceptions.

Claude hallucinates citations. For every paper Claude names:

• Confirm it exists (search arXiv / Semantic Scholar by title + authors)
• Open the actual PDF — do not trust the abstract
• Find the specific claim Claude attributes to it (table, page, section)
• Note the exact location: "[Author Year, Table 3, p.8]"

A citation that can't be verified to page level should not be used.

EDA before statistics:

1. Plot raw data distributions
2. Look at failure cases before aggregate metrics
3. Check for data leakage, label imbalance, demographic skew
4. Then run statistical tests
5. Then interpret

Aggregate metrics hide what matters. Never skip to interpretation.

Forbidden shortcuts:

• Do NOT report a metric without first plotting its distribution
• Do NOT call a result "significant" without p-value + CI + sample size
• Do NOT skip failure case analysis ("the average looks fine" is not enough)

4. Research Knowledge Base (RESEARCH.md)

One file checked into the project repo, shared across sessions:

## Dead Ends (do not repeat)
- GPT-2 on MedQA: too small, results not publishable (tried 2026-01)
- WinoBias for medical domain: domain mismatch, reviewers will flag it

## Confirmed Findings
- Llama-3 shows 12% gap on female pronouns in clinical notes (our result)

## Key Papers (verified)
- [Author, Year, Venue] — one-line contribution summary

Every failed experiment → RESEARCH.md immediately. Negative results documented prevent the same mistake next week, next month, by a collaborator.

5. Model Selection

Decision	Model
Research question formulation	Opus + thinking
Experimental design	Opus + thinking
Interpreting ambiguous results	Opus + thinking
Keyword generation	Sonnet is fine
Formatting bibliography	Sonnet is fine

A flawed research design costs weeks of compute. Wrong fast reasoning is slower than right slow reasoning.

Quick Reference

Situation	Action
Starting research session	Write research_plan.md first
Literature search	Parallel agents, multiple keyword clusters
Claude names a paper	Verify it exists + verify the claim to page level
Claude says "results show X"	Check the actual metric / log / plot
Experiment failed	Document in RESEARCH.md immediately
Long session (>1hr)	Check research_plan.md — are you still on track?
About to run stats	EDA first: plot distributions, check failures
Choosing baselines	Lock baselines in research_plan.md before running your method

Tool Stack

Stage	Tools
Literature search	arXiv API, Semantic Scholar API, pyalex (OpenAlex), ACL Anthology
Paper reading + RAG	PaperQA2 (precise citations, low hallucination)
Related work drafting	Storm (Stanford) — structure first, then fill
Experiment tracking	W&B or MLflow — ground truth for all metrics
Experiment search	AIDE — metric-guided tree search for ML experiments
Writing	/20-ml-paper-writing skill for venue-specific structure

Red Flags — Stop and Verify

• "Claude said this paper found X, so let's build on that"
• "The results look good" (without checking raw metrics)
• "I'll document this dead end later"
• "Let me search one more keyword before designing the experiment"
• "These baselines make sense" (chosen after seeing your method's results)
• "I remember what we decided — no need to check the plan"

All of these introduce errors that compound. Stop and verify.