| name | empirical-experiments |
| description | Conventions for projects that generate their own results — running models, labeling data, computing statistics. Use when writing or running experiment code, or when drafting sections that report original results. |
Empirical Experiments
Some projects don't just review literature — they produce results. For these:
- Experiment code lives in
scripts/, generated data in data/, API caches in cache/. Both data/ and cache/ are git-ignored — scripts are the source of truth; anyone rebuilds the data by running them.
- Cache every expensive call. LLM API responses, dataset downloads, embeddings, model outputs — cached to disk on first run, so experiments are cheap to re-run and results stay stable across sessions.
- Log to files, not just the terminal. Long-running scripts write to a log (e.g.
cache/run-YYYY-MM-DD.log). Terminal output disappears; the paper's numbers must be re-checkable later.
- One note per experiment in
notes/: the exact command, parameters, and headline result. When drafting, numbers come from these notes and logs — never from memory.
- API keys go in
.env (ANTHROPIC_API_KEY, OPENAI_API_KEY are pre-listed in .env.example). Never hardcode keys.
- Negative and null results get recorded too. A control that showed nothing is evidence — it belongs in progress.md and often in the paper.
- Estimate cost before big API runs and tell the user ("~1,200 calls, roughly $2").
Reporting results in the manuscript
- Every number in a table or sentence traces to a specific log or output file — cite the file in the draft's Sources table
- Re-run or re-open the analysis when transcribing; never quote a statistic from memory
- Ablations and robustness checks that failed to change the result are worth a sentence — reviewers ask
