| name | eacl-experiments |
| description | Use when designing or auditing the empirical evidence for an EACL paper, covering tuned and LLM baselines, multilingual breadth matched to the claim, significance and variance floors, human-evaluation agreement, data-contamination controls, ablations, and error taxonomies, so that every stated result is measured rather than asserted. |
EACL Experiments
Use this to make an EACL paper's evidence hold up under NLP review. EACL rewards well-scoped
questions answered with careful controls over leaderboard maximalism — its best papers include
analyses and critiques, not only state-of-the-art systems (see
../../resources/exemplars/library.md). Design the evidence to match the claim exactly, no
broader.
Baselines that make a comparison fair
- Include a tuned baseline, not a strawman: an under-tuned competitor makes a win
meaningless. State the search space for both your method and the baselines.
- For LLM-based work, include the obvious prompt/few-shot baseline and report its prompts and
decoding settings; a gain over an unreported baseline is not credible.
Match breadth to the claim
| Claim | Required breadth |
|---|
| "Works for language L" | Solid results on L, honestly scoped |
| "Cross-lingual / multilingual" | Enough languages across resource levels; per-language results |
| "General method" | Multiple tasks/datasets, not one convenient benchmark |
| "Robust" | Stress tests / shifts, not just in-distribution |
A multilingual claim backed by two high-resource languages is the classic EACL over-reach — the
morphology-across-57-languages exemplar shows the bar.
Significance and variance floor
Evidence floor for a headline comparison:
seeds: >= 3-5 runs
report: mean +/- CI (or std), never a lone run
significance: a test when systems are close
ablations: isolate each component's contribution
Contamination controls
- For any benchmark evaluated with LLMs, address whether the test data could have leaked.
Report an overlap/decontamination check where feasible, or bound the risk honestly for
closed models. This is a live EACL concern, not a formality.
Human evaluation done properly
- If human judgments are a result, report the number of annotators, guidelines, pay, and
inter-annotator agreement — an unmeasured human eval is a soft target for reviewers.
- Release the annotation materials (see
eacl-artifact-evaluation).
Error analysis as a first-class result
- A quantified error taxonomy ("X% agreement errors, Y% named-entity errors, examples in
Table N") often carries more scientific weight than another decimal of accuracy, and plays to
EACL's analysis-friendly reviewing.
Audit checklist
[ ] Baselines tuned, search spaces stated
[ ] LLM baselines with verbatim prompts + decoding
[ ] Breadth matches the claim (per-language results if multilingual)
[ ] >= 3-5 seeds; variance/CIs reported
[ ] Significance test where systems are close
[ ] Ablations isolate each component
[ ] Contamination addressed
[ ] Human eval: annotators, agreement, pay reported
[ ] Error analysis quantified
Output format
[Evidence strength] Strong / Adequate / Weak
[Baseline fairness] <tuned? LLM baseline reported?>
[Breadth vs claim] <matched / over-reaching>
[Variance + significance] <seeds, CIs, tests>
[Contamination + human eval] <controls present?>
[Fix order] <experiments to add/scope before the cycle deadline>
