sigir-experiments

name	sigir-experiments
description	Use when designing or auditing the empirical program of a SIGIR paper — choosing test collections that match the claim, metric-cutoff discipline, paired significance testing with multiple-comparison correction, baseline tuning symmetry, ablations that isolate mechanisms, efficiency reporting, and LLM-era evaluation pitfalls.

SIGIR Experiments

SIGIR inherits its experimental culture from the Cranfield/TREC tradition: shared test collections, pooled relevance judgments, and statistical comparison of systems. Reviewers audit the protocol before they admire the numbers. This skill designs an evidence program that survives that audit — and flags the LLM-era failure modes that current program committees have learned to probe.

Claim → collection matching

The collection lineup is an argument about the claim's scope:

Claim type	Minimum credible lineup	Watch out for
Ad-hoc passage/document ranking	MS MARCO dev + TREC DL (multiple years)	Shallow-judgment bias on MARCO; report judged@k on DL
Zero-shot / generalization	A BEIR-style multi-collection suite	Cherry-picking subsets; state the full suite or the selection rule
Domain-specific retrieval	The domain's standard collection + one general control	Claiming generality from the domain alone
Recommendation	≥2 public interaction datasets with standard splits	Nonstandard splits that break comparability
Efficiency	Same quality collections + a latency/memory protocol	Quality-only baselines at different operating points
New task / evaluation method	A purpose-built collection with documented judgments	See sigir-artifact-evaluation for judgment standards

Rule: every collection has known ceiling effects and judgment quirks; one sentence acknowledging the relevant quirk ("MARCO's sparse judgments penalize novel-document retrieval; we therefore also report ...") converts a vulnerability into credibility.

Metrics and cutoffs

• Pre-commit to metrics that match the task stage: recall-oriented (R@1000) for first-stage retrieval, precision-oriented (nDCG@10, RR@10) for re-ranking and user-facing quality, and report both when the pipeline has both stages.
• Fix cutoffs before running; a paper whose cutoff varies by table row is assumed to have shopped.
• Compute every system's metrics with the same tool and flags; cross-paper metric implementations differ measurably (see sigir-reproducibility).

Significance testing: the house protocol

The community default is the paired test on per-topic scores with correction for multiple comparisons. A defensible standard setup:

# per-topic paired comparison, the SIGIR-standard shape
import ir_measures, scipy.stats as st
ours  = per_topic_scores("runs/ours.trec",  qrels, "nDCG@10")
base  = per_topic_scores("runs/base.trec", qrels, "nDCG@10")
t, p = st.ttest_rel(ours, base)            # paired t-test across topics
# correct across the family of comparisons you actually report:
# Bonferroni/Holm over {baselines} x {collections} x {metrics}

• Report the test name, correction, alpha, and n (topics) in the caption.
• Randomization/permutation tests are equally accepted; Wilcoxon is common; what is not accepted is no test under close deltas.
• Neural systems: ≥3 seeds, mean ± sd; significance on the per-topic means, and say which seed's run files ship in the repository.
• Effect size beats star-counting: a 0.3-point significant gain on a 700-topic collection is publishable as analysis, not as a "substantial improvement."

Baseline fairness — the objection that kills

The most common fatal review at SIGIR is some form of "the baselines were not given the same care." Inoculation checklist:

• Equal tuning budget per system, documented (search space, trials, dev split).
• Baselines at current strength: a tuned BM25 (k1/b swept), the strongest published configuration of each neural baseline, and at least one recent (last ~2 SIGIR/ECIR cycles) system in the family you claim to beat.
• Never mix copied numbers with computed numbers in one table silently; if you must quote a published number, mark it and explain the setup match.
• Same first-stage candidates, same re-ranking depth, same truncation for everyone.

Ablations and analysis

• One ablation per named mechanism: remove/replace it and show the delta on the headline metric — "the gain comes from X" needs the X-less row.
• Per-query analysis: win/loss buckets against the best baseline, with one diagnosed pattern (query type, length, term rarity) rather than anecdote screenshots.
• Sensitivity: the hyperparameter the method is most proud of gets a sweep plot.

Pre-registration worksheet (internal, one page, before running)

Freezing these six answers before the first run prevents the shopping patterns reviewers detect:

1. Primary claim, one sentence, with its scope qualifier.
2. Collections and why each is load-bearing for that scope.
3. Metrics + cutoffs (primary vs secondary, pre-committed).
4. Baseline set + per-system tuning budget.
5. The significance test, correction family, and alpha.
6. The ablation matrix: mechanism → isolating row.

Deviations during the project are fine — logged deviations are method; silent ones are p-hacking with extra steps.

LLM-era pitfalls reviewers now probe

• Contamination: models trained on the web have seen MARCO/BEIR text; say what you can about training-data overlap, and prefer post-cutoff or held-out topics where the claim depends on unseen data.
• LLM-as-judge: if you evaluate with an LLM assessor, validate it against human judgments on a subsample and report agreement; unvalidated LLM judgments as sole evidence are a growing desk-level concern.
• Prompt sensitivity: report the prompt, temperature, and n-trials for any generative component; single-shot generative numbers without variance are the new single-seed problem.
• API drift: name model versions and dates; "GPT-4" is not a reproducible system identifier.

Output format

[Claim-collection match] lineup adequate for claimed scope y/n; quirks acknowledged y/n
[Metric discipline] task-stage match / fixed cutoffs / single tool: pass each
[Statistics] test + correction + n named / seeds >=3 / effect size discussed
[Baseline fairness] tuning symmetry / current-strength set / no silent copied numbers
[Ablation coverage] mechanisms with isolating rows: <k>/<n>
[LLM-era risks] contamination / judge-validation / prompt-variance / version-pinning
[Weakest link] <the single protocol element a hostile reviewer attacks first>