| name | webconf-experiments |
| description | Use when designing or auditing the empirical section of a Web Conference (WWW) paper — matching evidence to the claim's scale, choosing datasets with provenance and freshness, blocking temporal and popularity leakage, running honest baselines from the sibling circuit, and deciding when live-platform or user-study evidence is required. |
Web Conference Experiments
The empirical bar at this venue is not "more datasets" — it is evidence whose
scale, freshness, and realism match the claim. A method claiming web-scale
efficiency must show the scaling curve; a measurement claiming platform-general
behavior must show more than one platform; a system claiming deployability must
show cost under realistic load. Audit the claim-evidence match before adding
anything.
Claim-to-evidence contract
| Claim type | Minimum honest evidence | Habitual shortfall |
|---|
| "Outperforms" (quality) | Tuned recent baselines, repeated runs, variance, significance | Untuned baselines from 3-year-old code |
| "Scales" (efficiency) | Cost curves across ≥2 orders of magnitude, hardware stated | One big-dataset wall-clock number |
| "Generalizes" (external validity) | ≥2 platforms/domains or a stated single-platform scope | Silent single-platform universality |
| "Measures" (phenomenon) | Construct definition, sampling frame, bot/spam handling, error bars | Convenience crawl treated as census |
| "Deploys" (system) | Load, latency percentiles, failure behavior; A/B where claimed | Demo-grade throughput on toy traffic |
Dataset selection with provenance
- Prefer datasets with documented collection: source, crawl window, sampling
rule, known biases. An undocumented Kaggle mirror of a platform dump is a
provenance liability reviewers now flag.
- Freshness is evidential here: user behavior and platform mechanics drift, so
a 2015 interaction dataset supports historical claims, not claims about current
platforms. Either add a recent corpus or scope the claim in time.
- Report per-dataset statistics (nodes/edges/users/items/time span) in a table, and
state why this set of datasets spans the claim — heterogeneity in domain, size,
and density is the point, not the count.
- For data you collected: consent/ToS posture and aggregation level belong beside
the dataset description (body text, per
webconf-writing-style).
Leakage: the venue's most-caught methodological bug
Web data is temporal, popularity-skewed, and duplicated across the crawl. Three
specific leaks to audit:
- Temporal: random splits on interactions or evolving graphs train on the
future. Use a time-based split rule and state it ("train < T ≤ test").
- Popularity: negative sampling that mirrors test popularity inflates ranking
metrics; report at least one popularity-debiased or full-ranking metric.
- Duplication/contamination: near-duplicate pages across splits, and — for
LLM-era pipelines — benchmark text present in a foundation model's pretraining.
State the dedup rule; where an external LLM is a component, discuss
contamination explicitly rather than hoping no one asks.
Baselines and ablations
Baselines come from the sibling circuit's current cycle (WWW, WSDM, SIGIR,
KDD, CIKM within ~2 years), tuned with the same budget as your method — state the
search space for both, since asymmetric tuning is the most common quiet unfairness.
Ablate the claimed mechanism, not everything: if the paper's story is "the
crawl-time signal is what helps," the ablation table must contain exactly the row
that removes it.
Experiment matrix skeleton (fill per claim, not per dataset):
claim C1 "quality" : datasets D1-D3 x {ours, B1..B4} x 5 seeds -> mean±sd, test
claim C2 "scales" : D_synthetic sized 10^6..10^9 edges x {ours, B1} -> cost curve
claim C3 "mechanism": ours minus {signal S, module M} on D1-D3 -> delta table
claim C4 "external" : platform P2 replication of headline result -> one table
Every table row answers a named claim; rows answering nothing get cut.
Statistics that survive review
- Repeated runs with different seeds wherever training is stochastic; report
mean ± sd and the number of runs.
- Significance tests matched to the design (paired across datasets/queries), with
effect sizes — at web scale everything is "significant," so the delta's
magnitude and cost carry the argument.
- Percentiles, not means, for latency and exposure-type quantities; web
distributions are heavy-tailed and means mislead.
When offline evidence is not enough
Claims about user response (satisfaction, engagement shifts, behavior change)
need a user study or an online experiment, with ethics posture stated; claims about
production viability need load realism. If neither is obtainable, weaken the
claim to what offline evidence supports — "improves offline ranking quality" is
publishable; "improves user satisfaction" without users is a rebuttal-week wound
that cannot be healed in seven days.
Predictable objections and their cheap preemptions
Four objections recur across this venue's tracks; each has a preemption that
costs a sentence or a table row, not a new experiment campaign:
- "Gains may come from capacity, not the proposed component" — include one
parameter-matched ablation row and state parameter counts in the caption.
- "Datasets are all from one platform family" — either add the second-family
corpus or write the scope sentence in the conclusion; silence converts a
limitation into a discovered flaw.
- "The baseline numbers differ from the original paper" — declare the
re-implementation and the protocol difference (split, metric variant,
preprocessing) in a footnote at first mention; unexplained deltas read as
either sloppiness or gaming.
- "Efficiency claims lack a cost axis" — every quality table involving a
method whose selling point includes scale gets a companion column (time,
memory, or queries), because at web scale a 0.5-point win at 10x cost is a
negative result.
The meta-rule: reviewers here are drawn from a circuit that reviews the same
methods at WSDM, SIGIR, and KDD in the same year; they have seen this quarter's
common failure modes several times already. The paper that names its own
weaknesses first is the one whose rebuttal week is quiet.
Audit checklist
Output format
[Claim-evidence map] C1..Cn -> table/figure or GAP
[Leakage audit] temporal / popularity / duplication: clean or findings
[Baseline fairness] recency, tuning symmetry: pass/fail
[Scale realism] does evidence scale match claim scale? <notes>
[Required additions] <ranked by review risk, with cost estimate>
