| name | cikm-reproducibility |
| description | Use when hardening the reproducibility of a CIKM paper — pinning the pipeline stages where IR, mining, and knowledge-management results silently diverge, documenting KGs and enterprise data that cannot be released, keeping the GenAI disclosure consistent with how code and data were produced, and preparing the post-acceptance release. |
CIKM Reproducibility
Reproducibility at CIKM has a venue-specific difficulty: the typical paper chains
components from different communities — an index, a graph, a model, a ruleset — and
each link has its own silent-divergence habits. A reader who cannot rebuild the
chain cannot attribute the result, and a blended review panel contains someone able
to notice each weak link.
Divergence map for chained pipelines
| Chain link | How results silently drift | Pin |
|---|
| Text preprocessing / indexing | Tokenizer versions, stopword lists, index-time defaults differ across toolkits | Name toolkit + version + config file in the artifact |
| KG snapshot | Public KGs (Wikidata-class) change daily; entity counts drift | Freeze and state the dump date; ship the extracted subgraph if licensable |
| Candidate generation | Recall stage caps and thresholds rarely reported | Report every cutoff; they bound the final metrics |
| Training | Seeds, hardware nondeterminism, early-stopping criteria | Seed policy + selection rule in the protocol paragraph |
| Evaluation | Metric implementations disagree at tie-breaking and cutoffs | Name the evaluation library version; never hand-roll silently |
| LLM components | Model version/API drift; prompts unlogged | Pin model identifiers and dates; log prompts verbatim in the artifact |
The discipline: for each link, either the artifact pins it or the paper states it.
A link pinned nowhere is where a failed replication will land.
Unreleasable data, releasable knowledge
CIKM's KM lane routinely involves enterprise corpora, clickstreams, or proprietary
KGs that cannot ship. The venue-honest pattern:
- Describe the unreleasable data statistically (size, schema, class balance,
collection window) at a level where a reader could construct a synthetic
analog — then actually provide that analog generator when feasible.
- Run the public-data variant of every headline experiment, even if the effect is
smaller; a result that exists only on invisible data asks the panel for faith.
- State the release position explicitly in the paper ("logs cannot be released;
the sampling script and schema are in the artifact") rather than leaving the
reader to discover the gap.
GenAI disclosure as a reproducibility document
CIKM 2026's mandatory GenAI Usage Disclosure covers code and data, not just prose
(source map, 2026-07-08). Treat it as part of the methods record: if evaluation
scripts, synthetic data, prompts, or labels were generated with AI assistance, the
disclosure plus the artifact should together let a reader judge what that implies
for the result. A disclosure that says "AI used for coding" while the artifact
contains unexplained generated labels is an inconsistency automated compliance
checks — which the conference reserves — or reviewers can catch.
Environment capture
Chained pipelines multiply environment surface, so capture it in layers:
| Layer | Capture mechanism |
|---|
| OS + system libraries | Container image or a documented base image tag |
| Language environments | Lockfiles (exact versions), not loose requirement ranges |
| Toolkit configs | The actual config files, committed — not "default settings" prose |
| Data inputs | Checksums + download scripts, or the frozen extraction (see KG row) |
| Hardware assumptions | GPU/CPU class and memory floor stated where results are timed |
The test is transferability: a lab-mate on a clean machine, without the authors in
the room, reaches the headline table. Running that internal replication before
submission is the single highest-yield reproducibility exercise — it finds the
unpinned link while it can still be pinned.
Where reproducibility pays at this venue
Three concrete CIKM payoffs beyond principle. First, the blended panel: whichever
lane doubts the result will probe its own link of the chain, so pinning every link
is defense in all three directions. Second, resource-track reviewers and readers
judge adoptability, which is reproducibility wearing its public face
(cikm-artifact-evaluation). Third, follow-up work: CIKM's back catalog shows
methods becoming standard baselines (DRMM, BERT4Rec); papers get that afterlife
only when third parties can run them — the reproducible version of a method is the
one that accumulates citations as a baseline.
Release timeline
Anonymized review artifact during submission (see cikm-supplementary for what the
budget permits); public repository at camera-ready, with license, versioned release
tag, and the exact commit that produced the proceedings numbers. The 2026
notification-to-camera-ready window is thirteen days — build the release during
the review wait (cikm-workflow Mode A), not inside that window.
Honest-failure disclosure
Chained pipelines rarely reproduce perfectly, and the venue-credible move is to
say so first: a REPRODUCING.md that states which numbers regenerate exactly, which
vary within a stated tolerance (GPU nondeterminism, sampling), and which depend on
restricted inputs and therefore only regenerate in public-analog form. Declared
tolerance reads as competence; discovered variance reads as concealment. The same
document is where to state known environment sensitivities ("results verified on
CUDA X; version Y shifts Table 3 by ±0.2") — the sentence that saves a replicator
a week is the sentence that earns the citation.
One-command bar
git clone <repo> && cd <repo>
make setup
make table2
If make table2 cannot exist because data is restricted, the repo must say so at
the top and offer the public-variant target instead. Silent partiality — a repo that
looks complete but is not runnable — costs more reviewer goodwill than an honest
scope statement.
Output format
[Chain audit] <link → pinned where (paper / artifact / nowhere)>
[Data position] <releasable / described+analog / public-variant-only>
[Disclosure consistency] <GenAI section vs. artifact contents>
[Release plan] <review artifact state → camera-ready repo state, dated>
[Weakest link] <the divergence a replicator would hit first>
