| name | kdd-artifact-evaluation |
| description | Use when packaging code, datasets, configs, and deployment evidence for a KDD paper, where the repository cited in the submission is the only artifact reviewers can reach because rebuttals ban links. Covers anonymized repo construction, scale-claim harnesses, ADS evidence without production data, and post-acceptance release. |
KDD Artifact Evaluation
Use this while the submission is being assembled — not after. KDD's review mechanics
create one hard constraint that reorders all artifact work: the rebuttal phase does
not allow hyperlinks, so the anonymized repository referenced inside the submitted
PDF is the complete and final artifact channel for the whole review. There is no
"we'll share code if reviewers ask"; asking happens in a phase where you cannot answer
with a link.
Artifact strategy by track
| Track | Primary artifact | What reviewers actually probe | Non-shippable core, and its substitute |
|---|
| Research | Anonymized code + configs + data loaders | Can the headline table be regenerated? Does the scale claim have a runnable path? | Massive datasets → downsampled slice + full-scale download script |
| ADS | Measurement definitions + pipeline skeleton | Are post-launch metrics precisely defined? Is the eval window stated? | Production data/code → metric spec, schema, synthetic replay generator |
| Datasets & Benchmarks | The dataset itself + loaders + baseline harness | License, provenance, documentation, versioning | Nothing — the artifact is the paper |
Building the anonymized repository
- Create a fresh repository, never a scrubbed clone: git history, CI configs,
issue templates, and
.git metadata leak identity that no README edit removes.
- Sweep for: usernames in paths, institutional cluster names, cloud bucket names,
internal package registries, license headers, notebook execution metadata, and
dataset names that only one company uses.
- Cite the repository in the paper body (and optionally the abstract); after the
deadline this reference is unchangeable, so double-check the URL resolves logged out.
- Structure for a 10-minute reviewer, because that is the realistic inspection budget:
anonymous-artifact/
├── README.md
├── env/
├── configs/
├── data/
│ ├── get_data.sh
│ └── sample/
├── run.sh
└── results/expected/
docker run --rm -v $PWD:/w -w /w python:3.11 bash -c "pip install -r env/requirements.txt && bash run.sh --sample"
Scale claims need scale artifacts
KDD reviewers read "scales to billions of edges" as a checkable claim, not marketing:
- Ship the throughput/memory harness, not only accuracy scripts; a scalability figure
without its benchmark script is the least-trusted plot in the paper.
- Record hardware, dataset cardinalities (rows/edges/events), and wall-clock per run in
a machine-readable manifest, so the repo answers "on what?" precisely.
- Where the full-scale run needs a cluster no reviewer has, provide the mid-scale run
that completes on one machine plus the exact extrapolation logic the paper uses.
ADS evidence packaging without leaking production
The ADS track requires quantified post-launch performance, but production data almost
never ships. Reviewers accept that trade when the package contains:
- Exact metric definitions (numerator, denominator, unit, aggregation window) for
every post-launch number in the paper.
- The measurement design: A/B split or pre/post window, traffic share, duration, and
what guardrail metrics were tracked.
- A synthetic or replayed data generator that exercises the same pipeline shape, so
the method is runnable even though the evidence is observational.
- An honesty note distinguishing measured production numbers from simulated ones —
mixing them silently is a rejection pattern.
After acceptance
- Replace the anonymous mirror with the public, licensed, citable repository; the
camera-ready must carry the durable link (see
kdd-camera-ready).
- Prefer an archival deposit (DOI-stamped release) over a bare git URL for the version
that the proceedings reference.
- Check the current cycle for artifact-badging or code-availability initiatives before
assuming none exists (待核实 — such programs change year to year).
Vignette: packaging a billion-edge graph paper
A Research Track paper claims its sampler trains GNNs on a 3B-edge graph on one
machine. What the artifact must contain for that claim to survive contact with a
skeptical reviewer:
get_data.sh that downloads the public 3B-edge graph (or constructs it from
public parts) with checksums — a scale claim on an unfetchable graph is attested,
not rerunnable, and should be labeled accordingly (kdd-reproducibility).- A
--scale small|medium|full switch: small finishes on a laptop in minutes and
validates the pipeline; medium reproduces one main-table row on a single GPU
overnight; full documents the exact hardware used for the headline.
- The peak-memory tracker wired into every run, because the paper's "one machine"
claim is really a memory claim.
results/expected/ with per-scale reference outputs, so a reviewer's partial rerun
has something to diff against.
What it must not contain: the 40GB of intermediate artifacts (regenerable), the
authors' cluster submission scripts (identity leak), or a README promising "full
instructions after acceptance" — that sentence tells reviewers the artifact is
theater.
Common artifact failures at this venue
| Failure | Why it is fatal at KDD specifically |
|---|
| Repo created but never cited in the PDF | The link ban makes it undiscoverable during rebuttal |
| Git history preserved from the lab repo | Identity leak → desk-level anonymity problem |
| Accuracy scripts only, no efficiency harness | The paper's scale/efficiency axis becomes unverifiable |
| Sample data missing, full data gated | Reviewer's 10-minute budget ends at the download wall |
| ADS package with raw production extracts | Confidentiality violation risk transferred to reviewers |
Output format
[Artifact channel] repo cited in PDF: yes/no (if no: unrecoverable after deadline)
[Track register] research-repro / ads-deployment-evidence / dataset-release
[Regeneration level] one-command sample / scripted / descriptive only
[Scale evidence] throughput+memory harness: present / missing
[Anonymity sweep] <paths/history/metadata findings>
[Post-acceptance plan] <public repo, license, archival DOI>
