vldb-reproducibility

name	vldb-reproducibility
description	Use when engineering reproducibility into a VLDB paper before submission, covering hardware and configuration disclosure, dataset and workload provenance, run-to-run variance in systems measurements, competitor-version pinning, figure-to-raw-data traceability, and the disclosure floor PVLDB reviewers apply to performance claims.

VLDB Reproducibility

Use this while experiments are still running — reproducibility at a systems venue is an experimental-design property, not a packaging step. The question a PVLDB reviewer silently asks of every performance figure: could a competent lab, given this paper alone, land within noise of these curves?

The disclosure floor

Every performance claim needs its context recoverable from the paper (or its cited artifact):

• Hardware: CPU model and count, memory, storage class and interface, network fabric, node count. "A commodity server" reproduces nothing.
• Software: OS, kernel where it matters (I/O experiments), compiler and flags, and the exact versions and configurations of every system measured — including yours.
• Data: source, size, skew characteristics; for generated data, the generator, its parameters, and its seed.
• Workload: query mix, arrival pattern, client counts, warm-up protocol, and run duration.
• Measurement: what was timed, from where, and what was excluded.

Variance is a systems problem, not a seed problem

ML papers randomize over seeds; systems papers fight nondeterminism from caches, compaction timing, JIT warm-up, thermal throttling, and noisy neighbors. The floor:

Practice	Rule of thumb
Repetitions	≥3-5 runs per point; state the count
Reported statistic	Median or mean — say which; show spread when curves are close
Cache state	Declare warm or cold, and how you got there
Cloud runs	Same instance placement across systems; note the epoch
Background work	Disable or document (compaction, checkpoints, GC)

A speedup smaller than the run-to-run spread is not a result; either tighten the measurement or drop the claim.

Competitor fairness ledger

Reviewers here often built the systems you compare against. For each baseline record: version or commit, configuration changes from defaults, tuning effort spent, and any feature disabled — then disclose that ledger in the paper. An untuned competitor found by its author on the program committee is a one-review rejection.

Traceability from figure to raw data

paper figure N
  <- plots/make_fig_N.py
  <- results/expN/*.csv        (raw, one file per run)
  <- run.sh expN --config configs/expN.yaml
  <- git tag paper-vN + Dockerfile digest

Build this chain during the project, not after acceptance. It is what makes the revision window survivable — a reviewer-requested variation becomes a config edit instead of archaeology — and it is exactly what the pVLDB Reproducibility Committee will walk if you enter the evaluation.

The repro-honesty paragraph

State in the paper what is not reproducible and why: proprietary traces, production-only scale, licensed competitors. PVLDB's culture (availability badges, mandatory EA&B evaluation) rewards declared limits and punishes discovered ones. One honest paragraph outperforms a broken promise of full reproducibility.

Output format

[Disclosure floor] met / gaps (hardware/software/data/workload/measurement)
[Variance handling] reps, statistic, spread shown — weak points
[Competitor ledger] complete / untuned or unpinned baselines listed
[Trace chain] figure->script->raw->tag intact / broken links
[Declared limits] <what is stated as non-reproducible and why>
[Highest-risk claim] <claim whose evidence would not survive a rerun>