eurosys-reproducibility

name	eurosys-reproducibility
description	Use when hardening the reproducibility story of a EuroSys paper — recording hardware and software provenance for every number, taming performance variance with repeated runs and dispersion reporting, versioning workloads and traces, and writing an availability statement that survives both double-blind review and the sysartifacts AEC.

EuroSys Reproducibility

Use this while experiments are still running — reproducibility retrofitted in deadline week is transcription, not engineering. The venue context: EuroSys papers live on measured performance claims, the community runs a badge-granting artifact evaluation (sysartifacts.github.io), and SIGOPS has publicly digested five years of EuroSys AE lessons (sigops.org blog, 2025; rendered 2026-07-08). A paper whose numbers cannot be regenerated by its own authors three months later fails both review-time scrutiny and post-acceptance AE.

The provenance ledger

Keep one machine-readable record per reported number. Minimum fields:

Field	Why EuroSys reviewers care
Commit hash of system under test	"Which version got 2.1x?" is a real AE question
Baseline name + version + config	Untuned-baseline suspicion is the venue's default
Hardware: CPU/RAM/NIC/storage, topology	Systems results rarely transfer across boxes
OS/kernel, key library versions	Kernel changes move I/O and scheduler numbers
Workload/trace + generator seed	Trace provenance is checked, not assumed
Repetitions, warm-up policy	Distinguishes measurement from anecdote
Timestamp + raw-output path	Lets you rebuild any figure from raw logs

Variance discipline

Single-run numbers are the most common silent reproducibility failure in systems evaluation:

• Repeat every headline measurement enough times to see its spread; report median plus an explicit dispersion measure (stdev, IQR, or min–max), and say in the caption which one it is.
• Isolate noise sources you control: pin frequencies, disable turbo where it distorts comparisons, note co-located load, randomize run order across systems so drift does not favor yours.
• Tail metrics (p99 and beyond) need far more samples than means; state the sample count whenever a tail latency is claimed.
• When a difference is within run-to-run spread, say so — EuroSys reviewers reward calibrated claims over uniform victory narratives.

Automation floor

The practical bar: any figure regenerates from raw data with one command.

# Layout that keeps figures honest
experiments/
  fig7_throughput/
    run.sh        # executes the sweep, writes results/*.csv with metadata header
    plot.py       # reads results/, emits fig7.pdf — no hand-edited numbers
    results/      # raw outputs, never overwritten, one dir per run timestamp
make fig7         # the only path by which fig7.pdf ever changes

If a plot was ever touched manually, its provenance is broken and the AEC will find the discrepancy before you do.

Availability statement, two audiences

• Review time (double-blind): describe what exists — "an anonymized repository containing the system, workload generator, and run scripts accompanies the submission" — without leaking the lab's identity through URLs, paths, or commit authors.
• Camera-ready / AE time: replace with the DOI-backed archive and the badge set being sought. Restricted traces (production data, partner NDAs) need an honest fallback: a synthetic generator calibrated to the trace's published statistics, with the calibration method described.

Restricted evidence, stated honestly

Some EuroSys evidence legitimately cannot ship — production traces under NDA, partner clusters, proprietary workloads. The honest pattern:

• Name the restriction and its scope precisely ("the ingestion trace from operator X cannot be released; its summary statistics are in Table 3").
• Ship a calibrated synthetic substitute and the calibration procedure, so external readers can approximate the regime.
• Keep at least one headline result on fully public inputs; a paper whose every number depends on unreleasable data asks reviewers for faith the venue does not trade in.
• Never let the availability paragraph imply more openness than the AEC will find; the badge process makes overstatement visible in print.

Pre-deadline reproducibility drill

One week before the paper gate, run the drill on a machine that never ran the experiments:

1. Clean checkout, environment build from the lockfile alone — record every undocumented step it turns out to need.
2. Regenerate two figures end to end: one cheap, one expensive.
3. Diff regenerated numbers against the draft's numbers; investigate any drift beyond the reported dispersion.
4. Fix the documentation, not just the outcome — the drill's product is the README the AEC will eventually read.

Quick self-test

1. Can a new student regenerate Figure 7 from a clean checkout in one command?
2. Does every table cell trace to a raw log file with hardware metadata?
3. Are repetition counts and dispersion visible for every performance claim?
4. Would the availability paragraph survive both anonymity and the AEC?

Output format

[Repro grade] regenerable / scripted-with-gaps / manual
[Ledger coverage] <numbers with full provenance / total reported numbers>
[Variance findings] <single-run claims, missing dispersion, undersampled tails>
[Workload provenance] <traces and generators with version + seed status>
[Availability draft] <review-time text and camera-ready plan>