asplos-reproducibility

name	asplos-reproducibility
description	Use when hardening an ASPLOS paper's results for independent repetition — pinning simulator versions and configs, recording kernel/firmware/BIOS state, packaging FPGA bitstreams and RTL, documenting hardware dependencies an evaluator may lack, and writing availability statements that match what the ACM badges will later require.

ASPLOS Reproducibility

Systems results decay fast: a kernel update, a microcode revision, or a silently changed simulator default can move numbers by more than the paper's claimed margin. Reproducibility work at ASPLOS is therefore state capture — recording the full machine, model, and toolchain state behind every figure — done while the experiments run, not reconstructed at camera-ready time. It also front-loads artifact evaluation: the badge criteria (asplos-artifact-evaluation) are exactly a demand that this state capture exists and works.

The state ledger

Maintain one ledger row per experimental platform, committed alongside results:

Layer	Capture	Why it moves numbers
Silicon	CPU model + stepping, memory config/topology, device (e.g. CXL expander) firmware	Steppings differ in errata and prefetch behavior
Firmware/BIOS	Microcode revision; SMT, turbo, prefetcher, C-state, NUMA settings	Any one knob can swamp a 10% effect
OS	Kernel version + full config, relevant sysctls, mitigations state	Speculation mitigations alone shift syscall-heavy results
Toolchain	Compiler + flags, libraries, runtime versions	-O level and allocator choice are classic silent variables
Simulator	Exact commit, all config files, region/checkpoint method, warm-up length	Defaults change across releases without notice
FPGA	Board, toolchain version, constraints, bitstream hash, achieved clock	Re-synthesis at a different clock is a different experiment
Workloads	Suite versions, input sets, trace provenance and preprocessing	"SPEC" without input class is unrepeatable
Randomness	Seeds for any stochastic component + run counts	Needed for the dispersion numbers to mean anything

Scripted capture beats remembered capture

Run at the start of every measurement session; store output next to the data:

#!/bin/sh
# state-capture.sh — commit this file and its output with each result set
uname -a; cat /proc/cmdline
grep -m1 'model name' /proc/cpuinfo; grep microcode /proc/cpuinfo | sort -u
cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u
cat /sys/devices/system/cpu/smt/control 2>/dev/null
numactl --hardware 2>/dev/null | head -5
cc --version | head -1
git -C "$SIM_DIR" rev-parse HEAD 2>/dev/null   # simulator commit
sha256sum "$BITSTREAM" 2>/dev/null              # FPGA bitstream identity

The hardware-access problem, named honestly

ASPLOS artifacts often need hardware an independent evaluator will not have. The honest pattern is a three-tier availability statement drafted at submission time:

1. Repeatable anywhere: simulator experiments and analysis scripts — full configs and one command per figure.
2. Repeatable with named hardware: the exact platform requirements (board, expander, CPU family), plus what to expect if the evaluator's part differs.
3. Not independently repeatable: results on lab-only or pre-production hardware — say so, and provide either supervised access, raw logs with the analysis pipeline, or a scaled-down proxy. Silence here reads as concealment; a stated limitation reads as engineering.

Claim-preservation, not number-worship

State which conclusions should survive environmental drift and which are environment-specific: "the ordering of policies is stable across kernels 6.6-6.9; absolute runtimes are not." This single sentence pattern prevents the most common failed-reproduction dispute — an evaluator matching your ordering but not your absolute numbers and calling it a failure.

Timing across the ASPLOS cycle

• Before September 9: ledger current; capture script in the repo; availability tiers drafted (they inform the paper's own text).
• Response window: the ledger is your defense when a reviewer doubts a number — you can state the exact conditions instead of hand-waving.
• Major Revision: re-run under the captured original state where possible; where the environment has drifted, disclose the drift in the change note.
• After acceptance: the ledger becomes the Artifact Appendix's dependency section nearly verbatim; AE calendars for 2027 were 待核实 at pack-check time, so confirm dates when notified.

One command per figure

The internal gold standard that makes everything downstream cheap: every figure and table in the paper regenerates from a single committed command that reads raw results and emits the exact plot. It catches stale-figure bugs before submission, turns response-window questions into lookups, and becomes the Reproducible-badge run script with a rename. Institute it at the first result, when it costs minutes — retrofitting it at camera-ready costs days.

Trace and dataset provenance

Workload inputs decay independently of code. For each trace or dataset, record origin (public suite version, generated-by script + seed, or production source), preprocessing steps as scripts rather than prose, and a checksum of the exact bytes used. Production traces that cannot be released need a characterization (rate, skew, working-set curves) plus a matched synthetic generator committed to the repo — this is also the anonymity-safe form for submission, since a raw trace can identify its owner.

When numbers drift between submission and revision

The Major Revision window arrives months after the original runs, and environments drift. Protocol:

1. Re-run a sentinel subset (three representative experiments) under the captured original state before starting revision work; if the sentinels reproduce, extend confidently.
2. If they do not, bisect the ledger — kernel, microcode, simulator commit — until the moved variable is found; the ledger exists for exactly this moment.
3. Disclose in the change note which results were re-collected and under what changed conditions, and re-state the claim-preservation sentence for the new environment. Silent regeneration of all numbers invites a reviewer to ask which version was real.

Output format

[Ledger coverage] platforms with complete rows: N/N · gaps listed
[Capture automation] script committed + outputs stored with data: Y/N
[Simulator pinning] commit + configs + region method + warm-up recorded: Y/N
[Availability tiers] anywhere / named-hardware / not-repeatable — each populated
[Claim preservation] drift-stable vs environment-specific conclusions stated: Y/N
[Badge readiness] which badges the current package could already earn