| name | isca-reproducibility |
| description | Use when making an ISCA paper's results regenerable — pinning simulator versions and local patches, archiving per-figure configuration manifests, recording workload provenance and sampling seeds, quantifying run-to-run variation on real hardware, and keeping the environment resurrectable through the February window. |
ISCA Reproducibility
In architecture, "reproducible" means someone else — or you, three months later,
mid-rebuttal — can regenerate every reported number from recorded state. Because
most ISCA numbers come out of simulators, reproducibility here is largely
configuration archaeology: the result is a function of tool commit, local
patches, model parameters, workload build, region selection, and warm-up policy,
and losing any one of those breaks the chain. The venue reinforces this culture
with post-acceptance artifact evaluation under ACM badging
(isca-artifact-evaluation); this skill covers the discipline that must exist
before any AE form is filled.
The result chain, and what to pin at each link
| Link | What drifts silently | Pin it by |
|---|
| Simulator | Version-to-version behavior changes; forgotten local edits | Exact commit hash + git diff of local patches archived with results |
| Machine model | Config files edited during exploration | One immutable config per experiment family; configs referenced by hash |
| Workloads | Compiler/flags/inputs change binaries | Archive binaries or lockfile the build; record input sets by checksum |
| Regions & warm-up | Re-generated sampling points differ | Store the region/checkpoint files themselves, plus the generator seed |
| Post-processing | "Quick" notebook edits change aggregation | Scripted stats path from raw output to figure, in the repo |
| Real-hardware runs | Frequency scaling, thermal state, background load | Record governor, SMT/turbo state, kernel; report dispersion over trials |
One manifest per published number
Adopt the rule that every figure and table in the paper has a manifest and a
regeneration command. This is the same manifest format isca-experiments
specifies for methodology writing — one artifact serves both purposes.
results/
f07-headline/
manifest.ini
regen.sh
raw/
derived/f07.csv
f07.pdf
git tag isca27-submitted && \
sha256sum results/*/f*.pdf paper/fig/*.pdf | sort | uniq -c -w64 | \
awk '$1!=2 {print "FIGURE MISMATCH:", $0}'
The freeze ritual catches the classic disaster: a figure in the PDF produced by
a config that no longer exists because exploration continued after the plot was
made.
Nondeterminism gets measured, not ignored
- Deterministic simulators: verify determinism once (same commit + config +
workload → bit-identical stats) and record that check; if a threading mode
breaks it, either use the deterministic mode for reported numbers or report
dispersion.
- Real hardware: never a single trial. Report median and spread across ≥5
runs, with the machine-state record (governor, turbo, SMT, kernel, isolation
measures). Reviewers increasingly ask; artifact evaluators always do.
- Sampled simulation: the sampling procedure and seed are part of the
result. Different SimPoint runs are different experiments — archive the chosen
regions, don't regenerate them.
Paper-side reporting
The paper must let a skeptical reader reconstruct the setup without the
artifact: a full configuration table (structures, sizes, latencies, DRAM
timing), the workload list with inputs and build flags summarized, the region/
warm-up policy, and a variability statement wherever hardware was measured. Under
double-blind rules the repository link, if given, must be fully anonymized
(verified 2026 rule — see isca-submission); the common pattern is an
anonymized-mirror link at submission, replaced by the real archival link in the
camera-ready.
Resurrectability: the February requirement
The 2026 cycle's rebuttal/revision window (Feb 16 - Mar 6) arrived three months
after submission. Teams whose environment had rotted — simulator tree no longer
building, cluster images recycled, workload binaries lost — entered the window
unable to run the experiments that would have saved the paper. Protocol:
- At submission: container or environment image built and stored;
regen.sh
for at least the headline figure verified from the image, not from a dev
machine.
- Window-open minus one week (early February): resurrection drill — boot the
image, regenerate one figure end to end, confirm hash match.
- Keep one team member's environment untouched between November and March; do
not upgrade the shared toolchain mid-wait.
Habits that make all of this cheap
- Results directories are append-only; a changed config is a new experiment
ID, never an edit in place.
- The plotting path takes experiment IDs, not file paths typed by hand.
- A
METHODS.md in the repo grows in real time — every methodological choice
(why these regions, why this warm-up, why this DRAM model) written down when
made, because November-you will not remember July-you's reasoning.
- Weekly:
regen.sh for the current headline figure runs green in CI or by
hand. Regeneration that only works on deadline eve doesn't work.
Pre-submission reproducibility gate
Where each practice pays off later
| Practice | Pays off at... |
|---|
Per-figure manifests + regen.sh | Methodology section writing, rebuttal experiments, AE claims table |
| Submission-tag freeze ritual | Camera-ready number verification, artifact snapshot selection |
| Environment image + drill | The February window's first 48 hours |
| Hardware-state records | Reviewer variance questions, Functional-badge documentation |
METHODS.md running log | Every "why did we choose X" question from reviewers and evaluators |
Venue facts (AE program, badging, double-blind link rule) verified 2026-07-08 in
../../resources/official-source-map.md; the engineering protocol above is
community best practice, applicable regardless of cycle.
