| name | isca-artifact-evaluation |
| description | Use when preparing an accepted ISCA paper's artifact for evaluation under the ACM Review and Badging policy โ scoping which results are reproducible within evaluator budgets, packaging simulator-heavy workflows others can run, writing the evaluator-facing appendix, and earning badges that print on the paper. |
ISCA Artifact Evaluation
The verified 2026 arrangement: authors of accepted papers could voluntarily
submit artifacts, assessed under the ACM Artifact Review and Badging policy,
with earned badges printed on the paper and attached as ACM Digital Library
metadata โ and with no effect on the acceptance decision itself. The conference
framed AE as a recent, deliberate import into the architecture community
(following ASPLOS from 2020 and MICRO from 2021). Voluntary does not mean
optional in practice: badges are becoming the community's default trust signal,
and an architecture paper without them increasingly gets asked why.
Decide scope first: which claims does the artifact defend?
Architecture artifacts fail most often from over-promising. Before packaging,
split the paper's results three ways and say so in the artifact documentation:
| Tier | Contents | Evaluator experience |
|---|
| Reproduce | Headline figure(s) + key ablation, runnable on evaluator-accessible machines within hours | Runs it, matches within stated tolerance |
| Regenerate-with-resources | Full sweeps needing cluster-days; scripts and configs complete, cost stated honestly | Inspects, spot-runs a slice |
| Inspect-only | Anything needing hardware evaluators won't have (FPGA boards, silicon, internal traces) | Reads code + recorded raw outputs; verifies the pipeline from raw to figure |
Declaring tier boundaries up front converts "couldn't run everything" from a
failure into the documented plan. Never gate the Reproduce tier on proprietary
inputs โ substitute an open workload subset and show it preserves the trend.
Simulator-heavy artifacts: the specific frictions
- Wall-clock honesty. Cycle-level simulation of full suites is
CPU-months; evaluators have days. Ship reduced-input or single-region variants
of the headline experiment, with the full-run manifests alongside. State
expected runtimes per step, measured, not guessed.
- Build fragility. Simulator trees with local patches break on new
toolchains. Ship a container image with everything pre-built, and the
from-source path for evaluators who distrust images.
- Tolerance statement. Define "match": bit-identical stats for
deterministic runs; a stated percentage band where sampling or threading
varies. An evaluator staring at 33.7% vs the paper's 34% needs the band in
writing.
- Third-party licenses. SPEC-class suites cannot be redistributed; ship
build recipes plus checksums, and include one freely redistributable workload
so the pipeline is exercisable end to end regardless.
The package an evaluator can drive blind
artifact/
README.md # claims-to-experiments map; runtimes; tolerance bands
INSTALL.md # container route + source route, both tested cold
run_all.sh # Reproduce tier end to end, one command
experiments/
f07-headline/ # regen.sh + manifest.ini per figure (from
f09-ablation/ # isca-reproducibility โ same manifests, reused)
expected/ # our raw outputs + final CSVs, for diffing
workloads/ # free subset included; licensed suite: recipe + checksums
LICENSE
The README's first section is a claims table: paper claim โ figure โ command
โ expected output โ tolerance. Evaluators grade against exactly this table;
making them reverse-engineer the mapping from the paper costs goodwill and,
frequently, a badge.
docker load < artifact-image.tar.gz
docker run --rm -it artifact:v1 bash -lc './run_all.sh --tier reproduce'
diff <(csvcut -c workload,ipc results/f07.csv) \
<(csvcut -c workload,ipc expected/f07.csv)
Have a group member outside the project do this rehearsal before submission;
every question they ask is a missing README paragraph.
The badge set under ACM's policy
The ACM Review and Badging framework distinguishes availability badging (the
artifact is permanently retrievable from an archival location) from functional
and results-oriented badging (the artifact works as documented; the paper's
results can be obtained with it). Practical mapping:
- Availability is cheap and unconditional: deposit the evaluated snapshot
with a DOI-issuing archive (a lab GitHub alone is not archival). Do this even
if you pursue nothing else.
- Functionality rides on documentation and the cold-start test.
- Results reproduction rides on the Reproduce tier's design: modest
runtime, clear tolerance, deterministic where possible.
Which badges ISCA 2027 requests, the AE calendar, and the committee's platform
offerings (cloud credits, provided machines) are per-edition decisions โ ๅพ
ๆ ธๅฎ
against the 2027 AE page when it exists.
Working with evaluators
Architecture AE is conventionally collaborative and iterative: evaluators hit a
wall, authors fix and resubmit within the AE window. Budget author-days for this
in the post-acceptance calendar (isca-workflow), respond within a day while
the evaluator's context is warm, and version every fix (artifact:v1.1,
changelog in the README) so the final badge attaches to an identifiable object.
Timing and division of labor
AE happens in the same weeks as camera-ready preparation and talk writing.
Assign the artifact to someone who ran the original experiments โ packaging is
recall-heavy โ and start from the submission-time snapshot that
isca-reproducibility froze in November, not from the current dev tree, which
has drifted.
Submission gate
Venue facts verified 2026-07-08 (../../resources/official-source-map.md);
badge definitions follow ACM's published badging policy โ check the version the
2027 AE chairs cite.
