| name | usenixsec-artifact-evaluation |
| description | Use when packaging artifacts for USENIX Security Symposium evaluation — the mandatory Phase-1 availability check that acceptance is conditional on, the optional Phase-2 push for Artifacts Functional and Results Reproduced badges, and building security artifacts (exploits, scanners, datasets) that evaluators can run safely. |
USENIX Security Artifact Evaluation
USENIX Security '26 split artifact evaluation into two phases with different stakes:
Phase-1 (availability) is mandatory and paper acceptance is conditional on it,
running between acceptance and the final-paper deadline; Phase-2 (functionality
and reproducibility) is optional, running after finals are due. The community
process and submission mechanics live at secartifacts.github.io/usenixsec2026/;
the venue framing is on the usenix.org Call for Artifacts page (both checked
2026-07-08, via search renderings since usenix.org direct fetch returned 403).
The three badges
| Badge | What the AEC verifies | Phase |
|---|
| Artifacts Available | The artifacts named in the Open Science appendix exist at the stated location and are retrievable | Phase-1, required of all accepted papers (or a justified omission) |
| Artifacts Functional | Core functionality works as documented: installs, runs, produces sane output | Phase-2, opt-in |
| Results Reproduced | The paper's reported results were regenerated by the AEC | Phase-2, opt-in |
Badges are issued independently — Reproduced does not require re-litigating
Available. The strategic read: Phase-1 is table stakes; Phase-2 is where a systems
or measurement paper earns durable credibility (and eligibility for artifact
recognition such as the distinguished-artifact honors given in recent years).
Phase-1: make "available" mean available
The check is simple and unforgiving — an evaluator follows the pointer in your Open
Science appendix. Failure modes are all self-inflicted:
- The submission-time anonymous mirror was never replaced by a permanent home.
- The repository exists but the named tag/commit does not, or the dataset link
requires a lab account.
- The appendix says "code and data" but the archive holds code only.
Use an archival service that mints DOIs (e.g., Zenodo) or an institutional archive
for the frozen version, plus a living repository for maintenance. If something
genuinely cannot be released — live vulnerability details under embargo, PII-laden
measurement data, malware corpora with legal constraints — the appendix must say
so and say why; justified withholding is policy-compliant, silent withholding is
not (see usenixsec-reproducibility for the decision table).
Phase-2: security artifacts have special failure modes
Evaluators are volunteers on ordinary machines and a clock. Security artifacts add
hazards other fields do not have — design these out:
- Dangerous-by-default behavior. An exploit PoC or scanner must ship pointed
at a local target (container victim, testbed config), never at the live internet.
Document the blast radius of every script.
- Environment brittleness. Kernel-version-sensitive attacks, specific GPU
drivers, SGX hardware: state exact requirements up front and provide a VM or
container image when the host matters.
- Nondeterministic attacks. Success-rate claims need the trial count and the
tolerance you expect the evaluator to observe, not a single lucky run.
- Long horizons. If full reproduction takes days (fuzzing campaigns, large
scans), provide a scaled-down kit that reproduces the trend in under an hour,
plus the full recipe.
artifact/
├── README.md # claims ↔ experiments map; time + hardware budget
├── LICENSE
├── Dockerfile # or VM image link with checksum
├── setup.sh # one-shot environment build (prints versions)
├── run_minimal.sh # <1h: reproduces Fig 5 trend on the local testbed
├── run_full.sh # complete campaign; documents multi-day runtime
├── victim/ # self-contained target — nothing external is probed
├── data/ # released datasets + SHA256SUMS; withheld items listed
└── expected/ # reference outputs with tolerance notes per claim
The README's spine should be a claims table: each major claim in the paper mapped
to the command that regenerates it, its runtime, and the expected output range.
Write for an evaluator who reads the paper once and your README three times.
Working the process
- Register the artifact by the AE deadline for your cycle (dates on the
secartifacts instructions page — per-cycle, 待核实 each time).
- During evaluation, communication runs anonymously through the AE platform;
respond fast, since the window is short and iterations are the norm.
- Fix-and-rerun is expected: evaluators report blockers, authors patch. Budget
maintainer attention for the whole window, not just the submission day.
Reverify each cycle
- Whether Phase-1 remains mandatory and acceptance-conditional in the current CFP.
- Badge definitions and AE deadlines on the current secartifacts instance.
- Size/hosting norms and whether the AEC supplies compute for special hardware.
Output format
[Phase-1] artifact home (DOI/URL) + retrievability test result; omissions justified
[Phase-2 target] none / Functional / Reproduced — with reasoning
[Hazard review] network side effects, secrets, licensing of third-party payloads
[Kit status] minimal-run time, full-run time, claims table coverage n/m
[Deadlines] AE registration + evaluation window for this cycle
