| name | nsdi-reproducibility |
| description | Use when making an NSDI paper's results reconstructible — capturing testbed topology, trace provenance, and configuration while experiments run, planning which datasets can ship publicly, and keeping the paper and artifact from drifting apart so badge evaluation and the Community Award stay reachable. |
NSDI Reproducibility
A networked-systems result is a function of topology, traffic, timing, and code — and
three of those four are absent from the PDF unless deliberately recorded. NSDI
rewards the discipline institutionally: artifact badges after acceptance and a
Community Award for the best paper whose code and/or dataset is public by the
final-papers deadline. But the work happens during the experiments, not after the
decision email.
The four provenance ledgers
Keep each as a versioned file next to the results it explains:
| Ledger | Contents | Loss mode it prevents |
|---|
| Topology | node specs, NIC/switch models, link speeds, RTT matrix, kernel + NIC settings | "worked on our cluster," unreproducible knee points |
| Traffic | trace source + collection context, scaling/anonymization transforms, synthetic-generator parameters + seeds | headline numbers tied to data nobody can regenerate |
| Configuration | every config diff from defaults, per system and per baseline | untuned-baseline accusations you cannot rebut |
| Run | per-experiment: commit hash, config snapshot, fault schedule, raw-log location, analysis-script hash | figures that cannot be regenerated at revision time |
The run ledger has a second life NSDI makes unusually likely: a one-shot revision
letter can demand new measurements on the same setup, 3-4 months after submission,
at a subsequent deadline. Teams with ledgers rerun in days; teams without them
rebuild the testbed from memory.
Determinism is different here
Unlike ML reproducibility, networked-systems results are legitimately
non-deterministic — background load, timer jitter, and cross-traffic vary. The honest
posture is not "identical numbers" but characterized variance:
- Fix what can be fixed: seeds for generators and fault schedules, pinned software
versions, isolated or documented-shared testbeds.
- Measure what cannot: repeat headline experiments across days/trace slices; publish
the spread with the artifact so an evaluator knows a 7% delta is noise, not
failure.
- Timestamp everything; time-of-day and neighbor effects are real explanations, but
only if recorded.
What can actually ship?
Decide per data item before the camera-ready crunch, because the answer shapes the
paper's claims:
- Public now — code, synthetic generators, testbed configs, analysis scripts.
Default to shipping.
- Public after transformation — production-derived traces that can be
anonymized/aggregated. Document the transform and what properties it preserves;
state in the paper that released traces differ from raw ones.
- Never public — proprietary traces, customer data. Plan the fallback: a
synthetic workload with matched characteristics, shipped alongside a fidelity
note. Do not let a never-public trace be the only support for a headline claim
if a shippable proxy can corroborate it.
Anonymize the artifact itself for review-time supplements (repo owners, hostnames,
paths, company strings in configs); de-anonymize only at final-paper time.
Wiring it into the repo
experiments/
<exp-id>/run.sh
<exp-id>/provenance.json
figures/
Makefile
paper/
claims.md
The claims.md cross-map is the cheapest anti-drift device: at submission freeze,
walk it once; any claim whose exp-id is stale gets rerun or reworded. The shared
smoke-checker
(../../resources/code/README.md) covers package
hygiene but not topology/trace fidelity — those checks are yours.
Blind now, open later
Reproducibility material crosses the double-blind boundary twice, and each crossing
has a checklist:
- At submission (if any package is uploaded as auxiliary material): scrub
repository owners, commit-author emails, hostnames that embed the institution,
and license files naming the organization; keep the scrubbed and true versions as
separate branches so de-anonymization is a merge, not a rewrite.
- At final papers: flip to the public, attributed release — DOI'd archive,
real repository, institutional acknowledgment — dated before the final-papers
deadline if the Community Award matters (
nsdi-camera-ready).
Cheap wins ordered by payoff
provenance.json per experiment, written by the runner script (an hour to
automate).- Figure Makefile with raw-log inputs (removes an entire class of revision pain).
- Baseline config diffs recorded (rebuts the most common reviewer attack).
- Variance characterization for the two headline results.
- Early legal check on trace release — licensing takes longer than deadline gaps.
Signs the discipline is slipping
- A figure in the draft that nobody can regenerate this week.
- "The good run" language — selecting runs by outcome rather than by protocol.
- Baseline numbers copied forward from an earlier paper draft instead of re-measured
on the current topology.
- Testbed changes (kernel upgrade, NIC swap) mid-campaign without a ledger entry
splitting before/after results.
Any one of these, found at submission freeze, costs days; found by an artifact
evaluator or a revision reviewer, it costs the result's credibility.
Output format
[Ledger status] topology / traffic / configuration / run — each present/partial/absent
[Variance] headline results with characterized spread? which lack repeats?
[Shippability] data items -> public / transformable / never (fallback named)
[Drift check] claims.md walked? stale claim list
[Award posture] on track for public code+data by final-papers deadline?
[Next actions] ordered by payoff-per-hour
