| name | uist-reproducibility |
| description | Use when making a UIST paper's results replicable — reporting implementation parameters and measurement protocols so a lab could rebuild the system, specifying hardware down to parts and calibration, logging technical evaluations deterministically, and writing honest availability statements for interface systems. |
UIST Reproducibility
Reproducibility at UIST means something different from rerunning a training script:
the question is whether a competent lab could rebuild the artifact and reproduce
its measured behavior. That decomposes into three ledgers — build, measurement,
and study — and most UIST papers under-specify a different one than they think.
UIST posts no reproducibility checklist (none found for 2026 — 待核实), so this
discipline is self-imposed and reviewer-enforced.
The build ledger: could they make one?
Everything the system's behavior depends on, pinned:
- Software: exact framework and driver versions, OS, lockfiles; for anything
learned, model checkpoints and training data provenance.
- Hardware: part numbers (not "an IMU" but which IMU), mechanical tolerances
that matter, firmware version, and the calibration routine with expected outputs.
- Environment: the physical conditions the system assumes — lighting range,
acoustic environment, mounting geometry, surface materials.
- Magic numbers: every threshold, filter coefficient, debounce window, and
gain, with how each was set (tuned by hand? on which data?). These constants are
where re-implementations actually fail.
The measurement ledger: could they get your numbers?
A latency or accuracy figure is reproducible only with its protocol:
| Reported number | Must be pinned |
|---|
| Latency | Measurement boundary (sensor-to-photon? software-only?), instrument, event count, load conditions |
| Recognition accuracy | Dataset splits, per-user vs pooled, session separation, chance level |
| Tracking error | Ground-truth apparatus and its own accuracy, spatial sampling grid |
| Throughput / bitrate | Task, phrase set or corpus, session and rest structure |
| Power / weight / cost | Configuration measured, currency and date for cost |
Automate the protocol: a measurement harness checked into the supplement converts
"trust me" into "run this" (see uist-artifact-evaluation for packaging).
python analyze_latency.py logs/latency_10k.jsonl --out tables/table1.csv
python analyze_accuracy.py logs/study/ --split per-user --seed 17 --out tables/table2.csv
diff -u tables/table1.csv paper_tables/table1.csv
Log at the event level with timestamps and raw sensor values, not just computed
outcomes — future you, rebuttal you (see uist-author-response), and replicating
labs all consume the same logs.
The study ledger: could they rerun the human part?
For any user evaluation: full task instructions and stimuli, counterbalancing
scheme, practice/rest structure, apparatus placement (photograph it),
inclusion/exclusion criteria, compensation, and the analysis scripts from raw logs
to reported statistics. Share instruments even when raw human data cannot leave the
IRB envelope — protocol transparency and data availability are separable, and
saying so precisely is the honest move.
The availability statement
Write one even though UIST does not require a template, and make it specific:
GOOD: "Firmware, PCB design files, BOM, and the measurement harness are at
<archive-DOI> (tag: as-published). The gesture corpus (14 of 16
participants consented to release) is included; per-participant raw
video is withheld under IRB #—. The two commercial tracking SDKs
required are named in BUILD.md with tested versions."
BAD: "Code available upon reasonable request."
Papers with learned components inherit ML reporting norms too — seeds, training
configs, compute — and can borrow the shared kit's checklists (see
../../resources/code/README.md).
Learned components inherit ML norms
When the system embeds recognition or generation models, the ML reporting
conventions stack on top of the systems ledgers:
- Seeds, splits, and training configs pinned; per-user vs pooled evaluation made
explicit (interaction data is brutally user-dependent, and pooled splits
inflate accuracy).
- For third-party or hosted models: exact model identifiers and versions, the
full prompt set, decoding parameters, and dated transcripts — hosted models
drift, so the logged behavior is the only permanent record of what reviewers
saw.
- Report the interactive costs alongside accuracy: per-inference latency on the
deployment hardware, and per-session cost for metered APIs; a technique that is
reproducible but unaffordable to run is only half-replicable.
- The shared kit's checklists cover this lane; the adapter in
resources/code/README.md scopes what it can and cannot check.
The one-page REPRO.md
Compress the three ledgers into a single file at the archive root:
# Reproducing <SystemName> (paper §6-7)
## Rebuild parts: BOM.csv · firmware: v2.3 · calibration: docs/calib.md
## Environment tested: indoor 200-800 lux, 18-26°C · assumes: mounted per Fig 4
## Constants thresholds in config.yaml — tuned on pilot data (n=4), §5.2
## Remeasure eval/rerun.sh regenerates Tables 1-2 from logs/ (or raw capture)
## Study instruments/ · counterbalancing: latin square, §7.1 · IRB #—
## Known drift accuracy drops outdoors (§9); p95 latency sensitive to BLE stack
The "known drift" line is the credibility multiplier: it tells replicators you
know where the envelope ends before they find out.
Replication drift and its uses
When a rebuilt system misses the paper's numbers, the causes rank: unstated
environmental assumptions, hand-tuned constants, part substitutions, then genuine
bugs. Pre-empt the first two by stating the envelope and the tuning story in the
paper body — it costs three sentences and buys the paper years of credibility (and
Lasting Impact eligibility runs on decade-scale credibility).
Output format
[Build ledger] pinned / gaps: <software · hardware · environment · constants>
[Measurement ledger] protocols pinned for <k>/<n> reported numbers
[Study ledger] instruments · counterbalancing · analysis scripts — present?
[Availability statement] drafted? honest about withholdings?
[Top drift risk] <the unstated assumption most likely to break replication>
