| name | mobisys-reproducibility |
| description | Use when strengthening MobiSys reproducibility evidence — capturing device, SoC, OS build, framework and model versions, power-instrument setup, seeds, and thermal conditions so an on-device result survives a different phone, deciding what data and firmware can legally ship, and keeping the paper consistent with the artifact for the ACM badge pipeline. |
MobiSys Reproducibility
Use this before submission and again before the artifact-evaluation deadline. On-device
results are fragile across hardware and framework versions, so reproducibility at MobiSys is
mostly about provenance: recording enough of the device and measurement context that
someone on a different phone can rebuild the result.
Evidence map
- Map each latency, energy, throughput, and accuracy claim to a verifiable location in the
paper, appendix, or artifact package.
- For each on-device result, record device model, SoC, OS build, framework/runtime version,
model checkpoint hash, power source, ambient temperature, and thermal state at run start.
- Record the power-instrument setup and the energy boundary so an energy figure can be
re-derived rather than trusted.
- For stochastic results, report seeds, repeated runs, and the spread; a single number from a
single phone is not reproducible evidence.
- Explain missing data, firmware, or hardware honestly, and describe how a reader could
reproduce the result in principle on a different device.
- Keep the artifact consistent with the manuscript; a number in the PDF that the artifact
cannot regenerate is a review-risk multiplier and an AEC failure.
What survives a different device
| Provenance item | Turnkey answer | Common failure caught |
|---|
| Device + SoC + OS build | Named per experiment, in a table | "on a mobile device" with no model |
| Framework / runtime version | Pinned with hashes | Result drifts silently across a minor version |
| Energy instrument + boundary | Instrument, sampling rate, envelope stated | "efficient" with no measured joules |
| Thermal state | Ambient + steady-state trace | 30-second benchmark hides throttling |
| Model / cache version | Checkpoint hash, cache size | Accuracy irreproducible from a different weight file |
Marking a device result "reproducible" without its provenance is a recognizable MobiSys red
flag, because the AEC and reviewers cross-check the artifact against the PDF and read a gap as
carelessness about the rest of the paper.
Vignette: an on-device inference paper
Consider a runtime that keeps CNN latency stable under load, validated on four phones. Its
reproducibility spine: the four device models and OS builds, the runtime version and model
checkpoint hashes, the power-monitor model and sampling rate, the ambient temperature and the
steady-state thermal traces, the workload trace, the seeds and run counts — plus one honest
sentence about the device it could not test and why the result may differ there.
Degrees of reproducibility
- Turnkey: one script regenerates each figure from logged runs on the target device.
- Scripted: scripts exist but need the specific hardware or manual device setup.
- Descriptive: prose detailed enough that a competent reader could rebuild the pipeline on
a comparable device.
For MobiSys, the on-device measurement path should be as turnkey as the hardware allows, and a
hardware-optional downscaled or emulator path should let an evaluator without your exact
device reach at least Functional (mobisys-artifact-evaluation). State the achieved level
honestly rather than overpromising turnkey behavior that fails on a different phone.
Legal and privacy provenance
- Decide early what traces, firmware, and datasets can ship; on-device data often carries
user, sensor, or vendor-firmware restrictions.
- Strip identifying content from screenshots, logs, and demo media before release.
- If data cannot ship, provide a synthetic or public substitute and name which results it
cannot reproduce.
Output format
[Claim inventory] <claim -> evidence location>
[Provenance status] complete / partial / missing (device/OS/runtime/energy/thermal)
[Reproducibility gaps] <device pinning / seeds / energy boundary / model version>
[Paper fixes] <must appear in main PDF>
[Artifact fixes] <appendix or artifact additions, incl. hardware-optional path>
