mobisys-reproducibility

name	mobisys-reproducibility
description	Use when strengthening MobiSys reproducibility evidence — capturing device, SoC, OS build, framework and model versions, power-instrument setup, seeds, and thermal conditions so an on-device result survives a different phone, deciding what data and firmware can legally ship, and keeping the paper consistent with the artifact for the ACM badge pipeline.

MobiSys Reproducibility

Use this before submission and again before the artifact-evaluation deadline. On-device results are fragile across hardware and framework versions, so reproducibility at MobiSys is mostly about provenance: recording enough of the device and measurement context that someone on a different phone can rebuild the result.

Evidence map

• Map each latency, energy, throughput, and accuracy claim to a verifiable location in the paper, appendix, or artifact package.
• For each on-device result, record device model, SoC, OS build, framework/runtime version, model checkpoint hash, power source, ambient temperature, and thermal state at run start.
• Record the power-instrument setup and the energy boundary so an energy figure can be re-derived rather than trusted.
• For stochastic results, report seeds, repeated runs, and the spread; a single number from a single phone is not reproducible evidence.
• Explain missing data, firmware, or hardware honestly, and describe how a reader could reproduce the result in principle on a different device.
• Keep the artifact consistent with the manuscript; a number in the PDF that the artifact cannot regenerate is a review-risk multiplier and an AEC failure.

What survives a different device

Provenance item	Turnkey answer	Common failure caught
Device + SoC + OS build	Named per experiment, in a table	"on a mobile device" with no model
Framework / runtime version	Pinned with hashes	Result drifts silently across a minor version
Energy instrument + boundary	Instrument, sampling rate, envelope stated	"efficient" with no measured joules
Thermal state	Ambient + steady-state trace	30-second benchmark hides throttling
Model / cache version	Checkpoint hash, cache size	Accuracy irreproducible from a different weight file

Marking a device result "reproducible" without its provenance is a recognizable MobiSys red flag, because the AEC and reviewers cross-check the artifact against the PDF and read a gap as carelessness about the rest of the paper.

Vignette: an on-device inference paper

Consider a runtime that keeps CNN latency stable under load, validated on four phones. Its reproducibility spine: the four device models and OS builds, the runtime version and model checkpoint hashes, the power-monitor model and sampling rate, the ambient temperature and the steady-state thermal traces, the workload trace, the seeds and run counts — plus one honest sentence about the device it could not test and why the result may differ there.

Degrees of reproducibility

• Turnkey: one script regenerates each figure from logged runs on the target device.
• Scripted: scripts exist but need the specific hardware or manual device setup.
• Descriptive: prose detailed enough that a competent reader could rebuild the pipeline on a comparable device.

For MobiSys, the on-device measurement path should be as turnkey as the hardware allows, and a hardware-optional downscaled or emulator path should let an evaluator without your exact device reach at least Functional (mobisys-artifact-evaluation). State the achieved level honestly rather than overpromising turnkey behavior that fails on a different phone.

Legal and privacy provenance

• Decide early what traces, firmware, and datasets can ship; on-device data often carries user, sensor, or vendor-firmware restrictions.
• Strip identifying content from screenshots, logs, and demo media before release.
• If data cannot ship, provide a synthetic or public substitute and name which results it cannot reproduce.

Output format

[Claim inventory] <claim -> evidence location>
[Provenance status] complete / partial / missing (device/OS/runtime/energy/thermal)
[Reproducibility gaps] <device pinning / seeds / energy boundary / model version>
[Paper fixes] <must appear in main PDF>
[Artifact fixes] <appendix or artifact additions, incl. hardware-optional path>