| name | acmmm-reproducibility |
| description | Use when strengthening the reproducibility of an ACM MM (ACM Multimedia) paper or preparing for the ACM MM Reproducibility track and ACM artifact badging — capturing environments, media/data access, seeds, and multimodal pipelines so an independent reviewer can rebuild results and reach Artifacts Evaluated or Results Reproduced badges. |
ACM MM Reproducibility
Use this to make an ACM Multimedia result rebuildable — both for main-track credibility and
for the dedicated Reproducibility track, which routes artifacts through ACM's
badging pipeline. Multimedia adds a wrinkle: the data is often video, audio, or interactive
media, and "run the code" is not enough if a reviewer cannot obtain or render the media.
What reproducibility means here
ACM's artifact model distinguishes availability, evaluation, and reproduction. Map your goal
to the badge you are actually pursuing:
| Badge (ACM terminology) | What it asserts | What you must ship |
|---|
| Artifacts Available | The artifact is publicly, permanently retrievable | A DOI/archived repository with the code and media pointers |
| Artifacts Evaluated (Functional/Reusable) | Reviewers ran it and it works / is reusable | Build + run instructions, environment, documentation |
| Results Reproduced | An independent team reproduced the paper's results | A pipeline that regenerates the reported numbers/media |
Confirm the exact badge set offered for the current cycle on the Reproducibility-track call;
ACM's badge names and criteria evolve.
The multimodal reproducibility ledger
Keep a single record that ties each reported result to the code, data, and config that
produced it:
result: Table 2, row "full model"
code commit: <hash>
config: configs/full.yaml
data: <dataset name + version + anonymous mirror for review>
media preprocessing: <fps, sample rate, caption source>
seed(s): <list>
hardware: <GPU/CPU, hours>
expected output: results/table2_full.json
Media and data access
- Provide an anonymous, working path to the data during double-blind review — a mirror
that a reviewer can actually download, not a placeholder.
- State the license and any consent/usage terms; user-generated media often cannot be
redistributed, so document how a reviewer obtains it.
- Pin preprocessing: frame rate, resampling, transcription source, and alignment — small
differences here silently break multimodal results.
Determinism where it is achievable
- Fix and log seeds; note where nondeterminism is irreducible (e.g., some GPU kernels) and
report variance instead of pretending to bit-exactness.
- Version the environment (container or lockfile) and record hardware, since media models are
often memory- and throughput-sensitive.
Reproducibility-track readiness pass
- The Reproducibility and Open Source tracks are single-blind, so the artifact carries
its real identity — but the main-track review artifact must still be anonymous.
- Package for a stranger: a reviewer with your README and nothing else should build, run, and
hit an expected-output check within a bounded time.
- Include a smoke check (see
../../resources/code/README.md)
that verifies structure and media rendering before you submit.
Where multimodal pipelines silently break
Multimedia reproduction fails in places pure-code reproduction does not:
- Codec and container drift — a video re-encoded with a different codec changes pixel values
and breaks frame-exact results; pin the decode path.
- Sample-rate and resampling — audio resampled by a different library shifts features; record
the exact resampler and rate.
- Caption/transcript source — if captions come from an ASR system or a platform, name the
version; a different transcript is a different input.
- Frame sampling — "every k-th frame" depends on the container's frame rate; state fps and
the sampling rule.
A reproduction package that omits these looks complete but regenerates different numbers, which
is worse than an honest gap.
Anonymous review vs. public artifact
The review artifact and the release artifact have different rules, and conflating them causes
anonymity leaks or dead links:
- During review (double-blind tracks): anonymous repository, anonymous data mirror, no author
names in code comments, media metadata, or commit history.
- At release (camera-ready): the public, de-anonymized repository with a permanent archive
(DOI), the license, and the final media — replacing, not merely supplementing, the anonymous
mirror.
Output format
[Badge target] Available / Evaluated / Results Reproduced
[Ledger] complete / gaps: <which results lack a trace>
[Data access] anonymous + licensed / broken or unlicensed
[Media preprocessing] pinned / underspecified
[Determinism] seeds+env logged / gaps
[Track blinding] correct for chosen track / mismatch
[Top fixes] <ordered>
