acmmm-experiments

name	acmmm-experiments
description	Use when designing or auditing the experiments of an ACM MM (ACM Multimedia) paper — matched baselines per modality, ablations that isolate the cross-modal fusion, user studies or QoE measurement where the claim is subjective, dataset and media licensing, and honest compute reporting, so evidence supports a multimedia claim.

ACM MM Experiments

Use this to make an ACM Multimedia paper's evidence match its claim. The reviewer's implicit questions are: does it work, does the cross-modal part cause the gain, when does it fail, and — if the target is perceptual — do people actually prefer it.

The four questions and how to answer them

Question	Evidence that answers it
Does it work?	The headline metric on a recognized benchmark, against strong, matched baselines
Does the fusion cause the gain?	A leave-one-modality-out / component ablation isolating the cross-modal term
When does it fail?	Failure cases per modality (e.g., noisy audio, missing captions) shown honestly
Do people prefer it?	A user study with reported N, protocol, and inter-rater agreement — for subjective claims

The second row is what separates an ACM MM experiment section from a single-modality one: if removing a modality does not move the result, the paper is not really cross-modal.

Matched baselines

• Compare against the strongest existing method, re-run under your data and preprocessing where feasible, not a weakened reimplementation.
• Include a late-fusion / naive-concatenation baseline so the reader sees what the fancy fusion buys over the obvious one.
• Hold everything but the mechanism fixed: same backbone, same features, same training budget, so the delta is attributable.

Ablations that isolate the cross-modal claim

Full model .................... reference
- audio stream ................ tests whether audio carries signal
- text/caption stream ......... tests whether language carries signal
- alignment / fusion module ... replaced by concatenation: tests the MECHANISM
- synchronization assumption .. shuffled timing: tests whether cross-modal timing matters

Report each ablation with the same metric and variance as the headline, and state which term carries most of the gain — reviewers reward a paper that can point to why it works.

User studies and QoE

When the claim is subjective (quality, naturalness, engagement, aesthetics), a benchmark number is not enough:

• Pre-register the protocol: task, number of raters, stimuli, and the question asked.
• Report inter-rater agreement and a significance test, not just a mean preference.
• Describe compensation and consent briefly; a study a reviewer cannot assess is discounted.

Data, media, and compute honesty

• State dataset licenses and any consent/usage terms for media, especially for user-generated or scraped content.
• Report compute (hardware, training time) so cost is legible; a cross-modal model that only wins at 10x compute should say so.
• Fix and report seeds; report variance over runs where the margin is small.

Benchmarks and metrics per modality

A cross-modal paper is judged against each community's expectations at once, so pick metrics each sub-field recognizes rather than a single convenient number.

• Retrieval / recommendation — report ranking metrics (Recall@K, mAP, NDCG) and say which gallery/query split, because cross-modal retrieval numbers are split-sensitive.
• Generation / synthesis — pair a distributional metric with a human/QoE judgment; automatic scores for generated media correlate imperfectly with perceived quality.
• Recognition / detection — use the standard task metric, but show the multimodal case, not only the clean single-modality one.
• Systems / delivery — report latency, throughput, and bitrate/quality trade-offs, not just accuracy.

State the metric's direction and any threshold, and keep the same metric across the headline table and every ablation so the reader can trace the fusion's contribution row by row.

Statistical reporting

• Report variance (standard deviation or confidence interval) over runs when margins are small; a single-seed win on a close benchmark is not persuasive.
• For user studies, report a significance test and inter-rater agreement, not a bare mean.
• Do not average away modality-specific behavior: a model that helps on audio-rich clips and hurts on silent ones should show that split, not hide it in a global mean.

Common ACM MM experiment failures

• Fusion that does not matter — ablations show no modality is load-bearing.
• Weak baselines — beating only a vision-only or text-only strawman.
• Asserted perception — "more natural" with no user study.
• Unlicensed media — datasets used without stating rights.
• Hidden cost — big gains that quietly require far more compute.

Output format

[Works] strong/matched baselines / weak or unmatched: <which>
[Fusion causal] ablation isolates the mechanism / does not
[Failure analysis] present per modality / missing
[Perceptual claim] user study with agreement / asserted
[Data + compute] licenses and cost reported / gaps: <list>
[Top fixes] <ordered before submission or rebuttal>