ACM MM Experiments
Use this to make an ACM Multimedia paper's evidence match its claim. The reviewer's implicit
questions are: does it work, does the cross-modal part cause the gain, when does it fail,
and — if the target is perceptual — do people actually prefer it.
The four questions and how to answer them
| Question |
Evidence that answers it |
| Does it work? |
The headline metric on a recognized benchmark, against strong, matched baselines |
| Does the fusion cause the gain? |
A leave-one-modality-out / component ablation isolating the cross-modal term |
| When does it fail? |
Failure cases per modality (e.g., noisy audio, missing captions) shown honestly |
| Do people prefer it? |
A user study with reported N, protocol, and inter-rater agreement — for subjective claims |
The second row is what separates an ACM MM experiment section from a single-modality one:
if removing a modality does not move the result, the paper is not really cross-modal.
Matched baselines
- Compare against the strongest existing method, re-run under your data and preprocessing
where feasible, not a weakened reimplementation.
- Include a late-fusion / naive-concatenation baseline so the reader sees what the fancy
fusion buys over the obvious one.
- Hold everything but the mechanism fixed: same backbone, same features, same training budget,
so the delta is attributable.
Ablations that isolate the cross-modal claim
Full model .................... reference
- audio stream ................ tests whether audio carries signal
- text/caption stream ......... tests whether language carries signal
- alignment / fusion module ... replaced by concatenation: tests the MECHANISM
- synchronization assumption .. shuffled timing: tests whether cross-modal timing matters
Report each ablation with the same metric and variance as the headline, and state which term
carries most of the gain — reviewers reward a paper that can point to why it works.
User studies and QoE
When the claim is subjective (quality, naturalness, engagement, aesthetics), a benchmark
number is not enough:
- Pre-register the protocol: task, number of raters, stimuli, and the question asked.
- Report inter-rater agreement and a significance test, not just a mean preference.
- Describe compensation and consent briefly; a study a reviewer cannot assess is discounted.
Data, media, and compute honesty
- State dataset licenses and any consent/usage terms for media, especially for
user-generated or scraped content.
- Report compute (hardware, training time) so cost is legible; a cross-modal model that
only wins at 10x compute should say so.
- Fix and report seeds; report variance over runs where the margin is small.
Benchmarks and metrics per modality
A cross-modal paper is judged against each community's expectations at once, so pick metrics
each sub-field recognizes rather than a single convenient number.
- Retrieval / recommendation — report ranking metrics (Recall@K, mAP, NDCG) and say which
gallery/query split, because cross-modal retrieval numbers are split-sensitive.
- Generation / synthesis — pair a distributional metric with a human/QoE judgment;
automatic scores for generated media correlate imperfectly with perceived quality.
- Recognition / detection — use the standard task metric, but show the multimodal case, not
only the clean single-modality one.
- Systems / delivery — report latency, throughput, and bitrate/quality trade-offs, not just
accuracy.
State the metric's direction and any threshold, and keep the same metric across the headline
table and every ablation so the reader can trace the fusion's contribution row by row.
Statistical reporting
- Report variance (standard deviation or confidence interval) over runs when margins are
small; a single-seed win on a close benchmark is not persuasive.
- For user studies, report a significance test and inter-rater agreement, not a bare mean.
- Do not average away modality-specific behavior: a model that helps on audio-rich clips and
hurts on silent ones should show that split, not hide it in a global mean.
Common ACM MM experiment failures
- Fusion that does not matter — ablations show no modality is load-bearing.
- Weak baselines — beating only a vision-only or text-only strawman.
- Asserted perception — "more natural" with no user study.
- Unlicensed media — datasets used without stating rights.
- Hidden cost — big gains that quietly require far more compute.
Output format
[Works] strong/matched baselines / weak or unmatched: <which>
[Fusion causal] ablation isolates the mechanism / does not
[Failure analysis] present per modality / missing
[Perceptual claim] user study with agreement / asserted
[Data + compute] licenses and cost reported / gaps: <list>
[Top fixes] <ordered before submission or rebuttal>
1---2name: acmmm-experiments3description: Use when designing or auditing the experiments of an ACM MM (ACM Multimedia) paper — matched baselines per modality, ablations that isolate the cross-modal fusion, user studies or QoE measurement where the claim is subjective, dataset and media licensing, and honest compute reporting, so evidence supports a multimedia claim.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/acmmm-experiments6---78# ACM MM Experiments910Use this to make an ACM Multimedia paper's evidence match its claim. The reviewer's implicit11questions are: does it work, **does the cross-modal part cause the gain**, when does it fail,12and — if the target is perceptual — do people actually prefer it.1314## The four questions and how to answer them1516| Question | Evidence that answers it |17|---|---|18| Does it work? | The headline metric on a recognized benchmark, against strong, matched baselines |19| Does the *fusion* cause the gain? | A leave-one-modality-out / component ablation isolating the cross-modal term |20| When does it fail? | Failure cases per modality (e.g., noisy audio, missing captions) shown honestly |21| Do people prefer it? | A user study with reported N, protocol, and inter-rater agreement — for subjective claims |2223The second row is what separates an ACM MM experiment section from a single-modality one:24if removing a modality does not move the result, the paper is not really cross-modal.2526## Matched baselines2728- Compare against the **strongest** existing method, re-run under your data and preprocessing29 where feasible, not a weakened reimplementation.30- Include a **late-fusion / naive-concatenation** baseline so the reader sees what the fancy31 fusion buys over the obvious one.32- Hold everything but the mechanism fixed: same backbone, same features, same training budget,33 so the delta is attributable.3435## Ablations that isolate the cross-modal claim3637```text38Full model .................... reference39- audio stream ................ tests whether audio carries signal40- text/caption stream ......... tests whether language carries signal41- alignment / fusion module ... replaced by concatenation: tests the MECHANISM42- synchronization assumption .. shuffled timing: tests whether cross-modal timing matters43```4445Report each ablation with the same metric and variance as the headline, and state which term46carries most of the gain — reviewers reward a paper that can point to *why* it works.4748## User studies and QoE4950When the claim is subjective (quality, naturalness, engagement, aesthetics), a benchmark51number is not enough:5253- Pre-register the protocol: task, number of raters, stimuli, and the question asked.54- Report **inter-rater agreement** and a significance test, not just a mean preference.55- Describe compensation and consent briefly; a study a reviewer cannot assess is discounted.5657## Data, media, and compute honesty5859- State dataset **licenses** and any consent/usage terms for media, especially for60 user-generated or scraped content.61- Report **compute** (hardware, training time) so cost is legible; a cross-modal model that62 only wins at 10x compute should say so.63- Fix and report **seeds**; report variance over runs where the margin is small.6465## Benchmarks and metrics per modality6667A cross-modal paper is judged against each community's expectations at once, so pick metrics68each sub-field recognizes rather than a single convenient number.6970- **Retrieval / recommendation** — report ranking metrics (Recall@K, mAP, NDCG) *and* say which71 gallery/query split, because cross-modal retrieval numbers are split-sensitive.72- **Generation / synthesis** — pair a distributional metric with a **human/QoE** judgment;73 automatic scores for generated media correlate imperfectly with perceived quality.74- **Recognition / detection** — use the standard task metric, but show the *multimodal* case, not75 only the clean single-modality one.76- **Systems / delivery** — report latency, throughput, and bitrate/quality trade-offs, not just77 accuracy.7879State the metric's direction and any threshold, and keep the same metric across the headline80table and every ablation so the reader can trace the fusion's contribution row by row.8182## Statistical reporting8384- Report **variance** (standard deviation or confidence interval) over runs when margins are85 small; a single-seed win on a close benchmark is not persuasive.86- For **user studies**, report a significance test and inter-rater agreement, not a bare mean.87- Do not average away modality-specific behavior: a model that helps on audio-rich clips and88 hurts on silent ones should show that split, not hide it in a global mean.8990## Common ACM MM experiment failures9192- **Fusion that does not matter** — ablations show no modality is load-bearing.93- **Weak baselines** — beating only a vision-only or text-only strawman.94- **Asserted perception** — "more natural" with no user study.95- **Unlicensed media** — datasets used without stating rights.96- **Hidden cost** — big gains that quietly require far more compute.9798## Output format99100```text101[Works] strong/matched baselines / weak or unmatched: <which>102[Fusion causal] ablation isolates the mechanism / does not103[Failure analysis] present per modality / missing104[Perceptual claim] user study with agreement / asserted105[Data + compute] licenses and cost reported / gaps: <list>106[Top fixes] <ordered before submission or rebuttal>107```