Simmmdg Eval

simmmdg-eval

SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization — Dong et al. (2023) (arXiv:2310.19795, 2023)

What this evaluates

Evaluates a model's ability to generalize across unseen domains in multi-modal action recognition. It probes feature disentanglement, cross-modal translation for missing modalities, and robustness to domain shifts in video, audio, and optical flow inputs.

Datasets

• EPIC-Kitchens — total ?; splits: D1 (-1), D2 (-1), D3 (-1)
• HAC — total 3381; splits: H (-1), A (-1), C (-1)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted action labels out of the total number of test instances.

Input / output format

Input: Multi-modal inputs (video, audio, and/or optical flow) representing human actions across different source domains.

Output: Predicted action class label from the predefined set (8 classes for EPIC-Kitchens, 7 for HAC).

Scoring recipe

def top1_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

Common pitfalls

• Domain generalization splits are defined by source/target domain combinations (e.g., D2, D3 → D1) rather than fixed train/val/test sets.
• Missing modality handling drastically changes results; zero-filling embeddings often underperforms cross-modal translation or even unimodal baselines.
• Backbone architecture (I3D vs. SlowFast/ResNet-18) significantly shifts absolute accuracy scores, so comparisons must match the backbone.

Evidence (verbatim from paper)

We report the Top-1 accuracy for all experiments. The EPIC-Kitchens dataset includes eight actions ('put', 'take', 'open', 'close', 'wash', 'cut', 'mix', and 'pour') recorded in three different kitchens, forming three separate domains D1, D2, and D3. Our HAC dataset consists of seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures, forming three different domains H, A, and C. We collect 3381 video clips from the internet with around 1000 samples for each domain.

Citation

@misc{dong2023simmmdg,
  title={SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization},
  author={Dong et al. (2023)},
  year={2023},
  note={arXiv:2310.19795}
}

• arXiv: 2310.19795