Multimodal Benchmarks Eval

multimodal-benchmarks-eval

OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs — Gao et al. (2025) (arXiv:2511.19023, 2025)

What this evaluates

Evaluates multimodal perception and reasoning capabilities across image, video, and audio understanding tasks. It probes the model's ability to process heterogeneous modalities and answer complex questions or transcribe speech accurately.

Datasets

• AI2D — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• MMVet — total ?; splits: test (-1)
• Mathvista — total ?; splits: test (-1)
• LongVideoBench — total ?; splits: test (-1)
• DiDeMo — total ?; splits: test (-1)
• AVQA — total ?; splits: test (-1)
• MVBench — total ?; splits: test (-1)
• Video-MME — total ?; splits: test (-1)
• Aishell1 — total ?; splits: test (-1)
• LibriSpeech — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 100] percent
  • Percentage of correctly answered questions out of the total number of questions in the benchmark.
• word error rate (WER) — range: percent
  • Standard speech recognition metric calculating the ratio of substitutions, deletions, and insertions to the reference transcript.

Input / output format

Input: Multimodal inputs (images, video frames, audio clips) paired with text prompts or questions. Videos are uniformly sampled to 128 frames.

Output: Text responses containing the model's predicted answer or transcription.

Scoring recipe

def compute_score(predictions, golds, modality):
    if modality == 'audio':
        return calculate_wer(predictions, golds)  # Lower is better
    else:
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return (correct / len(golds)) * 100  # Higher is better

Common pitfalls

• Video evaluation uses a fixed 128-frame sampling strategy, which may discard critical temporal context for long videos.
• Audio WER is lower-is-better while visual/text accuracy is higher-is-better; averaging across modalities requires careful direction normalization.
• Benchmarks like MMVet and MathVista often rely on LLM-as-judge or custom grading rubrics, introducing potential scorer bias not detailed in the protocol.

Evidence (verbatim from paper)

Our model is evaluated across a diverse set of audio, image, and video understanding tasks to comprehensively assess its multimodal perception and reasoning capabilities. For image-text understanding, we select a suite of challenging multimodal and vision-centric benchmarks: AI2D, MMMU, MMStar, OCRBench, MMVet, and Mathvista. For video understanding, we include LongVideoBench, DiDeMo, AVQA, MVBench, and Video-MME, using a uniform sampling strategy that extract 128 frames per video. For audio performance, we evaluate the model on standard speech recognition benchmarks, including public Chinese datasets such as Aishell1 and English datasets such as LibriSpeech. As shown in Table 2, OrdMoE consistently improves over the baseline across all modalities: it reduces word error rate (WER) on audio benchmarks (lower is better), and increases accuracy on both image and video understanding tasks (higher is better).

Citation

@misc{gao2025ordmo,
  title={OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2511.19023}
}

• arXiv: 2511.19023