Xmodbench Eval

xmodbench-eval

XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models — Wang et al. (2025) (arXiv:2510.15148, 2025)

What this evaluates

This benchmark probes the cross-modal consistency and reasoning capabilities of omni-language models by evaluating semantic equivalence across all six possible modality combinations (text, vision, audio) for both context and candidate inputs. It measures how well models maintain performance when modalities are swapped or combined, highlighting modality-specific biases and directional asymmetries.

Datasets

• XModBench — total 60828; splits: test (60828)

Metrics

• accuracy (primary) — range: [0, 100] percent
  • Percentage of correctly answered multiple-choice questions. Computed per modality configuration (e.g., Audio→Text, Vision→Text) and averaged across tasks and configurations to report overall performance and standard deviation.

Input / output format

Input: Multiple-choice question where context and candidate options are provided in specific modality combinations (e.g., audio context with text candidates, text context with vision candidates).

Output: Selection of the correct candidate option from the provided set.

Scoring recipe

correct = 0
total = 0
for config in modality_configs:
    for q in dataset[config]:
        pred = model.predict(q.context, q.candidates)
        if pred == q.gold_answer:
            correct += 1
        total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Confusing context vs. candidate modalities (e.g., treating A→T the same as T→A), which masks directional imbalance.
• Assuming modality substitution yields symmetric performance; the benchmark explicitly tests for asymmetries where swapping context/candidate roles changes accuracy significantly.
• Overlooking that audio-vision combinations without text anchors yield significantly lower scores, which is a key finding rather than a model failure.

Evidence (verbatim from paper)

Table 2 reports results across five task families and six cross-modal directions among Text, Vision, and Audio. The first subtable summarizes the average accuracy across all tasks for each modality configuration, while the remaining subtables present detailed performance within each task family.

Citation

@misc{wang2025xmodbench,
  title={XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.15148}
}

• arXiv: 2510.15148