Worldsense Eval

worldsense-eval

WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs — Jack Hong et al. (arXiv:2502.04326, 2025)

What this evaluates

Evaluates multimodal large language models' ability to perform real-world omni-modal understanding by jointly processing tightly coupled audio and video inputs. It probes complex temporal reasoning, cross-modal integration, and fine-grained perception across diverse everyday scenarios.

Datasets

• WorldSense — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Calculated as the number of exact matches between model predictions and ground-truth answers divided by the total number of questions, multiplied by 100 to yield a percentage.

Input / output format

Input: Video frames (extracted per model-specific configurations) paired with the corresponding audio track. Ablation variants also accept video with transcribed subtitles/captions.

Output: Natural language text answer to a question about the video content.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p.strip().lower() == g.strip().lower())
    return (correct / len(predictions)) * 100

Common pitfalls

• Evaluating on unimodal inputs (video-only or audio-only) significantly underestimates performance, as accuracy drops by ~15% when either modality is removed.
• Using transcribed subtitles instead of raw audio fails to capture prosody and paralinguistic cues, leading to misleading metrics for true omni-modal integration.
• Direct string matching is used without explicit mention of LLM-as-a-judge or fuzzy matching, so paraphrased but correct answers may be incorrectly penalized.

Evidence (verbatim from paper)

Gemini 1.5 Pro, capable of processing both audio and visual information, achieves the highest accuracy of 48.0%.

Citation

@misc{hong2025worldsense,
  title={WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs},
  author={Jack Hong et al.},
  year={2025},
  note={arXiv:2502.04326}
}

• arXiv: 2502.04326