Morse 500 Eval

morse-500-eval

MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning — Cai et al. (2025) (arXiv:2506.05523, 2025)

What this evaluates

Evaluates multimodal reasoning capabilities of vision-language models across six categories: mathematical, abstract, spatial, temporal, physical, and planning. It uses programmatically generated video clips to test dynamic visual narrative comprehension while eliminating prompt-based shortcuts.

Datasets

• MORSE-500 — total 500; splits: test (500); repo https://github.com/morse-benchmark/morse-500-code

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions over the benchmark. Model predictions are extracted using an external LLM (e.g., Qwen2.5 72B AWQ) and compared to ground truth via exact string matching.

Input / output format

Input: Video clip (or sampled frames at 2fps, max 32 frames for image-only models, downsampled to max 512px side length) paired with the minimal instruction: 'Answer the question in this video.'

Output: Free-form text answer to the question embedded in the video.

Scoring recipe

def compute_accuracy(predictions, ground_truths):
    correct = 0
    for pred, gold in zip(predictions, ground_truths):
        extracted = extract_answer_with_llm(pred)
        if extracted.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(ground_truths)) * 100

Common pitfalls

• Image-only models require frame sampling (2fps, max 32 frames) and downscaling to 512px, which may degrade temporal and spatial reasoning performance.
• Answer extraction relies on an external LLM for string matching, which can introduce parsing errors or bias independent of the evaluated model's actual capability.
• No few-shot examples or format-specific guidance are provided, forcing models to self-format answers and complicating automated evaluation.

Evidence (verbatim from paper)

We report accuracy as the primary evaluation metric—the percentage of correctly answered questions over the benchmark. We provided detailed instructions on the output formatting in the video, and we extract the answers from the model prediction using a LLM (e.g. Qwen2.5 72B AWQ) and perform string matching for accuracy calculation, following MathVista [Lu et al., 2024].

Citation

@misc{cai2025morse500,
  title={MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2506.05523}
}

• arXiv: 2506.05523