Mavors Video Image Eval

mavors-video-image-eval

Mavors: Multi-granularity Video Representation for Multimodal Large Language Model — Shi et al. (2025) (arXiv:2504.10068, 2025)

What this evaluates

Evaluates multimodal large language models on video and image understanding tasks, including general knowledge QA, long-video QA, event understanding, temporal reasoning, and captioning, as well as image QA, cognitive understanding, and captioning.

Datasets

• MMWorld — total ?; splits: test (-1)
• PerceptionTest — total ?; splits: test (-1)
• Video-MME — total ?; splits: test (-1)
• MLVU — total ?; splits: test (-1)
• MVBench — total ?; splits: test (-1)
• EventHallusion — total ?; splits: test (-1)
• TempCompass — total ?; splits: test (-1)
• VinoGround — total ?; splits: test (-1)
• DREAM-1K — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• CapsBench — total ?; splits: test (-1)

Metrics

• score (primary) — range: percent
  • Average of task-specific scores (typically accuracy or normalized performance) across all tasks within a benchmark. Reported as a percentage.

Input / output format

Input: Video frames (processed in chunks of 16) or single images paired with text prompts/questions.

Output: Text responses (multiple-choice answers or descriptive captions).

Scoring recipe

def compute_score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if is_mcq(gold):
            correct += 1 if normalize(pred) == normalize(gold) else 0
        else:
            correct += caption_metric(pred, gold)
    return (correct / len(golds)) * 100

Common pitfalls

• Scores for most baseline models are taken directly from their original papers rather than reproduced, which may introduce inconsistencies due to different evaluation settings or versions.
• No benchmark-specific hyperparameters (e.g., frame sampling rate, resolution) were tuned during evaluation, meaning models are compared under a uniform but potentially suboptimal setting for each specific dataset.
• Video benchmarks cover diverse capabilities (long-video, event, temporal) that require different evaluation focuses, so a single average score may mask strengths/weaknesses in specific temporal reasoning tasks.

Evidence (verbatim from paper)

Video understanding capabilities are assessed across general knowledge QA (MMWorld, PerceptionTest), long-video QA (Video-MME, MLVU), event understanding QA (MVBench, EventHallusion), temporal understanding QA (TempCompass, VinoGround), and captioning (DREAM-1K). Image understanding evaluation includes comprehensive capabilities (MMMU), cognitive understanding (MathVista, AI2D), and captioning (CapsBench). Crucially, aside from prompt modifications, no benchmark-specific hyperparameters (e.g., frame sampling, resolution) were tuned during evaluation for any model, including Mavors.

Citation

@misc{shi2025mavors,
  title={Mavors: Multi-granularity Video Representation for Multimodal Large Language Model},
  author={Shi et al. (2025)},
  year={2025},
  note={arXiv:2504.10068}
}

• arXiv: 2504.10068