Mmlongbench Eval

mmlongbench-eval

MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly — Wang et al. (2025) (arXiv:2505.10610, 2025)

What this evaluates

Evaluates long-context vision-language models across five diverse tasks: visual retrieval-augmented generation, needle-in-a-haystack retrieval/reasoning, many-shot in-context learning, document summarization, and long-document VQA. It specifically probes how model performance scales with standardized context lengths from 8K to 128K tokens when processing interleaved text and images.

Datasets

• MMLongBench — total 13331; splits: test (13331); repo https://github.com/EdinburghNLP/MMLongBench

Metrics

• substring exact match (SubEM) (primary) — range: [0, 1]
  • 1 if the gold answer string is found as a substring within the model's predicted answer, else 0.
• accuracy (Acc) (primary) — range: [0, 1]
  • 1 if the predicted class ID, count, or retrieved item exactly matches the ground truth, else 0.
• LLM-based evaluation — range: [0, 1]
  • An external LLM judges the generated summary against the reference summary for faithfulness and completeness, replacing traditional ROUGE-L.

Input / output format

Input: Multimodal context comprising interleaved text passages and images, standardized to specific token lengths (8K, 16K, 32K, 64K, or 128K). Text tokens are counted using the Llama2 tokenizer. Image tokens are calculated by dividing each image into 14×14 patches and applying a 2×2 pixel unshuffle. Contexts are constructed by inserting gold passages/images among distractors, truncating long documents, or padding short ones.

Output: Model must generate a text response containing the answer to the prompt, which may be an entity name, a class ID, a numerical count, a retrieved image reference, or a summary paragraph.

Scoring recipe

def compute_metrics(predictions, golds, metric_type):
    scores = []
    for pred, gold in zip(predictions, golds):
        if metric_type == 'SubEM':
            scores.append(1.0 if gold.lower() in pred.lower() else 0.0)
        elif metric_type == 'Acc':
            scores.append(1.0 if pred == gold else 0.0)
        elif metric_type == 'LLM-based':
            scores.append(llm_judge_score(pred, gold))
    return sum(scores) / len(scores) if scores else 0.0

Common pitfalls

• Using raw image count instead of token count to define context length, which the paper explicitly criticizes as non-standard across modalities.
• Relying on ROUGE-L for summarization evaluation, whereas the benchmark mandates LLM-based judgment to better align with human assessment.
• Padding short documents with random negative documents that may inadvertently contain answer-related information, potentially altering ground-truth validity.

Evidence (verbatim from paper)

We use the substring exact match (SubEM) as the metric, following previous work[[74]]. In VH, we obtain the needle images and the target objects from the original dataset. Then, we accompany these needles with multiple negative distractor images until the image haystack reaches a given input length L. We report accuracy as the metric following the original work[[16]].

Citation

@misc{wang2025mmlongbench,
  title={MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.10610}
}

• arXiv: 2505.10610