Milebench Eval

milebench-eval

MileBench: Benchmarking MLLMs in Long Context — Song et al. (2024) (arXiv:2404.18532, 2024)

What this evaluates

Evaluates Multimodal Large Language Models (MLLMs) on long-context, multi-image comprehension. It probes capabilities like needle-in-a-haystack retrieval, image retrieval, temporal reasoning across multiple images, and semantic understanding in long multimodal contexts.

Datasets

• MileBench — total 6440; splits: test (6440)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice and needle-in-a-haystack questions.
• ROUGE-L — range: percent
  • n-gram-based metric used for open-ended generation tasks.

Input / output format

Input: Interleaved images and text prompts/questions. When input length exceeds the model's maximum context length, the instruction is kept and the interleaved image-text question is truncated from the left.

Output: Text answers generated via greedy decoding, with a designated generation length between 1 and 512 tokens.

Scoring recipe

def compute_metric(prediction, gold, task_type):
    if task_type in ['multiple_choice', 'needle_in_haystack']:
        return 1.0 if prediction.strip() == gold.strip() else 0.0
    elif task_type == 'open_ended':
        return rouge_l_score(prediction, gold)

Common pitfalls

• Many open-source models partially answer needle-in-a-haystack tasks (e.g., outputting only part of a numeric string) without fully matching, resulting in zero accuracy despite partial retrieval.
• Input truncation from the left when exceeding context limits may discard critical question information, as the instruction and question are deemed critical.
• Models may generate image captions instead of following multi-image QA instructions due to lack of multi-image training data.

Evidence (verbatim from paper)

Metrics for each dataset, as shown in Table[6], are consistent with the original work for tasks built on previous datasets. For open-ended generation tasks, the popular n-gram-based metric ROUGE-L is adopted, and accuracy is the metric for multiple-choice and needle-in-a-haystack tasks.

Citation

@misc{song2024milebench,
  title={MileBench: Benchmarking MLLMs in Long Context},
  author={Song et al. (2024)},
  year={2024},
  note={arXiv:2404.18532}
}

• arXiv: 2404.18532