Mmerealworld Eval

mmerealworld-eval

MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? — Yi-Fan Zhang et al. (2024) (arXiv:2408.13257, 2024)

What this evaluates

This benchmark evaluates multimodal large language models on high-resolution real-world image perception and complex reasoning tasks. It probes the models' ability to extract fine-grained details from large images and perform logical inference across diverse domains like autonomous driving, remote sensing, and document understanding.

Datasets

• MME-RealWorld — total 29429; splits: test (29429)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total predictions) * 100.
• weighted average accuracy — range: percent
  • Average accuracy across subtasks weighted by the number of QA pairs in each subtask.
• unweighted average accuracy — range: percent
  • Average accuracy across subtasks calculated without weighting by subtask size.

Input / output format

Input: High-resolution images (average 2000×1500 pixels) paired with natural language questions covering perception and reasoning tasks.

Output: Text responses answering the provided questions. Models may also output specific tokens like "E" to indicate object absence.

Scoring recipe

def calculate_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Closed-source models often have strict image resolution/size limits, requiring compression that degrades performance on high-res inputs.
• Models may adopt conservative strategies (e.g., outputting "E" for missing objects) or refuse to answer due to safety filters, artificially lowering accuracy.
• Accuracy is computed per subtask and then averaged; failing to account for subtask size differences can misrepresent overall capability.

Evidence (verbatim from paper)

“Avg” and “Avg-C” indicate the weighted average accuracy and the unweighted average accuracy across subtasks in each domain.

Citation

@misc{zhang2024mmerealworld,
  title={MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?},
  author={Yi-Fan Zhang et al. (2024)},
  year={2024},
  note={arXiv:2408.13257}
}

• arXiv: 2408.13257