# Mme Realworld Eval

> mme-realworld-eval

- Skill: `qhjqhj00/mme-realworld-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/mme-realworld-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mme-realworld-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/mme-realworld-eval

---


# mme-realworld-eval

> MM-RLHF: The Next Step Forward in Multimodal LLM Alignment — Zhang et al. (2025) (arXiv:2502.10391, 2025)

## What this evaluates

Evaluates multimodal large language models' perception and reasoning capabilities on high-resolution, real-world images across five domains: optical character recognition, remote sensing, diagrams/tables, monitoring, and autonomous driving.

## Datasets

- **MME-RealWorld** — total 29429; splits: test (29429)

## Metrics

- `Avg` **(primary)** — range: percent
  - Weighted average accuracy across subtasks/domains. Calculated as the sum of per-subtask accuracies multiplied by their respective QA pair counts, divided by the total number of QA pairs.
- `Avg-C` — range: percent
  - Unweighted average accuracy across subtasks/domains. Calculated as the simple arithmetic mean of per-subtask accuracies.

## Input / output format

**Input**: High-resolution image paired with a text question.

**Output**: Text response (typically multiple-choice or short answer).

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds) * 100

def compute_avg_accuracy(subtask_accuracies, subtask_sizes):
    weighted_sum = sum(acc * size for acc, size in zip(subtask_accuracies, subtask_sizes))
    return weighted_sum / sum(subtask_sizes)
```

## Common pitfalls

- Closed-source models often compress high-resolution images due to upload size/resolution limits, artificially lowering performance.
- Models may adopt conservative strategies, outputting 'E' (object not present) or refusing to answer due to safety filters, skewing accuracy.
- Weighted vs. unweighted averages can yield significantly different rankings when subtask sizes vary greatly.

## Evidence (verbatim from paper)

> “Avg” and “Avg-C” indicate the weighted average accuracy and the unweighted average accuracy across subtasks in each domain.

## Citation

```bibtex
@misc{zhang2025mmrlhf,
  title={MM-RLHF: The Next Step Forward in Multimodal LLM Alignment},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2502.10391}
}
```

- arXiv: 2502.10391