Mm Alignbench Eval

mm-alignbench-eval

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference — Zhao et al. (2025) (arXiv:2502.18411, 2025)

What this evaluates

Evaluates how well multi-modal large language models align with human preferences when answering open-ended questions about diverse images. It probes the model's ability to follow complex instructions, handle real-world scenarios, and produce responses that match human expectations better than baseline models.

Datasets

• MM-AlignBench — total ?; splits: test (-1); repo https://github.com/PhoenixZ810/OmniAlign-V

Metrics

• Win Rate (primary) — range: percent
  • Percentage of pairwise comparisons where the evaluated model's response is preferred over a reference model's response. Calculated as (Count(B) + Count(B+)) / Total Comparisons * 100.
• Reward — range: other
  • Average preference score or margin derived from pairwise comparisons, indicating the strength of alignment relative to the reference model.

Input / output format

Input: A single image paired with an open-ended question or instruction prompt.

Output: A free-form text response generated by the MLLM.

Scoring recipe

# Pairwise evaluation protocol
responses = model.generate(image, prompt)
reference_responses = get_reference_responses(image, prompt) # e.g., Claude-3-V-Sonnet
comparisons = pairwise_compare(responses, reference_responses) # Human or judge model
win_rate = (count(comparisons == 'B') + count(comparisons == 'B+')) / len(comparisons) * 100
reward = average_preference_score(comparisons)
return win_rate, reward

Common pitfalls

• The benchmark uses pairwise comparisons against a strong reference model (e.g., Claude-3-V-Sonnet), so scores are relative, not absolute.
• Win Rate and Reward are reported together; confusing the raw count of B+/B/T/W/W+ with the final Win Rate percentage is a common error.
• The evaluation focuses on open-ended, real-world questions rather than standard VQA, so models optimized for closed-set VQA may underperform despite high factual accuracy.

Evidence (verbatim from paper)

After applying DPO with OmniAlign-V-DPO, LLaVA-OA-32B-DPO achieves winning rate of 72.6 with an average reward of +33.5, surpassing the performance of Qwen2VL-72B.

Citation

@misc{zhao2025omnialignv,
  title={OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2502.18411}
}

• arXiv: 2502.18411