Wan Image Understanding Eval

wan-image-understanding-eval

Wan-Image: Pushing the Boundaries of Generative Visual Intelligence — Mao et al. (2026) (arXiv:2604.19858, 2026)

What this evaluates

Evaluates the model's multimodal and text understanding capabilities across a suite of standard academic benchmarks covering visual question answering, reasoning, hallucination detection, and text-based reasoning.

Datasets

• MMMU — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• HalluBench — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• AIME — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• HLE — total ?; splits: test (-1)
• LCBV6 — total ?; splits: test (-1)

Metrics

• average score (primary) — range: [0, 100]
  • Percentage of correctly answered questions or normalized benchmark score as reported by the official benchmark leaderboards. Averages are computed across selected benchmarks.

Input / output format

Input: Image and/or text prompts/questions provided by the benchmark.

Output: Textual answers or multiple-choice selections.

Scoring recipe

def compute_average_score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

scores = [compute_average_score(preds, golds) for preds, golds in dataset_splits]
avg_score = sum(scores) / len(scores)

Common pitfalls

• Scores are reported as averages across benchmarks rather than per-dataset in the main text.
• The table uses '–' for missing indicators, which should be excluded from averages.
• Comparison includes both understanding-only and unified models, so direct comparison requires matching model scale/type.

Evidence (verbatim from paper)

For Image Understanding, we selected 7 classical multi-modal benchmarks to assess the model’s performance, including MMMU, MMStar, MathVista, HalluBench, MMBench, OCRBench, and AI2D. The results shown in Table 4 demonstrate that our model exhibits superior instruct and thinking capabilities: in the instruct mode, it achieves an average score 1.6 points higher than the baseline and remains comparable to other open-source understanding-only instruct models. Concurrently, in thinking mode, our model outperforms the baseline by 5.4 points and exceeds the performance of other open-source understanding-only thinking models. Compared to other unified models, our approach maintains a distinct and absolute lead. Regarding Text Understanding, our model demonstrates significant improvements over the showcased understanding-only models in both instruct and thinking modes on four common benchmarks: AIME, GPQA, HLE, and LCBV6.

Citation

@misc{mao2026wanimage,
  title={Wan-Image: Pushing the Boundaries of Generative Visual Intelligence},
  author={Mao et al. (2026)},
  year={2026},
  note={arXiv:2604.19858}
}

• arXiv: 2604.19858