Innovator Vl Eval

innovator-vl-eval

Innovator-VL: A Multimodal Large Language Model for Scientific Discovery — Wen et al. (2026) (arXiv:2601.19325, 2026)

What this evaluates

Evaluates multimodal large language models across general vision, mathematical reasoning, and specialized scientific domains to measure visual perception, instruction following, and domain-specific knowledge retention.

Datasets

• AI2D — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• MMMU(Val) — total ?; splits: val (-1)
• MMMU-Pro (Standard) — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• VStar-Bench — total ?; splits: test (-1)
• MMBench-EN — total ?; splits: dev (-1), test (-1)
• MME-RealWorld — total ?; splits: test (-1)
• DocVQA(Val) — total ?; splits: val (-1)
• InfoVQA(Val) — total ?; splits: val (-1)
• SEED-Bench — total ?; splits: test (-1)
• SEED-Bench-2-plus — total ?; splits: test (-1)
• RealWorldQA — total ?; splits: test (-1)
• MathVision — total ?; splits: test (-1), mini (-1)
• MathVerse — total ?; splits: mini (-1)
• MathVista — total ?; splits: mini (-1)
• WeMath — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)
• RxnBench — total ?; splits: en (-1), zh (-1)
• MolParse — total ?; splits: test (-1); HF InnovatorLab/MolParse
• OpenRxn — total ?; splits: test (-1); HF InnovatorLab/OpenRxn
• EMVista — total ?; splits: test (-1); HF InnovatorLab/EMVista
• SuperChem — total ?; splits: en (-1), zh (-1)
• SmolInstruct — total ?; splits: test (-1)
• ProteinLMBench — total ?; splits: test (-1)
• SFE — total ?; splits: en (-1), zh (-1)
• MicroVQA — total ?; splits: test (-1)
• MSEarth-MCQ — total ?; splits: test (-1)
• XLRS-Bench-lite — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions across multiple-choice or open-ended prompts. Computed per benchmark and averaged within capability categories (general, math & reasoning, science).

Input / output format

Input: Multimodal instances containing domain-specific or general images paired with text prompts/questions.

Output: Text responses generated by the model, typically multiple-choice selections or short-form answers, decoded deterministically (temperature 0.0, top-p 1.0).

Scoring recipe

predictions = []
for dataset in benchmarks:
    for instance in dataset.test:
        output = model.generate(image=instance.image, prompt=instance.question, temperature=0.0, top_p=1.0)
        predictions.append(extract_answer(output))
    acc = compute_accuracy(predictions, dataset.gold_answers)
    category_scores[dataset.category].append(acc)
final_score = average(category_scores)

Common pitfalls

• Averaging accuracy across highly heterogeneous benchmarks (e.g., OCR vs. molecular parsing) can obscure domain-specific strengths or weaknesses.
• Using non-deterministic decoding (temperature > 0) on multiple-choice scientific benchmarks introduces significant score variance and reduces reproducibility.
• The lmms-eval framework requires specific prompt templates and answer extraction regexes per dataset; mismatched configurations yield artificially low scores.

Evidence (verbatim from paper)

We run all evaluations using the lmms-eval framework (Zhang et al., [2024b]). For Innovator-VL-8B-Instruct and Innovator-VL-8B-Thinking, we use deterministic decoding (temperature 0.0, top-p 1.0) across all benchmarks. ... Figure 5: Token efficiency comparison across vision reasoning benchmarks. ... (b) Accuracy-to-token ratio, which measures the reasoning efficiency, demonstrating that Innovator-VL-8B-Thinking achieves 1.4x to 2x higher accuracy-to-token ratio than MiMo-VL-7B-RL and 3.9x to 4.3x higher than Intern-S1-mini.

Citation

@misc{wen2026innovatorvl,
  title={Innovator-VL: A Multimodal Large Language Model for Scientific Discovery},
  author={Wen et al. (2026)},
  year={2026},
  note={arXiv:2601.19325}
}

• arXiv: 2601.19325