Finevision Eval

finevision-eval

FineVision: Open Data Is All You Need — Wiedmann et al. (2025) (arXiv:2510.17269, 2025)

What this evaluates

Evaluates vision-language models on a diverse suite of 11 multimodal benchmarks covering visual question answering, chart understanding, document parsing, and general multimodal reasoning. Additionally probes GUI/agentic capabilities on screen interaction tasks.

Datasets

• AI2D — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• InfoVQA — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• SEED-Bench — total ?; splits: test (-1)
• Screenspot-V2 — total ?; splits: test (-1)
• Screenspot-Pro — total ?; splits: test (-1)

Metrics

• mean normalized performance (%) (primary) — range: [0, 100]
  • Each benchmark's raw score is min-max normalized to [0,100]. The final metric is the arithmetic mean of these normalized scores across all 11 benchmarks.

Input / output format

Input: Image(s) and text prompt/question per instance, formatted according to each benchmark's specification.

Output: Text response generated by the model, evaluated against ground truth using benchmark-specific evaluators.

Scoring recipe

scores = []
for bench in benchmarks:
    raw = evaluate(model, bench)
    norm = min_max_normalize(raw, bench.min, bench.max)
    scores.append(norm)
final_metric = sum(scores) / len(scores)

Common pitfalls

• Benchmark scores are min-max normalized to [0,100] before averaging, which can obscure task-specific performance differences.
• Test data contamination varies across datasets (1.02% for FineVision vs 2.15–3.05% for baselines), directly impacting reported scores.
• Small models often fail completely at the base stage, requiring fine-tuning to show meaningful results on GUI benchmarks.

Evidence (verbatim from paper)

mean normalized performance (%) across 11 evaluation benchmarks (higher is better), with the training step shown in thousands ($\times 10^{3}$). Each benchmark score is min–max normalized to [0,100] and averaged per evaluation step; the model trained on FineVision (blue) leads throughout the second half of training and attains the best final score.

Citation

@misc{wiedmann2025finevision,
  title={FineVision: Open Data Is All You Need},
  author={Wiedmann et al. (2025)},
  year={2025},
  note={arXiv:2510.17269}
}

• arXiv: 2510.17269