Measurebench Eval

measurebench-eval

Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench — Lin et al. (2025) (arXiv:2510.26865, 2025)

What this evaluates

Evaluates vision-language models on fine-grained visual measurement reading, specifically testing their ability to accurately localize pointers and ticks on instrument scales, map visual cues to numerical values, and recognize measurement units from real-world and synthetic images.

Datasets

• MeasureBench — total ?; splits: real-world (-1), synthetic (-1)

Metrics

• Overall accuracy (primary) — range: percent
  • Percentage of instances where both the predicted numerical value and unit exactly match the ground truth.
• Value accuracy — range: percent
  • Percentage of instances where the predicted numerical value exactly matches the ground truth value, regardless of unit prediction.
• Unit accuracy — range: percent
  • Percentage of instances where the predicted measurement unit exactly matches the ground truth unit.

Input / output format

Input: An image of a measuring instrument (dial, digital, linear, or composite) accompanied by a prompt requesting the model to read the measurement.

Output: A textual response containing the predicted numerical value and the corresponding unit.

Scoring recipe

def compute_metrics(predictions, golds):
    correct_val = correct_unit = correct_overall = 0
    for pred, gold in zip(predictions, golds):
        p_val, p_unit = extract_value_and_unit(pred)
        g_val, g_unit = gold['value'], gold['unit']
        if p_val == g_val: correct_val += 1
        if p_unit == g_unit: correct_unit += 1
        if p_val == g_val and p_unit == g_unit: correct_overall += 1
    n = len(golds)
    return {
        'value_accuracy': correct_val / n * 100,
        'unit_accuracy': correct_unit / n * 100,
        'overall_accuracy': correct_overall / n * 100
    }

Common pitfalls

• Error cancellation: models may arrive at the correct final number through flawed intermediate reasoning, inflating accuracy if only the final answer is scored.
• Unit recognition is near-ceiling (>90%), so overall accuracy is heavily driven by value reading difficulty, masking OCR capabilities.
• Synthetic vs real-world performance gap is small, so synthetic data may not fully capture real-world visual clutter/distortion challenges.

Evidence (verbatim from paper)

Table [2] reports results on MeasureBench for 17 VLMs. The best model, Gemini 2.5 Pro, reaches only 30.3% overall accuracy on real images and 26.1% on synthetic images, showing that reading measuring instruments remains a challenging fine-grained vision task for current VLMs.

Citation

@misc{lin2025measurebench,
  title={Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2510.26865}
}

• arXiv: 2510.26865