Mathvista Eval

mathvista-eval

MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts — Lu et al. (2023) (arXiv:2310.02255, 2023)

What this evaluates

This benchmark evaluates the mathematical reasoning capabilities of foundation models (LLMs and LMMs) when processing visual contexts. It probes abilities such as figure interpretation, algebraic and geometric reasoning, and the integration of multimodal inputs like images, OCR text, and captions into mathematical problem-solving.

Datasets

• MATHVISTA — total ?; splits: testmini (-1)

Metrics

• accuracy (primary) — range: percent
  • Accuracy is computed by comparing the normalized extracted answer against the ground truth. The evaluation pipeline extracts a short answer from the model's detailed response, normalizes it to a required format (option letter or integer), and checks for exact match.

Input / output format

Input: Task description, question, choices (if applicable), and metadata, optionally accompanied by an image, generated image captions, and detected OCR text.

Output: A detailed response containing reasoning or code, followed by a short answer text (option letter or numerical value) that is extracted and normalized for scoring.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        extracted = extract_answer_llm(pred)
        normalized_pred = normalize(extracted)
        if normalized_pred == gold:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Using template matching or regex for answer extraction fails on long LLM responses; the protocol requires an LLM-based extractor (e.g., GPT-4) for >99.5% accuracy.
• Assuming augmented LLMs (with OCR/captions) can match native LMMs, as they lack direct visual perception and struggle with complex figure interpretation and shape detection.
• Reporting results on the full test set when the paper explicitly provides and highlights scores only on the 'testmini' subset.

Evidence (verbatim from paper)

Finally, the extracted answer is normalized to a required answer format (e.g., an option letter or an integer), and the target metric scores are computed. Taking advantage of the fact that the instances in MATHVISTA are either multiple-choice questions for textual answers or free-form questions for numerical answers, accuracy scores are used as metrics for deterministic evaluation.

Citation

@misc{lu2023mathvista,
  title={MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts},
  author={Lu et al. (2023)},
  year={2023},
  note={arXiv:2310.02255}
}

• arXiv: 2310.02255