Smart 840 Eval

smart-840-eval

Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads — Cherian et al. (2024) (arXiv:2406.15736, 2024)

What this evaluates

Evaluates large vision-and-language models on mathematical reasoning tasks from the Math Kangaroo Olympiad, testing their ability to solve grade-appropriate (K-12) multiple-choice problems that may require joint text and image interpretation.

Datasets

• SMART-840 — total 840; splits: test (840)

Metrics

• accuracy (primary) — range: percent
  • Percentage of problems correctly answered out of the total number of problems in the set. For human baselines, it is computed as the average across problems of the ratio of correct responses to total attempts per problem.

Input / output format

Input: Problem text and associated image (if applicable), accompanied by the prompt: 'Solve this question with explanation of the intermediate steps. Your response should end with one of the selected answer options from A1, B2, C3, D4, or E5.'

Output: Step-by-step explanation followed by the final selected answer option (A1, B2, C3, D4, or E5).

Scoring recipe

def compute_accuracy(predictions, gold_options):
    correct = 0
    total = len(gold_options)
    for pred, gold in zip(predictions, gold_options):
        if pred is None or pred == 'unsolved' or not is_valid_format(pred):
            continue
        extracted = extract_last_option(pred)  # Parse A1-E5
        if extracted == gold:
            correct += 1
    return (correct / total) * 100

Common pitfalls

• Models may refuse to solve problems due to perceived security issues in images, counting as unsolved/invalid.
• Invalid output formats or failure to follow the exact ending option format lead to automatic rejection during parsing.
• Significant variance in model accuracy across multiple runs, especially for lower-grade problems, requires multiple queries per problem.

Evidence (verbatim from paper)

For the LVLMs, we use the API interface to query the model using a suitable hand-crafted prompt. Specifically, we found the following prompt to work well for all closed-source LVLMs: "Solve this question with explanation of the intermediate steps. Your response should end with one of the selected answer options from A1, B2, C3, D4, or E5." which is accompanied by the text for the problem question and the image data. For AI models, we report their accuracy as the (percentage) of problems correctly answered to the total number of problems in the set.

Citation

@misc{cherian2024evaluating,
  title={Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads},
  author={Cherian et al. (2024)},
  year={2024},
  note={arXiv:2406.15736}
}

• arXiv: 2406.15736