Dynamath Eval

dynamath-eval

DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models — Zou et al. (2024) (arXiv:2411.00836, 2024)

What this evaluates

Evaluates the robustness of Vision-Language Models in mathematical reasoning by measuring performance across dynamically generated variants of seed questions. It probes how well models handle numerical, geometric, and contextual perturbations while maintaining consistent logical deduction.

Datasets

• DynaMath — total 5010; splits: test (5010)

Metrics

• average-case accuracy (primary) — range: [0, 1]
  • Mean accuracy across all variants of all seed questions: $\frac{1}{N}\sum_{i=1}^{N}\frac{1}{M}\sum_{j=1}^{M}\mathbb{I}[\text{Ans}(i,j)=\text{GT}(i,j)]$.
• worst-case accuracy — range: [0, 1]
  • Mean of the minimum accuracy per seed question across its variants: $\frac{1}{N}\sum_{i=1}^{N}\min_{j\in[1,M]}\mathbb{I}[\text{Ans}(i,j)=\text{GT}(i,j)]$.
• Reasoning Robustness (RR) — range: [0, 1]
  • Ratio of worst-case to average-case accuracy: $RR = \mathcal{A}{wst} / \mathcal{A}{avg}$.
• repetition consistency (RC) — range: [0, 1]
  • Fraction of repeated generations matching the primary answer for a given variant: $RC(i,j) = \frac{1}{K}\sum_{k=1}^{K}\mathbb{I}[\text{Ans}_k(i,j)=\text{Ans}(i,j)]$.

Input / output format

Input: Multimodal input consisting of a generated image and a text prompt containing the mathematical question with specific variant parameters (e.g., numerical values, geometric transformations, colors).

Output: Short answer extracted from the model's full or short response using prompt engineering and template matching.

Scoring recipe

# N = 501 seeds, M = 10 variants per seed
avg_acc_sum = 0
wst_acc_sum = 0
for i in range(N):
    variant_correct = []
    for j in range(M):
        pred = extract_short_answer(model.generate(image[i,j], prompt[i,j]))
        gold = ground_truth[i,j]
        is_correct = 1 if normalize(pred) == normalize(gold) else 0
        variant_correct.append(is_correct)
    avg_acc_sum += sum(variant_correct) / M
    wst_acc_sum += min(variant_correct)
avg_case_acc = avg_acc_sum / N
wst_case_acc = wst_acc_sum / N
rr = wst_case_acc / avg_case_acc if avg_case_acc > 0 else 0

Common pitfalls

• Evaluating only the 501 seed questions ignores the dynamic variant generation, failing to measure robustness.
• Multiple-choice questions (34.7% of dataset) allow chance guessing; the paper emphasizes free-form questions for precise evaluation.
• Repetition consistency (RC) requires K generations per variant to measure confidence, which is often skipped in standard single-pass evaluations.

Evidence (verbatim from paper)

Our dataset contains $N=501$ seed questions in total. For each seed question in the dataset, we generate $M=10$ variants, resulting in a total of $5,010$ concrete questions. We evaluate two metrics: average-case accuracy ($\mathcal{A}{avg}$) and worst-case accuracy ($\mathcal{A}{wst}$) over these variants. The two metrics are different from prior benchmarks that evaluate only a single instance of a question.

Citation

@misc{zou2024dynamath,
  title={DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models},
  author={Zou et al. (2024)},
  year={2024},
  note={arXiv:2411.00836}
}

• arXiv: 2411.00836