qhjqhj00 ege-math-assessment-eval
CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam — Khrulev et al. (2025) (arXiv:2507.22958, 2025)
What this evaluates
This benchmark evaluates vision-language models' ability to assess handwritten mathematical solutions against a standardized educational rubric. It probes the models' capacity for error diagnosis, step-by-step reasoning alignment, and accurate grade assignment under varying levels of contextual guidance.
Datasets
- EGE-Math Solutions Assessment Benchmark — total 122; splits: test (122); repo https://github.com/Karifannaa/Auto-check-EGE-math
Metrics
final_score(primary) — range: percent- Percentage of instances where the model's extracted final score exactly matches the expert-provided grade. Calculated as (number of exact matches / total instances) * 100.
Input / output format
Input: A problem statement (text), a handwritten solution image, and optionally the correct final answer or a complete reference solution, depending on the evaluation mode.
Output: A structured assessment containing a step-by-step analysis of the solution, a final score, and a justification for that score.
Scoring recipe
def compute_final_score(predictions, gold_grades):
correct = 0
for pred, gold in zip(predictions, gold_grades):
# Extract the numeric score from the model's structured output
pred_score = extract_numeric_score(pred)
if pred_score == gold:
correct += 1
return (correct / len(gold_grades)) * 100
Common pitfalls
- Handwriting variability and low image resolution can cause recognition failures, leading to incorrect error diagnosis regardless of the model's mathematical reasoning.
- The rubric-based grading requires strict adherence to step-wise credit allocation; models often award partial credit incorrectly or ignore minor procedural deviations.
- Automated score extraction may fail if the model deviates from the exact prompt template format, causing parsing errors in the structured output.
Evidence (verbatim from paper)
The models were instructed to output their assessment in the structured format, including the analysis of the solution, the final score, and the justification for that score. This structured output facilitated automated extraction of the assigned scores for quantitative analysis.
Citation
@misc{khrulev2025checkmat,
title={CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam},
author={Khrulev et al. (2025)},
year={2025},
note={arXiv:2507.22958}
}
- arXiv: 2507.22958