qhjqhj00 mwp-localization-eval
Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages — Azime et al. (2025) (arXiv:2508.14913, 2025)
What this evaluates
Evaluates LLMs' ability to solve math word problems after socio-cultural localization of entities into low-resource languages. Probes whether models maintain reasoning accuracy when cultural context shifts, focusing solely on final answer correctness rather than step-by-step reasoning.
Datasets
- Unspecified — total ?; splits: test (-1)
Metrics
Exact Match (EM)(primary) — range: [0, 1]- Evaluates whether the predicted answer exactly matches the reference answer as a string.
Numeric Match (NM)— range: [0, 1]- Checks whether the predicted numerical value matches the ground truth after converting both to floating point, ignoring formatting, units, or punctuation, and accounting for minor numerical errors.
Input / output format
Input: Localized math word problems presented via three customized prompt variations, designed to elicit only the final answer without intermediate reasoning steps.
Output: A single final answer string or numerical value.
Scoring recipe
def exact_match(pred, gold):
return 1.0 if pred.strip() == gold.strip() else 0.0
def numeric_match(pred, gold):
try:
p = float(pred.replace(',', '').replace('%', '').strip())
g = float(gold.replace(',', '').replace('%', '').strip())
return 1.0 if abs(p - g) < 1e-6 else 0.0
except ValueError:
return 0.0
Common pitfalls
- Focusing on intermediate reasoning steps instead of final answers, which contradicts the paper's explicit focus on final outputs due to multilingual extraction challenges.
- Applying strict string matching for Numeric Match, which fails to account for the paper's instruction to ignore formatting, units, and punctuation after float conversion.
- Using inconsistent prompt variations, as the study specifically adopts three variants from Adelani et al. (2024) customized for final-output-only responses.
Evidence (verbatim from paper)
Exact Match (EM) evaluates whether the predicted answer exactly matches the reference answer as a string. Numeric Match (NM) checks whether the predicted numerical value matches the ground truth, ignoring differences in formatting such as units or punctuation after converting them into floating point data and account for errors between them.
Citation
@misc{azime2025bridging,
title={Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages},
author={Azime et al. (2025)},
year={2025},
note={arXiv:2508.14913}
}
- arXiv: 2508.14913