Cl Gsmsym Eval

cl-gsmsym-eval

Multi-lingual Functional Evaluation for Large Language Models — Ojewale et al. (2025) (arXiv:2506.20793, 2025)

What this evaluates

Assesses mathematical reasoning and symbolic computation capabilities of LLMs across multiple languages. It uses dynamic, variable-driven templates to generate verifiable ground truths for each instance. The evaluation probes model resilience to linguistic variations and template-specific weaknesses.

Datasets

• CL-GSMSym — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly solved mathematical problems where the model's output matches the verifiable ground truth.

Input / output format

Input: Mathematical reasoning prompts translated into English, French, Spanish, Hindi, Arabic, or Yoruba, featuring dynamic variables and symbolic constraints.

Output: Model-generated text responses containing the solution to the mathematical problem.

Scoring recipe

For each instance, extract the model's final answer and compare it against the verifiable ground truth.
Mark as correct (1) if they match exactly, else incorrect (0).
Average across instances to compute accuracy.

Common pitfalls

• Performance on low-resource languages like Yoruba drops drastically compared to high-resource languages.
• Template-level robustness varies significantly; some templates (e.g., probabilistic inference) consistently yield lower accuracy across all models and languages.
• Static benchmark rankings do not correlate with functional benchmark rankings, leading to misleading model comparisons.

Evidence (verbatim from paper)

We define the language performance gap to be the difference of the model’s accuracy on its lowest performant language, and the highest performant language.

Citation

@misc{ojewale2025multilingual,
  title={Multi-lingual Functional Evaluation for Large Language Models},
  author={Ojewale et al. (2025)},
  year={2025},
  note={arXiv:2506.20793}
}

• arXiv: 2506.20793