qhjqhj00 multilingual-lang-prof-eval
The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks — Pomerenke et al. (2025) (arXiv:2507.08538, 2025)
What this evaluates
Evaluates large language models' multilingual capabilities across 100–200 languages by aggregating performance on translation, question answering, mathematics, and reasoning tasks. It tracks proficiency trends over time and correlates them with language speaker counts, GDP, and data availability.
Datasets
- Aggregated Multilingual Tasks (Translation, QA, Math, Reasoning) — total ?; splits: test (-1)
Metrics
language proficiency scores(primary) — range: unspecified- Aggregated performance score across translation, question answering, mathematics, and reasoning tasks evaluated across 100–200 languages. Exact mathematical formulation is not provided in the text.
Input / output format
Input: Not specified in the provided text.
Output: Not specified in the provided text.
Scoring recipe
def compute_language_proficiency_scores(predictions, gold):
lang_scores = {}
for lang in languages:
task_scores = [score(task) for task in ['translation', 'qa', 'math', 'reasoning']]
lang_scores[lang] = mean(task_scores)
return aggregate(lang_scores)
Common pitfalls
- Academic benchmark scores may not correlate with downstream application performance in a given language.
- Current version lacks cross-lingual trend analysis and has limited task coverage.
- Evaluation is auto-updating and dynamic, making static reproduction difficult without version pinning.
Evidence (verbatim from paper)
The graph confirms already known trends: English as the language with most resources employs the highest language proficiency scores. [Figure 2] shows language proficiency scores by different models over time, showcasing that Google models currently perform best.
Citation
@misc{pomerenke2025ailanguageproficiencymonitor,
title={The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks},
author={Pomerenke et al. (2025)},
year={2025},
note={arXiv:2507.08538}
}
- arXiv: 2507.08538