qhjqhj00 la-leaderboard-eval
La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America — Grandury et al. (2025) (arXiv:2507.00999, 2025)
What this evaluates
Evaluates LLMs on multilingual proficiency across Spanish varieties and regional languages of Spain and Latin America (Basque, Catalan, Galician). It probes capabilities in natural language inference, reasoning, question answering, summarization, and linguistic acceptability using a resource-efficient few-shot configuration.
Datasets
- La Leaderboard (66 datasets) — total ?; splits: test (-1)
Metrics
exact-match(primary) — range: [0, 1]- Percentage of predictions that exactly match the gold reference string. The leaderboard also aggregates task-specific accuracy/F1 scores for NLI, QA, classification, and summarization.
Input / output format
Input: Few-shot prompts containing task instructions and examples, followed by the target query in the target language.
Output: Model-generated text response.
Scoring recipe
def score_exact_match(predictions, golds):
correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
return correct / len(golds)
Common pitfalls
- Assuming translation-based datasets capture sociolinguistic nuance
- Overlooking the difference between base and instruction-tuned model performance on the same task
- Ignoring the few-shot configuration's impact on computational overhead and results
Evidence (verbatim from paper)
Each model-task pair was evaluated once, and the raw, pre-normalization results analyzed in this work are publicly available. ... math reasoning yields the lowest results, which could be related to a too strict metric (exact match).
Citation
@misc{grandury2025laleaderboard,
title={La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America},
author={Grandury et al. (2025)},
year={2025},
note={arXiv:2507.00999}
}
- arXiv: 2507.00999