Multilingual LLM Downstream Eval

multilingual-llm-downstream-eval

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection — Messmer et al. (2025) (arXiv:2502.10361, 2025)

What this evaluates

Evaluates the downstream capabilities of multilingual LLMs trained on filtered pretraining data. It probes reading comprehension, general knowledge, natural language understanding, common-sense reasoning, and generative tasks across multiple languages.

Datasets

• FineTasks — total ?; splits: test (-1)
• SmolLM tasks suite — total ?; splits: test (-1)

Metrics

• average rank (primary) — range: other
  • Models are ranked per task based on accuracy (1 = best). The average rank is computed across all tasks and languages. Lower values indicate better overall performance.
• task accuracy — range: [0, 1]
  • Standard exact-match or option-selection accuracy per downstream benchmark task (e.g., MMLU, ARC, HellaSwag, Belebele).

Input / output format

Input: Text prompts from downstream evaluation benchmarks (FineTasks/SmolLM) covering reading comprehension, general knowledge, NLU, common-sense reasoning, and generation.

Output: Model-generated text or selected options for each benchmark task.

Scoring recipe

def compute_average_rank(task_accuracies):
    # task_accuracies: dict of {task_name: {model_id: accuracy}}
    ranks_per_task = {}
    for task, scores in task_accuracies.items():
        sorted_models = sorted(scores.items(), key=lambda x: x[1], reverse=True)
        ranks_per_task[task] = {m: r + 1 for r, (m, _) in enumerate(sorted_models)}
    avg_ranks = {m: sum(ranks_per_task[t][m] for t in ranks_per_task) / len(ranks_per_task) 
                 for m in ranks_per_task[list(ranks_per_task.keys())[0]]}
    return avg_ranks

Common pitfalls

• Average rank is a lower-is-better metric, contrary to standard accuracy scores.
• Evaluation suites differ by language (SmolLM for English, FineTasks for others), so direct score comparison across languages is invalid; only the aggregated rank is comparable.
• Decontamination removes entire documents flagged by 13-gram overlap, which can disproportionately affect shorter documents or specific domains.

Evidence (verbatim from paper)

The average rank is computed across FineTasks performance of 1B-parameter models evaluated after 70B and 119B tokens were consumed.

Citation

@misc{messmer2025enhancing,
  title={Enhancing Multilingual LLM Pretraining with Model-Based Data Selection},
  author={Messmer et al. (2025)},
  year={2025},
  note={arXiv:2502.10361}
}

• arXiv: 2502.10361