Finetasks Eval

finetasks-eval

Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets — Alrashed et al. (2025) (arXiv:2512.18834, 2025)

What this evaluates

Evaluates the downstream quality of multilingual pretraining corpora by training language models on them and measuring performance on a standardized suite of fine-tuning tasks across Arabic, Hindi, and Turkish.

Datasets

• FineTasks — total ?; splits: test (-1)

Metrics

• FineTasks scores (primary) — range: [0, 1]
  • Aggregate score computed as the average of normalized task scores across the FineTasks benchmark suite. Values are reported as floating-point numbers between 0 and 1.

Input / output format

Input: Language models are trained on variants of multilingual pretraining corpora (Arabic, Hindi, Turkish) for up to 30 billion tokens, then evaluated on the FineTasks benchmark suite.

Output: Aggregate FineTasks score (a single floating-point number representing average task performance across the benchmark).

Scoring recipe

scores = []
for task in finetasks_tasks:
    acc = evaluate_model_on_task(task)
    scores.append(normalize(acc))
aggregate_score = sum(scores) / len(scores)
return aggregate_score

Common pitfalls

• Confusing the size of the pretraining corpus (billions of tokens) with the evaluation benchmark size.
• Assuming performance gains are solely due to data quality without controlling for training compute or checkpoint differences.
• Overlooking that FineTasks scores are aggregate averages, which can mask per-language or per-task variations.

Evidence (verbatim from paper)

We evaluate MixMinMatch through the training procedure outlined in Section[6.2] on each corpus variant and measure aggregate FineTasks scores as described in Section[6.3].

Citation

@misc{alrashed2025mixminhash,
  title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
  author={Alrashed et al. (2025)},
  year={2025},
  note={arXiv:2512.18834}
}

• arXiv: 2512.18834