Fineweb2 Early Signal Eval

fineweb2-early-signal-eval

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language — Penedo et al. (2025) (arXiv:2506.20920, 2025)

What this evaluates

Evaluates the impact of different pre-training data processing steps on downstream model quality by training small language models and measuring performance on a curated suite of multilingual zero-shot benchmarks.

Datasets

• FineWeb2 Early-Signal Benchmark Suite — total ?; splits: test (-1)

Metrics

• per-category macro-average score (primary) — range: [0, 1]
  • Task scores are rescaled relative to a random baseline: new_score = (score - random_baseline) / (1 - random_baseline). Scores below the random baseline are clamped to 0. The final metric is computed by averaging within task categories (RC, GK, NLU, CR) and then taking a macro-average across categories.

Input / output format

Input: Zero-shot prompts/questions for reading comprehension, general knowledge, NLU, and common-sense reasoning tasks across 9 languages.

Output: Model-generated text/answer evaluated against gold labels using task-specific metrics before rescaling.

Scoring recipe

def compute_aggregate(task_results, baselines, categories):
    rescaled = []
    for task, score in task_results.items():
        rb = baselines[task]
        new_score = max(0, (score - rb) / (1 - rb))
        rescaled.append((task, new_score))
    cat_scores = {}
    for task, score in rescaled:
        cat_scores.setdefault(categories[task], []).append(score)
    cat_avgs = [sum(s)/len(s) for s in cat_scores.values()]
    return sum(cat_avgs) / len(cat_avgs)

Common pitfalls

• Failing to rescale scores relative to the random baseline before averaging, which skews results for tasks with high random performance.
• Averaging across all tasks directly instead of using the specified per-category macro-average, misrepresenting performance when task categories are unbalanced across languages.
• Assuming a single multilingual model is used; the protocol explicitly trains separate monolingual models per language to avoid confounders.

Evidence (verbatim from paper)

To produce an aggregate score across tasks, we follow the approach used by Fourrier et al. (2024); Li et al. (2024b) and average scores across tasks after first rescaling scores based on the random baseline – any score below the random baseline is considered 0, and for the remaining scores we subtract the random baseline value and shift the scores as new_score=(score-random_baseline)/(1-random_baseline). As some languages might have an unbalanced number of tasks for each task category (RC, GK, NLU and CR), during score averaging we first average within categories themselves and then take the average of each category. This per-category macro-average score is our final reported aggregate score.

Citation

@misc{penedo2025fineweb2,
  title={FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language},
  author={Penedo et al. (2025)},
  year={2025},
  note={arXiv:2506.20920}
}

• arXiv: 2506.20920