Refinedweb Zero Shot Eval

refinedweb-zero-shot-eval

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only — Penedo et al. (2023) (arXiv:2306.01116, 2023)

What this evaluates

Evaluates the zero-shot generalization capability of autoregressive language models across multiple task aggregates. It measures how well models trained on raw web data perform on downstream tasks without any fine-tuning or prompt engineering.

Datasets

• Eleuther AI LM evaluation harness (zero-shot aggregates) — total ?; splits: test (-1); repo https://github.com/EleutherAI/lm-evaluation-harness

Metrics

• zero-shot accuracy (primary) — range: percent
  • Percentage of tasks where the model's generated completion exactly matches the ground-truth label or correct answer in a zero-shot setting. Averaged across task aggregates (small, core, main, ext).

Input / output format

Input: Task description or prompt followed by the question, provided in a zero-shot format without in-context examples.

Output: Autoregressive text completion. Accuracy is determined by parsing the generated text and comparing it to the gold label.

Scoring recipe

def compute_zero_shot_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Perplexity or validation loss alone can be misleading and does not correlate well with end-task zero-shot performance.
• Comparing models trained/evaluated with different codebases, numerical precisions, or prompts introduces externalities that skew results.
• Filtering heuristics do not guarantee consistent improvements across different source datasets; they often require source-specific tuning.

Evidence (verbatim from paper)

We base our evaluation on the popular Eleuther AI evaluation harness (Gao et al., 2021), allowing us to evaluate across a wide range of tasks in the zero-shot setting. ... Average accuracy in zero-shot on the small-agg aggregate.

Citation

@misc{penedo2023refinedweb,
  title={The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only},
  author={Penedo et al. (2023)},
  year={2023},
  note={arXiv:2306.01116}
}

• arXiv: 2306.01116