Pile Perplexity Eval

pile-perplexity-eval

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — Leo Gao et al. (arXiv:2101.00027, 2020)

What this evaluates

This evaluation probes a language model's ability to capture statistical patterns in diverse English text domains and its cross-domain generalization. It measures next-token prediction accuracy across academic, technical, legal, and conversational corpora.

Datasets

• The Pile — total 825; splits: test (-1); repo https://github.com/EleutherAI/the-pile

Metrics

• perplexity (BPB) (primary) — range: other
  • Test perplexity converted to bits per UTF-8 encoded byte. Calculated as the exponential of the average negative log-likelihood per byte, where lower values indicate better language modeling performance.

Input / output format

Input: Raw text documents/sequences from the evaluation split.

Output: Next-token probability distributions or log-probabilities for language modeling.

Scoring recipe

def compute_bpb(model, test_dataset):
    total_log_prob = 0.0
    total_bytes = 0
    for doc in test_dataset:
        tokens = model.tokenize(doc)
        log_probs = model.log_prob(tokens)
        total_log_prob += sum(log_probs)
        total_bytes += len(doc.encode('utf-8'))
    perplexity = math.exp(-total_log_prob / total_bytes)
    return perplexity

Common pitfalls

• Decontamination is required: evaluation set instances must be removed from training data using 13-gram overlap filtering to prevent data leakage.
• Size control: datasets are downsampled to 40GB for fair comparison, which makes the evaluation generous to smaller baselines like CC-100.
• Evaluation scope: only one-tenth of the Pile test set is used, and results are averaged per-document.

Evidence (verbatim from paper)

Table 2: Test perplexity of the Pile using GPT-2 and GPT-3, converted to bits per UTF-8 encoded byte (BPB). Evaluation is performed on one-tenth of the test data of the Pile, on a per-document basis.

Citation

@misc{gao2021pile,
  title={The Pile: An 800GB Dataset of Diverse Text for Language Modeling},
  author={Leo Gao et al.},
  year={2020},
  note={arXiv:2101.00027}
}

• arXiv: 2101.00027