Signature Overlap

signature_overlap

Mapping Overlaps in Benchmarks through Perplexity in the Wild — Wu et al. (2025) (arXiv:2509.23488, 2025)

What this evaluates

This metric quantifies the degree of overlap between different LLM benchmarks by comparing their token-level perplexity signatures derived from in-the-wild pretraining corpora, revealing whether performance correlations stem from shared latent capacity familiarity or benchmark-orthogonal factors like question format.

Datasets

• RedPajama — total ?; splits: (unstated)

Metrics

• signature_overlap (primary) — range: [-1, 1]
  • Pairwise Pearson correlation between benchmark signature vectors, where signatures are derived via linear regression on token-level perplexity patterns from in-the-wild corpora.

Input / output format

Input: Token sequences from in-the-wild corpora (RedPajama) used to compute per-benchmark token-level perplexity vectors.

Output: A scalar correlation coefficient representing the signature overlap between two benchmarks.

Scoring recipe

# Compute token-level perplexity for each benchmark's questions on RedPajama
perplexity_vectors = {bench: compute_perplexity(bench_questions, corpus) for bench in benchmarks}
# Extract signature via linear regression on pretraining exposure patterns
signatures = {bench: linear_regression(perplexity_vectors[bench]) for bench in benchmarks}
# Compute pairwise correlation between signature vectors
overlap_matrix = {
    (b1, b2): pearson_correlation(signatures[b1], signatures[b2])
    for b1, b2 in itertools.combinations(benchmarks, 2)
}
return overlap_matrix

Common pitfalls

• Confusing signature overlap with semantic or performance overlap, as the latter show much weaker discriminative ability and are heavily biased by question format.
• Assuming high overlap indicates true semantic alignment, when it often reflects benchmark-orthogonal factors like shared question formats or training data contamination.
• Ignoring the impact of benchmark category size when calculating cross-category overlaps, which requires equal weighting to avoid skewing results.

Evidence (verbatim from paper)

We extract benchmark signatures from the open dataset RedPajama (Weber et al., 2024). ... signature overlap (panel a), on the left, we compare within-category overlap against the average cross-category overlap. ... signature-level analysis reveals statistically insignificant tiny differences based on the Mann–Whitney U test, yielding results around 0. This aligns with intuition, as the signature provides a good approximation of the true overlap and variation.

Citation

@misc{wu2025mappingoverlaps,
  title={Mapping Overlaps in Benchmarks through Perplexity in the Wild},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2509.23488}
}

• arXiv: 2509.23488