# Signature Overlap

> This metric quantifies the degree of overlap between different LLM benchmarks by comparing their token-level perplexity signatures derived from in-the-wild pretraining corpora, revealing whether performance correlations stem from shared latent capacity familiarity or benchmark-orthogonal factors like question format. Use when the user has predictions and gold and needs to compute signature_overlap.

- Skill: `qhjqhj00/signature-overlap` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/signature-overlap`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/signature-overlap/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/signature-overlap

---


# signature_overlap

> Mapping Overlaps in Benchmarks through Perplexity in the Wild — Wu et al. (2025) (arXiv:2509.23488, 2025)

## What this evaluates

This metric quantifies the degree of overlap between different LLM benchmarks by comparing their token-level perplexity signatures derived from in-the-wild pretraining corpora, revealing whether performance correlations stem from shared latent capacity familiarity or benchmark-orthogonal factors like question format.

## Datasets

- **RedPajama** — total ?; splits: (unstated)

## Metrics

- `signature_overlap` **(primary)** — range: [-1, 1]
  - Pairwise Pearson correlation between benchmark signature vectors, where signatures are derived via linear regression on token-level perplexity patterns from in-the-wild corpora.

## Input / output format

**Input**: Token sequences from in-the-wild corpora (RedPajama) used to compute per-benchmark token-level perplexity vectors.

**Output**: A scalar correlation coefficient representing the signature overlap between two benchmarks.

## Scoring recipe

```python
# Compute token-level perplexity for each benchmark's questions on RedPajama
perplexity_vectors = {bench: compute_perplexity(bench_questions, corpus) for bench in benchmarks}
# Extract signature via linear regression on pretraining exposure patterns
signatures = {bench: linear_regression(perplexity_vectors[bench]) for bench in benchmarks}
# Compute pairwise correlation between signature vectors
overlap_matrix = {
    (b1, b2): pearson_correlation(signatures[b1], signatures[b2])
    for b1, b2 in itertools.combinations(benchmarks, 2)
}
return overlap_matrix
```

## Common pitfalls

- Confusing signature overlap with semantic or performance overlap, as the latter show much weaker discriminative ability and are heavily biased by question format.
- Assuming high overlap indicates true semantic alignment, when it often reflects benchmark-orthogonal factors like shared question formats or training data contamination.
- Ignoring the impact of benchmark category size when calculating cross-category overlaps, which requires equal weighting to avoid skewing results.

## Evidence (verbatim from paper)

> We extract benchmark signatures from the open dataset RedPajama (Weber et al., 2024). ... signature overlap (panel a), on the left, we compare within-category overlap against the average cross-category overlap. ... signature-level analysis reveals statistically insignificant tiny differences based on the Mann–Whitney U test, yielding results around 0. This aligns with intuition, as the signature provides a good approximation of the true overlap and variation.

## Citation

```bibtex
@misc{wu2025mappingoverlaps,
  title={Mapping Overlaps in Benchmarks through Perplexity in the Wild},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2509.23488}
}
```

- arXiv: 2509.23488