Effective Dimensionality

effective-dimensionality

BenchScope: How Many Independent Signals Does Your Benchmark Provide? — Sha et al. (2026) (arXiv:2603.29357, 2026)

What this evaluates

Effective Dimensionality (ED) quantifies the number of independent signals or latent axes captured by a benchmark, measuring how much redundancy exists across its tasks. It probes whether a benchmark's claimed breadth actually reflects diverse evaluation dimensions or merely correlated task performance.

Datasets

• BigCodeBench — total ?; splits: test (-1)
• SWE-bench Verified — total ?; splits: test (-1)
• BFCL — total ?; splits: test (-1)
• LiveCodeBench — total ?; splits: test (-1)
• Open LLM Leaderboard — total ?; splits: test (-1)

Metrics

• Effective Dimensionality (ED) (primary) — range: other
  • ED is computed as the participation ratio of the centered benchmark score spectrum. Given a centered score matrix, singular values are extracted, and ED is calculated as the ratio of the squared sum of singular values to the sum of their fourth powers. It represents the effective number of independent axes in the evaluation.

Input / output format

Input: A benchmark evaluation matrix of shape (N models × T tasks) containing per-instance pass rates or continuous scores.

Output: A single scalar float representing the effective dimensionality.

Scoring recipe

def compute_ed(score_matrix):
    # score_matrix: N x T matrix of pass rates or continuous scores
    centered = score_matrix - score_matrix.mean(axis=0)  # task-centering
    U, S, Vt = np.linalg.svd(centered, full_matrices=False)
    ed = (np.sum(S)**2) / np.sum(S**4)
    return ed

Common pitfalls

• ED systematically overestimates the true latent dimensionality (k) due to binary noise creating a floor of small singular values; it should be interpreted as an upper bound.
• Binarizing continuous scores at arbitrary thresholds (e.g., 0.3 vs 0.5) significantly alters absolute ED values, though qualitative conclusions often remain stable.
• Task-centering is the default and recommended method; model-centering or double-centering can drastically change ED magnitudes and should be reported explicitly.

Evidence (verbatim from paper)

Effective Dimensionality (ED) introduces a fast, population-conditional diagnostic based on the participation ratio of centered benchmark score spectra to quantify measurement redundancy. Applied to 22 benchmarks across 8 domains and >8,400 model evaluations, ED reveals up to 27× variation in effective dimensionality—e.g., Open LLM Leaderboard behaves like 1.7 axes, BBH and MMLU-Pro are nearly redundant (ρ=0.96), and MATH/MuSR show Simpson’s paradox—demonstrating that claimed breadth often overstates true measurement diversity.

Citation

@misc{sha2026benchscope,
  title={BenchScope: How Many Independent Signals Does Your Benchmark Provide?},
  author={Sha et al. (2026)},
  year={2026},
  note={arXiv:2603.29357}
}

• arXiv: 2603.29357