Simba Benchmark Analysis Eval

simba-benchmark-analysis-eval

SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone — Subramani et al. (2025) (arXiv:2510.17998, 2025)

What this evaluates

Evaluates a framework for analyzing language model performance matrices by identifying dataset-model correlations, discovering minimal representative dataset subsets, and predicting held-out model performance while preserving model rankings.

Datasets

• HELM — total ?; splits: train (23), test (6)
• MMLU — total ?; splits: train (63), test (16)
• BigBenchLite — total ?; splits: train (36), test (9)

Metrics

• coverage ($\eta$) (primary) — range: [0, 1]
  • Measures how well a representative dataset subset captures the performance patterns of the full benchmark as the subset size grows. Calculated by comparing model performance distributions between the subset and the full set.
• $R^2$ — range: [0, 1]
  • Regression-based coefficient of determination used to quantify linear, exponential, or power-law correlations between datasets and models. Computed as 1 - (SS_res / SS_tot).

Input / output format

Input: Pre-computed performance matrices containing model scores across datasets, partitioned into training and test model sets.

Output: Minimal representative dataset subsets, regression fit parameters, predicted performance matrices for held-out models, and preserved model rankings.

Scoring recipe

def compute_coverage(full_matrix, subset_matrix):
    # Compare performance patterns between full and subset matrices
    similarity = pearson_correlation(full_matrix.flatten(), subset_matrix.flatten())
    return similarity

def compute_r_squared(y_true, y_pred):
    ss_res = sum((y - y_hat)**2 for y, y_hat in zip(y_true, y_pred))
    ss_tot = sum((y - mean(y_true))**2 for y in y_true)
    return 1 - (ss_res / ss_tot)

Common pitfalls

• Models are randomly split 80/20 across train/test, not the dataset instances.
• All datasets appear in both train and test splits, so evaluation tests model generalization across model splits, not dataset splits.
• Coverage and R² are computed at the matrix level, ignoring instance-level prediction errors.

Evidence (verbatim from paper)

Figure 2: Here, we measure the coverage ( $\eta$) across HELM, MMLU, and BigBenchLite as our representative subset $S$ grows. We report performance for all three baselines and all nine similarity measures discussed in [§˜3]. For our analysis, we look at three benchmarks: HELM, MMLU, and BigBenchLite... splitting each benchmark into training and test sets to validate our analysis. All datasets are included in both splits, but models are separated across training and test sets randomly with 80% of models in training and 20% of the models in test.

Citation

@misc{subramani2025simba,
  title={SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone},
  author={Subramani et al. (2025)},
  year={2025},
  note={arXiv:2510.17998}
}

• arXiv: 2510.17998