Disco Eval

disco-eval

DISCO: Diversifying Sample Condensation for Efficient Model Evaluation — Rubinstein et al. (2025) (arXiv:2510.07959, 2025)

What this evaluates

This protocol evaluates how well a metamodel can predict the benchmark performance of unseen models using a highly condensed subset of test samples. It probes the trade-off between evaluation cost reduction and the fidelity of accuracy estimation and model ranking preservation across language and vision benchmarks.

Datasets

• MMLU — total 14000; splits: test (-1)
• HellaSwag — total 10000; splits: test (-1)
• Winogrande — total 1300; splits: test (-1)
• ARC — total 1200; splits: test (-1)
• ImageNet-1k — total 1280000; splits: test (-1)

Metrics

• MAE (primary) — range: percent
  • Mean Absolute Error of model accuracies, calculated as the average absolute difference between true and predicted accuracy scores.
• Spearman rank correlation (primary) — range: [0, 1]
  • Spearman rank correlation coefficient measuring the consistency of the relative ordering of models between true and estimated performances.

Input / output format

Input: Concatenated model signatures (predictions/responses on the condensed test set) for each target model.

Output: Predicted accuracy score and predicted rank for the target model.

Scoring recipe

def compute_metrics(true_accs, pred_accs, true_ranks, pred_ranks):
    mae = np.mean(np.abs(np.array(true_accs) - np.array(pred_accs)))
    rank_corr = spearmanr(true_ranks, pred_ranks).correlation
    return {'MAE': mae, 'Rank': rank_corr}

Common pitfalls

• Metabench requires significantly more samples to converge (150-450 points), making direct comparison at 100 points unfair.
• Using performance-based splits instead of chronological splits artificially inflates prediction difficulty by testing on models that outperform training models.
• Failing to apply dimensionality reduction (e.g., PCA) to model signatures causes overfitting and drops rank correlation from ~0.987 to ~0.918.

Evidence (verbatim from paper)

We evaluate DISCO and baseline approaches using two complementary metrics. First, the Mean Absolute Error (MAE) of the model accuracies, reported as percentage points (%p), captures the absolute error of accuracy prediction. Second, to assess the consistency of the relative ordering of models, we report the Spearman rank correlation (Rank) in model ranking between the true and estimated model performances.

Citation

@misc{rubinstein2025disco,
  title={DISCO: Diversifying Sample Condensation for Efficient Model Evaluation},
  author={Rubinstein et al. (2025)},
  year={2025},
  note={arXiv:2510.07959}
}

• arXiv: 2510.07959