Subgroup Benchmarking Eval

subgroup-benchmarking-eval

Precise Model Benchmarking with Only a Few Observations — Fogliato et al. (2024) (arXiv:2410.05222, 2024)

What this evaluates

Evaluates the precision of statistical estimators (Empirical Bayes, Synthetic Regression, Direct Training) for estimating model performance on data subgroups with limited observations. It probes how well these methods reduce mean squared error and produce reliable confidence intervals when benchmarking LLMs, vision models, and tabular classifiers on niche tasks.

Datasets

• LLM MC QA tasks — total ?; splits: test (-1); repo https://github.com/EleutherAI/lm-evaluation-harness
• Computer Vision tasks (LAION CLIP benchmark) — total ?; splits: test (-1)
• COCO Captions — total 5000; splits: test (5000)
• Tabular Fairness Datasets (ACS, COMPAS, Student) — total ?; splits: test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between the estimated subgroup performance and the true subgroup performance, averaged across subgroups and datasets.
• Relative Efficiency — range: other
  • Ratio of the average MSE of a competitor method (SR or EB) to the average MSE of the Direct Training (DT) baseline.
• CLAIR score — range: other
  • LLM-based similarity score between generated and reference captions, averaged over reference captions per image.

Input / output format

Input: A subset of $n_g$ model predictions and ground truth labels from a task subgroup, plus subgroup features (for SR) or direct sample statistics.

Output: Point estimate of the subgroup's true performance metric (e.g., accuracy, cross-entropy, or CLAIR score).

Scoring recipe

# For each subgroup g:
# 1. Compute direct estimate mu_hat_g from sample (DT)
# 2. Compute synthetic regression estimate mu_hat_g_SR (SR)
# 3. Compute empirical Bayes estimate mu_hat_g_EB (EB)
# 4. Compute ground truth mu_g over full test set
# 5. MSE_g = (mu_hat_g - mu_g)^2
# 6. MSE_SR_g = (mu_hat_g_SR - mu_g)^2
# 7. MSE_EB_g = (mu_hat_g_EB - mu_g)^2
# Average MSEs across subgroups/datasets
# Relative Efficiency = MSE_competitor / MSE_DT

Common pitfalls

• Confusing the estimator's MSE with the underlying model's task accuracy.
• Assuming Synthetic Regression (SR) consistently outperforms Direct Training (DT); SR fails when subgroup performances are highly heterogeneous.
• Ignoring subgroup size $n_g$ when interpreting relative efficiency gains, as precision differences shrink as $n_g$ increases.

Evidence (verbatim from paper)

Table 2: Comparison of estimation methods on LLM MC QA tasks with subgroups of equal size. The table compares the median over datasets of the relative efficiency (rel. eff.) of DT with respect to SR and EB, namely the ratio of the average MSE of SR or EB divided by the average MSE of DT when all subgroups have size $n_g$.

Citation

@misc{fogliato2024precise,
  title={Precise Model Benchmarking with Only a Few Observations},
  author={Fogliato et al. (2024)},
  year={2024},
  note={arXiv:2410.05222}
}

• arXiv: 2410.05222