Multi Prompt Eval

multi-prompt-eval

Efficient multi-prompt evaluation of LLMs — Felipe Maia Polo et al. (2024) (arXiv:2405.17202, 2024)

What this evaluates

This protocol evaluates how accurately a statistical estimation method can reconstruct the full performance distribution and specific quantiles of large language models across hundreds of prompt templates, using a fraction of the standard evaluation budget. It probes the robustness of LLM performance metrics against arbitrary prompt selection and measures the efficiency of borrowing strength across prompts and examples.

Datasets

• MMLU — total 14000; splits: test (14000)
• BIG-bench Hard — total 1500; splits: test (1500)
• LMentry — total ?; splits: test (-1)

Metrics

• Wasserstein-1 distance ($W_1$) (primary) — range: [0, 1] or percent
  • Average absolute difference between sorted true performance scores and sorted estimated performance scores: $W_1(F,\hat{F}) = \frac{1}{I}\sum_{i=1}^{I}|S_{(i)}-\hat{S}_{(i)}|$. Equivalent to the integral of absolute quantile estimation errors over [0,1].
• Quantile estimation error — range: [0, 1] or percent
  • Absolute difference between the true quantile $Q(p)$ and the estimated quantile $\hat{Q}(p)$ at probability $p \in {0.05, 0.25, 0.5, 0.75, 0.95}$: $|Q(p) - \hat{Q}(p)|$.

Input / output format

Input: LLM prediction scores (or binary correctness) for a subset of benchmark examples evaluated under multiple prompt templates.

Output: Estimated performance distribution $\hat{F}$ and estimated quantiles $\hat{Q}(p)$ for specified probabilities $p$.

Scoring recipe

def compute_w1(true_scores, est_scores):
    true_sorted = sorted(true_scores)
    est_sorted = sorted(est_scores)
    return sum(abs(t - e) for t, e in zip(true_sorted, est_sorted)) / len(true_sorted)

def compute_quantile_error(true_scores, est_scores, p):
    true_q = sorted(true_scores)[int(len(true_scores) * p)]
    est_q = sorted(est_scores)[int(len(est_scores) * p)]
    return abs(true_q - est_q)

Common pitfalls

• Assuming single-prompt evaluation accuracy reflects robustness across prompt variations.
• Confusing the estimation budget (e.g., 200 evaluations) with the full evaluation budget required for ground truth.
• Extreme quantiles (5th/95th) require significantly more evaluations to estimate accurately compared to central quantiles like the median.

Evidence (verbatim from paper)

To compare the full performance distribution $F$ and its estimate $\hat{F}$, both defined in §[3], we use the Wasserstein 1-distance which is equivalent to the average quantile estimation error in this case, i.e., $W_{1}(F,\hat{F}) = \int_{0}^{1}|Q(t)-\hat{Q}(t)|\mathrm{d}t=\frac{1}{I}\sum_{i=1}^{I}|S_{(i)}-\hat{S}_{(i)}|$. Second, we estimate some quantiles of interest (e.g., $5/25/50/75/95$-th) for the performance distribution across prompt formats and compare them with the true quantiles, that is, for some $p\in[0,1]$, we use $|Q(p)-\hat{Q}(p)|$ to measure the quality of our estimations.

Citation

@misc{polo2024prompteval,
  title={Efficient multi-prompt evaluation of LLMs},
  author={Felipe Maia Polo et al. (2024)},
  year={2024},
  note={arXiv:2405.17202}
}

• arXiv: 2405.17202