Dove Eval

dove-eval

DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation — Habba et al. (2025) (arXiv:2503.01622, 2025)

What this evaluates

This evaluation probes the robustness and prompt sensitivity of large language models on multiple-choice benchmarks by measuring how performance varies across hundreds of millions of intent-preserving prompt perturbations across multiple dimensions.

Datasets

• DOVE — total 250000000; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted answer options. Correctness is determined by identifying the answer option with the highest semantic similarity to the model’s generated response and checking if it matches the ground truth.
• Marginalized Accuracy — range: [0, 1]
  • Average accuracy across all other perturbation dimensions for a fixed value of a specific dimension: Acc_{d1} = sum(Acc(M, Dom, d1, ..., dn)) / (|D2| * ... * |Dn|).
• Divergence Score — range: other
  • Number of standard deviations by which the accuracy on the original prompt deviates from the mean accuracy across all prompt variations: |Acc_original - mean(Acc_all)| / std(Acc_all).

Input / output format

Input: Multiple-choice question instances with prompt variations across multiple dimensions (e.g., enumerators, formatting, phrasing).

Output: Free-text model response.

Scoring recipe

def compute_accuracy(predictions, golds, options):
    correct = 0
    for pred, gold, opts in zip(predictions, golds, options):
        sims = [semantic_similarity(pred, opt) for opt in opts]
        predicted_opt = opts[argmax(sims)]
        if predicted_opt == gold:
            correct += 1
    return correct / len(predictions)

def compute_divergence(original_acc, all_accs):
    return abs(original_acc - mean(all_accs)) / std(all_accs)

Common pitfalls

• API-based models modify prompts for safety or performance, so evaluations must use locally run open-weight models to avoid interference.
• Semantic similarity matching is used instead of exact string matching, requiring careful embedding selection and potentially threshold tuning.
• Marginalized accuracy averages over all other dimensions, which may obscure interaction effects between specific prompt perturbations.

Evidence (verbatim from paper)

To evaluate model outputs we use semantic similarity matching Mitkov et al. ([2009]); Obot et al. ([2023]). For each response, we identify the answer option with highest semantic similarity to the model’s output and consider the prediction correct if it matches the ground truth. Following Mizrahi et al. ([2024]), we quantified performance variance by calculating divergence scores, defined as the number of standard deviations by which performance using the original prompt deviates from the mean performance across all prompts.

Citation

@misc{habba2025dove,
  title={DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation},
  author={Habba et al. (2025)},
  year={2025},
  note={arXiv:2503.01622}
}

• arXiv: 2503.01622