Platinum Benchmarks Eval

platinum-benchmarks-eval

Do Large Language Model Benchmarks Test Reliability? — Vendrow et al. (2025) (arXiv:2502.03461, 2025)

What this evaluates

Evaluates LLM reliability on curated, low-noise subsets of standard benchmarks (VQA v2.0, SQuAD 2.0, HotPotQA, DROP, BIG-bench) by removing ambiguous examples and re-labeling to minimize ground-truth errors, revealing true model failures on elementary reasoning tasks.

Datasets

• VQA v2.0 — total ?; splits: (unstated)
• SQuAD 2.0 — total ?; splits: (unstated)
• HotPotQA — total ?; splits: (unstated)
• DROP — total ?; splits: (unstated)
• BIG-bench — total ?; splits: (unstated)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as the proportion of correct predictions out of total evaluated instances. A prediction is correct if it exactly matches any manually enumerated valid response for the platinum subset, or is deemed semantically equivalent by an LLM equivalence checker for the original benchmark.

Input / output format

Input: Open-ended: '{category} question text'. Multiple-choice: question text followed by options A) through D). VQA v2.0: image paired with a yes/no question.

Output: Open-ended: 'Answer: XXX'. Multiple-choice: 'Answer: X' (where X is A, B, C, or D). VQA v2.0: direct text answer.

Scoring recipe

def calculate_accuracy(predictions, valid_answers):
    correct = 0
    for pred, valids in zip(predictions, valid_answers):
        if pred.strip() in [v.strip() for v in valids]:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Open-ended QA benchmarks often have multiple valid answers; failing to enumerate all valid responses or use an equivalence checker leads to artificially low accuracy.
• Original benchmarks contain significant label noise and ambiguity; evaluating on uncurated versions obscures true model capabilities and inflates perceived failures.

Evidence (verbatim from paper)

Their accuracy metric then assigns a score to a model prediction based on the overlap between the prediction and these ten labels.

Citation

@misc{vendrow2025platinum,
  title={Do Large Language Model Benchmarks Test Reliability?},
  author={Vendrow et al. (2025)},
  year={2025},
  note={arXiv:2502.03461}
}

• arXiv: 2502.03461