qhjqhj00 simpleqa-verified-eval
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge — Haas et al. (2025) (arXiv:2509.07968, 2025)
What this evaluates
This benchmark evaluates an LLM's parametric factuality and internal knowledge recall on short-form questions. It measures whether models can correctly answer factual queries without relying on external search tools or retrieval augmentations.
Datasets
- SimpleQA Verified — total 1000; splits: test (1000)
Metrics
F1-Score(primary) — range: percent- Harmonic mean of Accuracy and Attempted rate, or computed via precision/recall from autorater judgments. Balances correctness with the model's willingness to attempt an answer.
Accuracy— range: percent- Percentage of questions answered correctly out of all questions in the benchmark.
Acc.|Attempted— range: percent- Percentage of correctly answered questions out of only those the model attempted to answer (excluding refusals or heavy hedging).
Input / output format
Input: A single short-form factual question.
Output: A short textual answer, refusal, or hedge. Models are evaluated without external tools.
Scoring recipe
N = len(dataset)
for each question in dataset:
gold = get_gold_answer(question)
pred = model.generate(question, tools=False)
is_correct = autorater_judge(pred, gold) == 'correct'
is_attempted = not is_hedge_or_refusal(pred)
accuracy = sum(is_correct) / N
attempted_rate = sum(is_attempted) / N
acc_attempted = sum(is_correct and is_attempted) / sum(is_attempted)
f1 = 2 * (accuracy * attempted_rate) / (accuracy + attempted_rate) if (accuracy + attempted_rate) > 0 else 0
Common pitfalls
- Enabling external tools or retrieval causes near-perfect scores, invalidating the parametric factuality measurement.
- Models that frequently hedge or refuse to answer (high Hedged rate) will show artificially low Accuracy but may retain high Acc.|Attempted, skewing overall performance interpretation.
- Results on SimpleQA Verified differ significantly from the original SimpleQA due to rigorous de-duplication and ground-truth correction, so cross-benchmark comparisons require caution.
Evidence (verbatim from paper)
In our results, we report the same metrics computed in Wei et al. ([2024a]) and use gpt-4.1-2025-04-14 as an autorater. Additionally, we measure the difference in scores between SimpleQA Verified and SimpleQA on all models. The results of GPT 4o, Claude Opus 4, Claude Sonnet 4 are statistically significantly worse on SimpleQA Verified compared to SimpleQA, whereas o4-mini’s score improves. On average, model performance on SimpleQA Verified is almost exactly the same as on SimpleQA Verified – our cleaning process described in [Section 2] removes erroneous and ambiguous questions which makes the benchmark easier. This is balanced by adversarially selecting a subset of challenging samples in [Section 2.6]. Gemini 2.5 Pro leads SimpleQA and SimpleQA Verified in both Accuracy and Accuracy Given Attempted (Acc.|Attempted) metrics, resulting in the highest F1-Score among frontier models.
Citation
@misc{haas2025simpleqa,
title={SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge},
author={Haas et al. (2025)},
year={2025},
note={arXiv:2509.07968}
}
- arXiv: 2509.07968