Head QA V2 Eval

head-qa-v2-eval

HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning — Correa-Guillén et al. (2025) (arXiv:2511.15355, 2025)

What this evaluates

This benchmark evaluates large language models on complex medical reasoning using real Spanish medical licensing exam questions. It probes domain-specific knowledge retention, cross-lingual generalization, and the effectiveness of various inference strategies like prompting, retrieval-augmented generation, and log-probability selection.

Datasets

• HEAD-QA v2 — total 12751; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered questions out of the total number of questions.
• normalized exam score — range: [0, 1]
  • Calculated as (correct_answers - wrong_answers / 3) / total_items, following the official Spanish medical exam penalty scheme where three wrong answers cancel one correct answer.
• unanswered ratio — range: [0, 1]
  • Fraction of questions for which the model produces no valid response.

Input / output format

Input: Multiple-choice medical question with options, provided in either English or Spanish. Inputs may optionally include few-shot examples, retrieved context passages (RAG), or be presented as isolated options for log-probability scoring.

Output: A single selected option (e.g., A, B, C, D) or a generated text response containing the answer. For log-probability evaluation, the model outputs log-probabilities for each option.

Scoring recipe

def compute_metrics(predictions, golds, total_items):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    wrong = sum(1 for p, g in zip(predictions, golds) if p != g and p is not None)
    unanswered = sum(1 for p in predictions if p is None)
    accuracy = correct / len(golds)
    exam_score = (correct - wrong / 3) / total_items
    unanswered_ratio = unanswered / len(golds)
    return {'accuracy': accuracy, 'normalized_exam_score': exam_score, 'unanswered_ratio': unanswered_ratio}

Common pitfalls

• Chain-of-thought (CoT) prompting consistently reduces accuracy and increases non-response rates in this domain, contrary to typical LLM behavior.
• Retrieval-augmented generation (RAG) does not reliably improve performance; retrieved context is often noisy or weakly relevant, and models may ignore it in favor of internal knowledge.
• Log-probability selection yields lower scores than prompting because it evaluates options independently, preventing joint elimination reasoning.

Evidence (verbatim from paper)

Performance is evaluated using three metrics:(1) accuracy, the proportion of correct answers; (2) the normalized exam score, based on the official Spanish medical exam scheme (three wrong answers cancel one correct) and normalized by total items; and (3) the unanswered ratio, the fraction of questions with no valid response.

Citation

@misc{correaguillen2025headqav2,
  title={HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning},
  author={Correa-Guillén et al. (2025)},
  year={2025},
  note={arXiv:2511.15355}
}

• arXiv: 2511.15355