Head QA Eval

head-qa-eval

HEAD-QA: A Healthcare Dataset for Complex Reasoning — Vilares et al. (2019) (arXiv:1906.04701, 2019)

What this evaluates

Evaluates complex reasoning and domain-specific knowledge integration in healthcare by testing models on multi-choice questions derived from real Spanish medical specialization exams. It probes the ability to handle long, context-rich questions requiring cross-domain inference and precise medical knowledge.

Datasets

• HEAD-QA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered questions out of the total number of questions.
• POINTS — range: other
  • A scoring system from the official exams where a correct answer adds 3 points and an incorrect answer subtracts 1 point.

Input / output format

Input: A multi-choice question with multiple answer options, derived from real healthcare specialization exams.

Output: A single selected answer option from the provided choices.

Scoring recipe

def compute_metrics(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    wrong = sum(1 for p, g in zip(predictions, golds) if p != g)
    total = len(golds)
    accuracy = correct / total
    points = (3 * correct) - (1 * wrong)
    return {'accuracy': accuracy, 'POINTS': points}

Common pitfalls

• The POINTS metric penalizes incorrect answers (-1 point), so models that guess randomly will score negatively, unlike standard accuracy.
• Performance heavily depends on question length and domain; medicine questions (MIR) are consistently the hardest due to longer text, while pharmacology (FIR) and biology (BIR) are easier.
• Cross-lingual information retrieval models sometimes outperform in-domain Spanish models, indicating translation quality or retrieval robustness plays a major role.

Evidence (verbatim from paper)

Metrics We use accuracy and a POINTS metric (used in the official exams): a right answer counts 3 points and a wrong one subtracts 1 point.

Citation

@misc{vilares2019headqa,
  title={HEAD-QA: A Healthcare Dataset for Complex Reasoning},
  author={Vilares et al. (2019)},
  year={2019},
  note={arXiv:1906.04701}
}

• arXiv: 1906.04701