Gaps Eval

gaps-eval

GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians — Chen et al. (2025) (arXiv:2510.13734, 2025)

What this evaluates

Evaluates AI clinicians on clinical reasoning depth, answer completeness, robustness to input perturbations, and safety risk mitigation. It probes how well models retrieve, synthesize, and apply evidence-based guidelines under varying cognitive loads and adversarial conditions.

Datasets

• GAPS-NCCN-NSCLC-preview — total 92; splits: test (92); repo https://github.com/AQ-MedAI/MedicalAiBenchEval

Metrics

• GAPS score (normalized rubric-based) (primary) — range: [0, 1]
  • Scores are normalized to [0, 1] using rule-based and rubric-based scoring evaluated by an LLM-as-a-judge ensemble. The score reflects the proportion of correctly satisfied rubric elements against a guideline-anchored standard.
• hit rate — range: [0, 1]
  • Proportion of satisfied positive rubric elements (A1 Must-have, A2 Should-have, A3 Nice-to-have) or avoided negative rubric elements (S2 Near miss, S3 Suboptimal, S4 Critical) out of total rubrics per item.

Input / output format

Input: Clinical question prompts (clean P0 or perturbed P1–P3) derived from NCCN NSCLC guidelines, often accompanied by evidence corpora or knowledge graph structures.

Output: Free-text clinical recommendations or answers addressing the prompt.

Scoring recipe

def score_response(response, gold_rubrics):
    satisfied = 0
    total = len(gold_rubrics)
    for rubric in gold_rubrics:
        if llm_judge_check(response, rubric):  # LLM-as-a-judge verifies alignment
            satisfied += 1
    return satisfied / total if total > 0 else 0.0

# Adequacy hit rate = score_response(response, adequacy_rubrics)
# Safety hit rate = score_response(response, safety_rubrics)
# Overall GAPS score = normalized aggregate across Grounding, Adequacy, Safety, and Perturbation axes

Common pitfalls

• High factual recall (G1/G2) does not translate to complex decision-making (G3/G4), leading to overestimation of clinical reasoning capabilities.
• Models are highly sensitive to adversarial premise perturbations (P3), often aligning with misleading cues rather than critically evaluating them.
• Safety violations (S4) are not uniformly distributed; they spike dramatically under high cognitive load and vary significantly across different LLM architectures.

Evidence (verbatim from paper)

We next evaluate representative state-of-the-art LLMs on the automatically constructed GAPS benchmark GAPS-NCCN-NSCLC-preview using rule-based and rubric-based scoring (normalized to [0,1]), details of which are covered in Section[3.6]. Scores were obtained by an ensemble of LLM-as-a-judge models, with clinician audits on stratified subsets for validation. Performance was analyzed across the four GAPS axes: Grounding levels (G1–G4), Perturbations (P1–P3), and Adequacy and Safety outcomes (hit rates).

Citation

@misc{chen2025gaps,
  title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2510.13734}
}

• arXiv: 2510.13734