Healthbench Eval

healthbench-eval

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries — Ravichandran et al. (2025) (arXiv:2509.02594, 2025)

What this evaluates

Evaluates LLM responses to realistic clinical queries using a fine-grained, rubric-based scoring system. It measures medical accuracy, instruction following, completeness, context awareness, and safety by assigning positive or negative points to specific behavioral criteria, then normalizing the total to a [0, 1] scale.

Datasets

• HealthBench — total ?; splits: test (-1)

Metrics

• HealthBench Score (primary) — range: [0, 1]
  • Sum of point values for all met criteria (positive for correct/helpful, negative for harmful/misinformation). Normalized by dividing by the sum of absolute point values in the rubric. Finally clipped to [0, 1].

Input / output format

Input: Clinical query/prompt from the user and a predefined rubric containing criteria with associated point values.

Output: Model-generated response to the clinical query, which is evaluated against the rubric to produce a normalized HealthBench Score between 0 and 1.

Scoring recipe

total_points = 0
max_possible_points = sum([abs(p) for (_, p) in rubric])
for (criterion, point_value) in rubric:
    if criterion_is_met(response, criterion):
        total_points += point_value
normalized_score = total_points / max_possible_points
final_score = max(0, min(1, normalized_score))

Common pitfalls

• Point values can be negative, so raw sums may be negative before normalization, but the final score is strictly clipped to [0, 1].
• Normalization divides by the sum of absolute values of all rubric points, not just the positive ones or the maximum possible positive score.
• The criterion_is_met function is not algorithmically defined in the paper, requiring human expert annotation or a separate judge model to evaluate.

Evidence (verbatim from paper)

The scoring algorithm follows these steps: total_points=0; max_possible_points=sum([abs(p)for(_,p)inrubric]); for(criterion,point_value)inrubric: ifcriterion_is_met(response,criterion): total_points+=point_value; normalized_score=total_points/max_possible_points; final_score=max(0,min(1,normalized_score)). Normalized HB (HealthBench) Score: HB Score = total_points / max_possible_points.

Citation

@misc{ravichandran2025healthbench,
  title={OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries},
  author={Ravichandran et al. (2025)},
  year={2025},
  note={arXiv:2509.02594}
}

• arXiv: 2509.02594