Cxreasonbench Eval

cxreasonbench-eval

CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays — Lee et al. (2025) (arXiv:2505.18087, 2025)

What this evaluates

Evaluates multi-stage structured diagnostic reasoning in chest X-rays, probing a model’s ability to perform visual grounding, anatomical segmentation, quantitative measurement derivation, and clinical threshold application. It tests whether models can consistently link abstract diagnostic criteria with accurate visual interpretation across direct and guided reasoning paths.

Datasets

• CXReasonBench — total ?; splits: test (-1); repo https://github.com/ttumyche/CXReasonBench

Metrics

• Completion (primary) — range: percent
  • Percentage of cases where the model successfully completes all reasoning stages. Computed using the Wilson score interval to account for varying numbers of attempts across models.
• Depth — range: [0, 4]
  • Average number of reasoning stages the model reaches across the benchmark. Reflects progression through the multi-stage pipeline.
• Alignment — range: percent
  • Percentage of cases where the model’s initial and final decisions agree and are both correct. Computed using the Wilson score interval.
• Consistency — range: percent
  • Percentage of cases where the calculated value returned in the final stage falls within the range selected in the preceding stage. Computed using the Wilson score interval.

Input / output format

Input: Chest X-ray images accompanied by prompts requesting structured diagnostic reasoning. Path 1 provides direct prompts for anatomical segmentation, measurement, and diagnosis. Path 2 provides guided prompts with explicit visual landmarks and computational steps.

Output: A structured sequence of reasoning steps including anatomical region selection, intermediate calculated measurements/values, and a final diagnostic decision. Models must return the calculated value alongside the final decision.

Scoring recipe

def score(predictions, gold):
    completion = [1 if p['stages_reached'] == gold['total_stages'] else 0 for p in predictions]
    depth = sum(p['stages_reached'] for p in predictions) / len(predictions)
    aligned = [1 if p['initial_decision'] == p['final_decision'] == gold['correct_decision'] else 0 for p in predictions]
    consistent = [1 if gold['preceding_range'][0] <= p['final_value'] <= gold['preceding_range'][1] else 0 for p in predictions]
    return {
        'Completion': wilson_score_mean(completion),
        'Depth': depth,
        'Alignment': wilson_score_mean(aligned),
        'Consistency': wilson_score_mean(consistent)
    }

Common pitfalls

• Models often bypass structured measurement reasoning and rely on visual heuristics or shortcuts, leading to high decision accuracy but low consistency scores.
• The Alignment metric is only definitive for tasks with standardized thresholds; for other tasks, initial decisions rely on implicit thresholds, making alignment scores less reliable.
• Performance heavily depends on structural saliency and reference line clarity, so overall averages can mask severe failures on less salient anatomical structures.

Evidence (verbatim from paper)

Final Stage Completion (reported as Completion) measures how often the model successfully completes the reasoning process through to the final stage across the benchmark. Average Reasoning Depth (Depth) indicates the average number of stages the model reaches throughout the benchmark, reflecting its ability to progress through the reasoning pipeline. Decision Alignment (Alignment) applies only to Path 1 and Re-evaluated Path 1, measuring the agreement between the model’s initial and final decisions, and counting them as aligned only when both are correct. Measurement Consistency (Consistency) For measurement-type tasks, the model is instructed to return the calculated value in the preceding stage along with its final decision in the final stage (i.e., Stages 3 and 4 for Path 1, and Stages 2 and 3 for Path 2). We evaluate consistency by checking whether this value falls within the range selected in the preceding stage.

Citation

@misc{lee2025cxreasonbench,
  title={CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2505.18087}
}

• arXiv: 2505.18087