Cxreasondial Eval

cxreasondial-eval

CXReasonAgent: Evidence-Grounded Diagnostic Reasoning Agent for Chest X-rays — Lee et al. (2026) (arXiv:2602.23276, 2026)

What this evaluates

Evaluates multi-turn diagnostic reasoning agents on chest X-rays, measuring their ability to identify tasks, extract evidence, maintain coverage, avoid hallucinations, and sustain coherent dialogue success.

Datasets

• CXReasonDial — total 1946; splits: test (-1)

Metrics

• Diagnostic Task Identification (DTI) — range: percent
  • Binary score indicating whether the agent correctly identifies the requested diagnostic task from the user query. Evaluated via LLM-as-a-Judge (Gemini-3-Flash).
• Evidence Type Identification (ETI) — range: percent
  • Binary score indicating whether the agent correctly identifies the requested evidence type. Evaluated via LLM-as-a-Judge.
• Coverage (Cov) — range: percent
  • Binary score assessing whether the response fully addresses the user query, regardless of factual correctness. Evaluated via LLM-as-a-Judge.
• Faithfulness (Faith) (primary) — range: percent
  • Binary score evaluating whether the response is consistent with the request-specific ground-truth evidence used during dataset construction. Evaluated via LLM-as-a-Judge.
• Hallucination (Hall) — range: percent
  • Binary score defined as Coverage=1 and Faithfulness=0. Evaluated via LLM-as-a-Judge.
• Average Dialogue Success (Avg) — range: percent
  • Average ratio of successful turns per dialogue. A turn is successful only if all applicable turn-level metrics are correct.
• Strict Dialogue Success (Strict) — range: percent
  • Proportion of dialogues in which all turns are successful.

Input / output format

Input: Chest X-ray image paired with a multi-turn user query requesting specific diagnostic tasks or evidence.

Output: Textual response (and optionally tool calls/visual overlays for CXReasonAgent) addressing the user query at each dialogue turn.

Scoring recipe

# Turn-level metrics (LLM-as-Judge)
dti = judge(model_out, query, "task_id")
eti = judge(model_out, query, "evidence_type")
cov = judge(model_out, query, "coverage")
faith = judge(model_out, query, "ground_truth_evidence")
hall = 1 if (cov == 1 and faith == 0) else 0

# Dialogue-level metrics
successful_turns = sum(1 for t in dialogue if dti[t] and eti[t] and cov[t] and faith[t])
avg_success = (successful_turns / len(dialogue)) * 100
strict_success = (successful_turns == len(dialogue)) * 100

Common pitfalls

• LVLM baselines exclude turns requesting visual evidence, making their evaluation scope narrower than CXReasonAgent's.
• Faithfulness scores for LVLMs artificially inflate when ground-truth history is provided, as they may copy evidence from previous turns rather than the image.
• LLM-as-a-Judge evaluation introduces potential bias and variability compared to automated exact-match metrics.

Evidence (verbatim from paper)

We evaluate models using the following metrics. 1) Turn-level Metrics. Each metric is evaluated as a binary score at the turn level using an LLM-as-a-Judge (i.e., Gemini-3-Flash). (i) Diagnostic Task Identification and (ii) Evidence Type Identification assess whether the agent correctly identifies, from the user query, the requested diagnostic task and evidence type, respectively. These two metrics are evaluated only for CXReasonAgents, as they assess the tool planning stage. (iii) Coverage assesses whether the response fully addresses the user query, regardless of factual correctness. (iv) Faithfulness evaluates whether the response is consistent with the request-specific ground-truth evidence used during CXReasonDial construction. (v) Hallucination is defined as the case where Coverage=1 and Faithfulness=0. When evaluating LVLMs, turns requesting visual evidence are excluded, as these models do not produce visual evidence. 2) Dialogue-level Metrics. A turn is considered successful only if all applicable turn-level metrics are correct. (i) Average Dialogue Success is the average ratio of successful turns per dialogue. (ii) Strict Dialogue Success is the proportion of dialogues in

Citation

@misc{lee2026cxreasonagent,
  title={CXReasonAgent: Evidence-Grounded Diagnostic Reasoning Agent for Chest X-rays},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2602.23276}
}

• arXiv: 2602.23276