Chexbench Eval

chexbench-eval

A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation — Chen et al. (2024) (arXiv:2401.12208, 2024)

What this evaluates

Evaluates vision-language foundation models on chest X-ray interpretation across two axes: image perception (view classification, disease identification/classification, VQA, reasoning) and textual understanding (findings generation, summarization). It probes clinical reasoning, visual grounding, and medical text generation capabilities.

Datasets

• MIMIC-CXR — total ?; splits: test (-1)
• CheXpert — total ?; splits: test (-1)
• SIIM — total ?; splits: test (-1)
• RSNA — total ?; splits: test (-1)
• OpenI — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• Rad-Restruct — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted options in multiple-choice tasks. Computed as correct predictions divided by total predictions.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence-based recall/precision/f1 score for text generation tasks.
• CheXbert-Score — range: [0, 1]
  • Automated metric measuring factual correctness of generated radiology findings against reference reports using a clinical NLP model.
• RadGraph-Score — range: [0, 1]
  • Automated metric evaluating clinical entity and relation extraction accuracy in generated reports.

Input / output format

Input: A chest X-ray image paired with a text prompt (multiple-choice question with options, or an instruction to generate/summarize findings).

Output: For perception tasks: the selected multiple-choice option. For generation tasks: free-text radiology findings or a concise summary statement.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(predictions)

def compute_generation_metrics(predictions, references):
    # Uses standard implementations for ROUGE-L, CheXbert-Score, RadGraph-Score
    rouge_l = rouge_l_score(predictions, references)
    chexbert = chexbert_score(predictions, references)
    radgraph = radgraph_score(predictions, references)
    return rouge_l, chexbert, radgraph

Common pitfalls

• Data leakage: Several baseline models (e.g., XrayGPT, RadFM) were fine-tuned on datasets included in the benchmark (OpenI, SLAKE), potentially inflating their reported accuracy.
• Log-likelihood selection: The benchmark avoids open-ended generation evaluation for perception tasks by scoring log-likelihoods of multiple-choice options, which may not capture true clinical reasoning or hallucination rates.
• Hard negatives: The image-text reasoning task uses carefully crafted hard negatives that are exceptionally difficult for all evaluated models, making performance gaps small and potentially masking model limitations.

Evidence (verbatim from paper)

In line with prior benchmarks designed for general domain FMs, we use a multiple-choice format, where an image and a question are posed to the FM and multiple options are considered. Since open-ended, free-text outputs from FMs are challenging to evaluate, we instead compute log-likelihood scores associated with each option; the option with the highest score is then selected as the response. For each task, we report accuracy.

Citation

@misc{chen2024chexagent,
  title={A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2401.12208}
}

• arXiv: 2401.12208