rexrank-eval
ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation — Zhang et al. (2024) (arXiv:2411.15122, 2024)
What this evaluates
Evaluates AI models' ability to generate accurate and clinically relevant radiology reports from chest X-ray images. It assesses both linguistic quality and clinical entity extraction/alignment across diverse clinical datasets.
Datasets
- ReXGradient — total 10000; splits: test (-1)
- MIMIC-CXR — total ?; splits: test (-1)
- IU X-ray — total ?; splits: test (-1)
- CheXpert Plus — total ?; splits: test (-1)
Metrics
1/RadCliQ-v1 (primary) — range: other
- Inverse of the RadCliQ-v1 clinical entity alignment score. Higher values indicate better clinical accuracy. Computed as 1 / RadCliQ_v1_score.
BLEU-2 — range: [0, 1]
- Standard 2-gram BLEU score measuring n-gram overlap between generated and reference reports.
BertScore — range: [0, 1]
- F1 score computed using contextual embeddings from a BERT model to measure semantic similarity.
SembScore — range: [0, 1]
- Semantic similarity score based on sentence embeddings, measuring contextual alignment.
RadGraph — range: [0, 1]
- F1 score measuring the overlap of extracted radiological entities and relations between prediction and ground truth.
RaTEScore — range: [0, 1]
- Radiology Text Embedding score measuring semantic similarity using domain-specific embeddings.
GREEN — range: [0, 1]
- Generative Radiology Evaluation metric assessing clinical correctness and completeness of generated reports.
1/FineRadScore — range: other
- Inverse of the FineRadScore, which evaluates fine-grained clinical correctness. Higher values indicate better performance.
Input / output format
Input: Chest X-ray image(s) (and optionally clinical metadata).
Output: Radiology report text, generated either as 'Findings' only or 'Findings + Impression'.
Scoring recipe
def compute_metrics(pred_report, gold_report):
radcliq = compute_radcliq_v1(pred_report, gold_report)
bleu2 = compute_bleu2(pred_report, gold_report)
bertscore = compute_bertscore(pred_report, gold_report)
sembscore = compute_sembscore(pred_report, gold_report)
radgraph = compute_radgraph(pred_report, gold_report)
rtescore = compute_rtescore(pred_report, gold_report)
green = compute_green(pred_report, gold_report)
fineradscore = compute_fineradscore(pred_report, gold_report)
return {
'1/RadCliQ-v1': 1.0 / radcliq,
'BLEU-2': bleu2,
'BertScore': bertscore,
'SembScore': sembscore,
'RadGraph': radgraph,
'RaTEScore': rtescore,
'GREEN': green,
'1/FineRadScore': 1.0 / fineradscore
}
Common pitfalls
- The primary metric is inverted (1/RadCliQ-v1), so higher values indicate better performance, contrary to standard RadCliQ scores.
- Models are evaluated on two distinct tasks: 'Findings' only and 'Findings + Impression', which should not be mixed when comparing results.
- Confidence intervals (95% CI) are reported alongside mean scores, indicating statistical variance that should be considered when ranking models.
Evidence (verbatim from paper)
Table 1, 2, 3 and 4 summarize the performance of various medical report generation models across four different datasets: ReXGradient, MIMIC-CXR, IU X-ray and CheXpert Plus. Among these, MedVersa demonstrates superior performance, achieving the best 1/RadCliQ-v1 scores on ReXGradient (1.01 ± 0.01), MIMIC-CXR (1.10 ± 0.02), and IU X-ray (1.46 ± 0.03) on the findings section.
Citation
@misc{zhang2024rexrank,
title={ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation},
author={Zhang et al. (2024)},
year={2024},
note={arXiv:2411.15122}
}
1---2name: rexrank-eval3description: Evaluates AI models' ability to generate accurate and clinically relevant radiology reports from chest X-ray images. It assesses both linguistic quality and clinical entity extraction/alignment across diverse clinical datasets. Use when the user wants to benchmark on ReXGradient, MIMIC-CXR, IU X-ray, CheXpert Plus, or asks about evaluating this task. Reports 1/RadCliQ-v1.4metadata:5 source: https://skillmd.com/skills/qhjqhj00/rexrank-eval6 skill_kind: dataset_eval7 source_arxiv: 2411.151228 bibtex_key: zhang2024rexrank9 confidence: high10---1112# rexrank-eval1314> ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation — Zhang et al. (2024) (arXiv:2411.15122, 2024)1516## What this evaluates1718Evaluates AI models' ability to generate accurate and clinically relevant radiology reports from chest X-ray images. It assesses both linguistic quality and clinical entity extraction/alignment across diverse clinical datasets.1920## Datasets2122- **ReXGradient** — total 10000; splits: test (-1)23- **MIMIC-CXR** — total ?; splits: test (-1)24- **IU X-ray** — total ?; splits: test (-1)25- **CheXpert Plus** — total ?; splits: test (-1)2627## Metrics2829- `1/RadCliQ-v1` **(primary)** — range: other30 - Inverse of the RadCliQ-v1 clinical entity alignment score. Higher values indicate better clinical accuracy. Computed as 1 / RadCliQ_v1_score.31- `BLEU-2` — range: [0, 1]32 - Standard 2-gram BLEU score measuring n-gram overlap between generated and reference reports.33- `BertScore` — range: [0, 1]34 - F1 score computed using contextual embeddings from a BERT model to measure semantic similarity.35- `SembScore` — range: [0, 1]36 - Semantic similarity score based on sentence embeddings, measuring contextual alignment.37- `RadGraph` — range: [0, 1]38 - F1 score measuring the overlap of extracted radiological entities and relations between prediction and ground truth.39- `RaTEScore` — range: [0, 1]40 - Radiology Text Embedding score measuring semantic similarity using domain-specific embeddings.41- `GREEN` — range: [0, 1]42 - Generative Radiology Evaluation metric assessing clinical correctness and completeness of generated reports.43- `1/FineRadScore` — range: other44 - Inverse of the FineRadScore, which evaluates fine-grained clinical correctness. Higher values indicate better performance.4546## Input / output format4748**Input**: Chest X-ray image(s) (and optionally clinical metadata).4950**Output**: Radiology report text, generated either as 'Findings' only or 'Findings + Impression'.5152## Scoring recipe5354```python55def compute_metrics(pred_report, gold_report):56 radcliq = compute_radcliq_v1(pred_report, gold_report)57 bleu2 = compute_bleu2(pred_report, gold_report)58 bertscore = compute_bertscore(pred_report, gold_report)59 sembscore = compute_sembscore(pred_report, gold_report)60 radgraph = compute_radgraph(pred_report, gold_report)61 rtescore = compute_rtescore(pred_report, gold_report)62 green = compute_green(pred_report, gold_report)63 fineradscore = compute_fineradscore(pred_report, gold_report)64 return {65 '1/RadCliQ-v1': 1.0 / radcliq,66 'BLEU-2': bleu2,67 'BertScore': bertscore,68 'SembScore': sembscore,69 'RadGraph': radgraph,70 'RaTEScore': rtescore,71 'GREEN': green,72 '1/FineRadScore': 1.0 / fineradscore73 }74```7576## Common pitfalls7778- The primary metric is inverted (1/RadCliQ-v1), so higher values indicate better performance, contrary to standard RadCliQ scores.79- Models are evaluated on two distinct tasks: 'Findings' only and 'Findings + Impression', which should not be mixed when comparing results.80- Confidence intervals (95% CI) are reported alongside mean scores, indicating statistical variance that should be considered when ranking models.8182## Evidence (verbatim from paper)8384> Table 1, 2, 3 and 4 summarize the performance of various medical report generation models across four different datasets: ReXGradient, MIMIC-CXR, IU X-ray and CheXpert Plus. Among these, MedVersa demonstrates superior performance, achieving the best 1/RadCliQ-v1 scores on ReXGradient (1.01 ± 0.01), MIMIC-CXR (1.10 ± 0.02), and IU X-ray (1.46 ± 0.03) on the findings section.8586## Citation8788```bibtex89@misc{zhang2024rexrank,90 title={ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation},91 author={Zhang et al. (2024)},92 year={2024},93 note={arXiv:2411.15122}94}95```9697- arXiv: 2411.15122