Regradient 160k Eval

regradient-160k-eval

SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning — Gulluk et al. (2026) (arXiv:2604.09887, 2026)

What this evaluates

Evaluates the ability of vision-language models to generate accurate and semantically rich chest X-ray radiology reports from medical images. It tests both lexical overlap and clinical semantic alignment of the generated findings and impressions against ground-truth reports.

Datasets

• ReXGradient-160K — total 160000; splits: train (-1), test (-1)

Metrics

• COMET (primary) — range: [0, 100]
  • Semantic similarity metric that penalizes unnecessary, irrelevant, and repetitive sentences. Scores are averaged across the test set.
• BERTScore-F1 — range: [0, 1]
  • Contextual embedding-based similarity metric evaluating semantic alignment between generated and reference texts by balancing precision and recall.
• Sentence BLEU-1 — range: [0, 100]
  • Measures unigram n-gram overlap between generated and reference sentences.
• CheXbert-14 F1 — range: [0, 100]
  • F1 score for 14 clinical findings extracted from radiology reports using the CheXbert pipeline.
• RadGraph-F1 — range: [0, 100]
  • F1 score for finding-modality-attribute triples extracted from reports using the RadGraph pipeline.

Input / output format

Input: Chest X-ray image (224x224 pixels) paired with a ground-truth radiology report (findings and impression sections).

Output: Generated radiology report text (findings and/or impression sections).

Scoring recipe

def compute_metrics(predictions, references):
    comet_scores = [comet_model.predict(pred, ref) for pred, ref in zip(predictions, references)]
    bert_scores = [bertscore_f1(pred, ref) for pred, ref in zip(predictions, references)]
    bleu_scores = [sentence_bleu_1(pred, ref) for pred, ref in zip(predictions, references)]
    chexbert_f1 = compute_chexbert_f1(predictions, references)
    radgraph_f1 = compute_radgraph_f1(predictions, references)
    return {
        'COMET': np.mean(comet_scores),
        'BERTScore-F1': np.mean(bert_scores),
        'Sentence BLEU-1': np.mean(bleu_scores),
        'CheXbert-14 F1': chexbert_f1,
        'RadGraph-F1': radgraph_f1
    }

Common pitfalls

• Corpus BLEU scores are explicitly noted as biased in favor of longer outputs and may be artificially inflated by the enrichment method adding sentences from the dataset itself.
• The test dataset is never enriched during evaluation; enrichment is only applied during training/inference batch loading, so metrics must be computed on raw test references.
• CheXbert and RadGraph scores can vary significantly across different base models, requiring careful per-model reporting rather than just averaging.

Evidence (verbatim from paper)

One of the most important metrics is COMET score in our case, it is because our method involves adding new sentences to the existings sentences for each sample. As COMETis a metric that punishes unnecessary, irrelevant and repetitive sentences, we might expect that our model would perform worse than the baseline training if the added sentences were noisy.

Citation

@misc{gulluk2026sem enrich,
  title={SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning},
  author={Gulluk et al. (2026)},
  year={2026},
  note={arXiv:2604.09887}
}

• arXiv: 2604.09887