Rexxkg Eval

rexxkg-eval

Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs — Zhang et al. (2024) (arXiv:2408.14397, 2024)

What this evaluates

Evaluates the medical knowledge understanding and entity/relation coverage of AI-generated chest X-ray radiology reports by comparing structured knowledge graphs extracted from generated text against ground-truth clinical reports. It specifically probes whether models capture nuanced anatomical relationships, medical devices, and quantified measurements beyond surface-level lexical overlap.

Datasets

• CheXpert Plus — total 223228; splits: benchmark (24086), intra-dataset-baseline (24085)
• MIMIC-CXR — total 377110; splits: intra-dataset-baseline (24085)

Metrics

• ReXKG-NSC (primary) — range: [0, 1]
  • Node Similarity Coverage: Measures the overlap and similarity of extracted entity nodes (anatomy, disorder, concept, device, procedure, size) between the generated report's knowledge graph and the ground-truth clinical report's graph.
• ReXKG-AMS — range: [0, 1]
  • Edge Distribution Similarity: Evaluates the distributional similarity of relation edges (suggestive of, located at, modify) between the generated and ground-truth knowledge graphs.
• ReXKG-SCS — range: [0, 1]
  • Subgraph Coverage Score: Assesses how well the generated report covers important subgraphs (k=2 nodes, top 10% by importance) from the ground-truth knowledge graph.

Input / output format

Input: Chest X-ray image (and optionally clinical prompt) for report generation models; ground-truth radiology reports for knowledge graph construction.

Output: Free-text radiology report (specifically the findings section).

Scoring recipe

# Extract entities and relations from ground truth (GT) and generated reports using PURE/BERT pipeline
gt_entities, gt_relations = extract_kg(gt_report)
gen_entities, gen_relations = extract_kg(gen_report)

# Compute ReXKG-NSC (Node Similarity Coverage)
nsc = len(set(gt_entities) & set(gen_entities)) / len(set(gt_entities))

# Compute ReXKG-AMS (Edge Distribution Similarity)
ams = distribution_similarity(gt_relations, gen_relations)

# Compute ReXKG-SCS (Subgraph Coverage Score)
gt_subgraphs = get_top_k_subgraphs(gt_entities, gt_relations, k=2, top_pct=0.1)
scs = coverage_score(gt_subgraphs, gen_entities, gen_relations)

return {"ReXKG-NSC": nsc, "ReXKG-AMS": ams, "ReXKG-SCS": scs}

Common pitfalls

• Relies on a fixed IE pipeline (PURE + BERT) that may fail to extract rare or complex medical entities, artificially deflating scores.
• Evaluates only the 'findings' section, ignoring impressions, indications, or other structured report components.
• Ground-truth knowledge graphs are constructed from a single dataset (CheXpert Plus I), limiting assessment of cross-institutional generalization.

Evidence (verbatim from paper)

It proposes three novel metrics—ReXKG-NSC (node similarity), ReXKG-AMS (edge distribution), and ReXKG-SCS (subgraph coverage)—to assess model understanding beyond surface-level report similarity.

Citation

@misc{zhang2024uncovering,
  title={Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2408.14397}
}

• arXiv: 2408.14397