Histgen Report Gen Eval

histgen-report-gen-eval

Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning — Liu et al. (2025) (arXiv:2506.17645, 2025)

What this evaluates

Evaluates a vision-language model's ability to generate accurate and clinically relevant histopathology reports from whole slide images (WSIs). It probes lexical overlap, semantic coherence, and medical entity coverage in generated text.

Datasets

• HistGen — total 7690; splits: train (6152), val (769), test (769)

Metrics

• BLEU-4 (primary) — range: [0, 1]
  • Standard n-gram precision with brevity penalty. BLEU-4 computes geometric mean of unigram through 4-gram precisions, penalizing overly short outputs.
• METEOR — range: [0, 1]
  • Harmonic mean of unigram precision and recall, incorporating synonym matching, stemming, and paraphrase matching to better capture semantic similarity than BLEU.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence (LCS) based F-score that measures structural similarity and sentence-level recall/precision between generated and reference text.
• fact_ENT — range: [0, 1]
  • Exact Entity Match Reward from Miura et al. (2021); measures entity coverage by extracting medical entities from both generated and ground-truth reports using BioBERT-v1.1, then computing the ratio of matched entities to ground-truth entities.

Input / output format

Input: A whole slide image (WSI) and optionally retrieved similar WSI-report pairs as in-context examples.

Output: A natural language histopathology report describing the tissue sample.

Scoring recipe

def compute_metrics(gen_report, gt_report):
    # Truncate to first 100 tokens per paper protocol
    gen_tok = gen_report.split()[:100]
    gt_tok = gt_report.split()[:100]
    bleu4 = nltk.bleu([gt_tok], gen_tok, weights=(0.25, 0.25, 0.25, 0.25))
    meteor = calculate_meteor(gen_tok, gt_tok)
    rouge_l = calculate_rouge_l(gen_tok, gt_tok)
    # fact_ENT: BioBERT-v1.1 NER extraction
    gen_ent = ner_model.predict(gen_tok)
    gt_ent = ner_model.predict(gt_tok)
    fact_ent = len(set(gen_ent) & set(gt_ent)) / len(set(gt_ent)) if gt_ent else 0.0
    return {'BLEU-4': bleu4, 'METEOR': meteor, 'ROUGE-L': rouge_l, 'fact_ENT': fact_ent}

Common pitfalls

• Standard NLP metrics (BLEU, METEOR, ROUGE) may not capture domain-specific clinical accuracy or inferential consistency in medical reports.
• Main results are evaluated on truncated outputs (first 100 tokens), which can artificially inflate or deflate length-sensitive metrics.
• fact_ENT scores are only reported for the proposed method, preventing direct cross-method comparison on entity coverage.

Evidence (verbatim from paper)

To fairly compare with existing methods on the HistGen dataset, we adopt BLEU, METEOR, and ROUGE-L as the evaluation metrics. These metrics collectively measure lexical similarity, semantic relevance, and structural coherence between the generated and ground-truth reports. However, they were proposed from the natural language processing perspective and may not well reflect domain entities or inferential consistency Miura et al. ([2021]). To enhance the evaluation, we further show performance in terms of Exact Entity Match Reward ($\text{fact}_{\text{ENT}}$) proposed in Miura et al. ([2021]), which captures the completeness of a generated report by measuring its coverage of entities.

Citation

@misc{liu2025histgenic,
  title={Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.17645}
}

• arXiv: 2506.17645