Histgen Wsi Report Eval

histgen-wsi-report-eval

HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction — Guo et al. (2024) (arXiv:2403.05396, 2024)

What this evaluates

Evaluates a model's ability to generate clinical histopathology reports from gigapixel whole slide images (WSIs). It probes cross-modal alignment between dense visual patches and concise textual descriptions using standard natural language generation metrics.

Datasets

• TCGA WSI-Report — total 7753; splits: train (-1), val (-1), test (-1); repo https://github.com/dddavid4real/HistGen

Metrics

• BLEU-4 (primary) — range: [0, 1]
  • Computes 4-gram precision between predicted and reference reports, penalizing overly short outputs via a brevity penalty. Values range from 0 to 1.
• METEOR — range: [0, 1]
  • Measures unigram precision and recall with stemming and synonym matching, applying a penalty for fragmentation. Values range from 0 to 1.
• ROUGE-L — range: [0, 1]
  • Calculates the F1-score of the longest common subsequence between the prediction and reference, capturing sentence-level fluency. Values range from 0 to 1.

Input / output format

Input: Sequence of visual patch features extracted from a whole slide image (WSI) via a pre-trained encoder (e.g., DINOv2 ViT-L).

Output: Natural language text string representing the histopathology report.

Scoring recipe

import nltk
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
from rouge_score import rouge_scorer

def compute_metrics(pred: str, gold: str) -> dict:
    pred_tok = nltk.word_tokenize(pred)
    gold_tok = nltk.word_tokenize(gold)
    bleu4 = sentence_bleu([gold_tok], pred_tok, smoothing_function=SmoothingFunction().method4)
    scorer = rouge_scorer.RougeScorer(['rougeL'], use_stemmer=True)
    rouge_l = scorer.score(gold, pred)['rougeL'].fmeasure
    return {'bleu4': bleu4, 'rougeL': rouge_l}

Common pitfalls

• WSIs are gigapixel-sized; naive full-image processing is infeasible. Models must use multiple instance learning (MIL) or patch pooling to handle the sequence length.
• Standard NLG metrics (BLEU, ROUGE) measure lexical overlap and may not correlate well with clinical diagnostic accuracy or report quality.
• External dataset results use Monte Carlo cross-validation, so scores should be reported as mean ± standard deviation over multiple runs, not single splits.

Evidence (verbatim from paper)

For report generation, our model is evaluated on the WSI-report dataset (specified in Sec. 2.1), using NLG metrics including BLEU [24], METEOR [11], and ROUGE-L [20].

Citation

@misc{guo2024histgen,
  title={HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction},
  author={Guo et al. (2024)},
  year={2024},
  note={arXiv:2403.05396}
}

• arXiv: 2403.05396