Meddocbench Eval

meddocbench-eval

Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning — Wang et al. (2025) (arXiv:2509.19090, 2025)

What this evaluates

Evaluates a model's ability to parse and reason over real-world medical documents, including laboratory test reports and general medical documents, through tasks like table extraction, simple/complex QA, and free-form scoring.

Datasets

• MedDocBench — total ?; splits: Hard test set (-1)

Metrics

• Field-level micro Precision/Recall/F1 & Macro-Doc F1 (primary) — range: [0, 1]
  • Bipartite matching between predicted and gold entry_names, then field-level correctness for entry_name, result, reference, and unit. Reports micro P/R/F1 at field level and macro-averaged scores at document level.
• Exact-match accuracy — range: [0, 1]
  • Computed after canonicalization for simple QA subtasks.
• LLM judge mean score — range: [0, 1]
  • Free-form answers scored by an LLM judge on a continuous scale s∈[0,1]; mean score reported.

Input / output format

Input: Medical document images (laboratory test reports or general medical documents) paired with parsing instructions or questions.

Output: Markdown tables for full parsing; JSON containing result, reference, and abnormality label for complex QA; free-form text for simple QA and GMD.

Scoring recipe

def score_meddocbench(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        if pred.task == 'full_parsing':
            pred_table = canonicalize(pred.markdown)
            matches = bipartite_match(pred_table, gold.table)
            scores.append(f1_score(matches))
        elif pred.task == 'simple_qa':
            scores.append(exact_match(canonicalize(pred.text), gold.text))
        elif pred.task == 'gmd':
            scores.append(llm_judge_score(pred.text, gold.text, pred.question))
    return mean(scores)

Common pitfalls

• Canonicalization is strictly required before exact-match or rule-based matching; skipping it causes severe score drops.
• LLM judge is used as a fallback for rule-based matching failures, introducing potential non-determinism and prompt sensitivity.
• Macro-averaged document-level scores can be skewed by rare entry types in sparse tables.

Evidence (verbatim from paper)

LTR: All predictions are canonicalized as a preprocessing step. For full parsing, models output a Markdown table; we perform bipartite matching between predicted and gold entry_names, then assess field-level correctness for entry_name, result, reference, and unit. We report micro Precision/Recall/F1 at the field level and macro-averaged scores at the document (image) level. For complex QA, models return JSON containing the result, reference, and an abnormality label; evaluation follows the same matching procedure with P/R/F1 reporting. For simple QA, we compute exact-match accuracy after canonicalization. In all LTR subtasks, an LLM judge is used as a fallback when rule-based matching failed. GMD: Free-form answers are scored by an LLM judge given the question, the predicted answer, and the gold answer. The judge produces a continuous score $s\in[0,1]$; we report the mean score.

Citation

@misc{wang2025citrusv,
  title={Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.19090}
}

• arXiv: 2509.19090