Pmc Mi Bench Eval

pmc-mi-bench-eval

From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validation — Chen et al. (2025) (arXiv:2511.22232, 2025)

What this evaluates

Evaluates multi-modal large language models on medical reasoning tasks involving compound figures, single images, and text-only prompts. It probes the model's ability to synthesize cross-modal information, perform clinical diagnosis, and generate accurate medical explanations across diverse imaging modalities and specialties.

Datasets

• PMC-MI-Bench — total ?; splits: test (-1); repo https://github.com/franciszchen/M3LLM

Metrics

• BLEU@4 (primary) — range: [0, 100]
  • 4-gram precision between predicted and reference text, averaged over the dataset.
• ROUGE-L — range: [0, 100]
  • Longest common subsequence recall/precision between predicted and reference text.
• BERTScore — range: [0, 100]
  • Cosine similarity between contextual embeddings of predicted and reference tokens, aggregated via F1.
• STS — range: [0, 100]
  • Semantic Textual Similarity score measuring the degree of equivalence between two sentences.
• Accuracy (primary) — range: [0, 100]
  • Proportion of correctly predicted options in multi-choice VQA tasks.

Input / output format

Input: Single or multiple medical images paired with a question or instruction; text-only QA receives only the text prompt.

Output: Free-text answer for open-ended VQA/QA tasks; selected option letter or text for multi-choice VQA.

Scoring recipe

def score(predictions, golds, task_type):
    if task_type == 'open_ended':
        bleu = nltk.translate.bleu_score.sentence_bleu([g], p, weights=(0.25,0.25,0.25,0.25)) * 100
        rouge = rouge_score(g, p, rouge_types=['rougeL']) * 100
        bert = bert_score.compute([p], [g], lang='en')
        sts = semantic_similarity(g, p) * 100
        return {'BLEU@4': bleu, 'ROUGE-L': rouge, 'BERTScore': bert, 'STS': sts}
    elif task_type == 'multi_choice':
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return {'Accuracy': (correct / len(golds)) * 100}

Common pitfalls

• BLEU@4 and ROUGE-L heavily penalize clinically correct but semantically equivalent paraphrases, potentially underestimating model capability.
• LLM-as-a-judge evaluations using GPT-4o may exhibit length bias, favoring verbose responses over concise correct answers.
• Modality-specific accuracy on public benchmarks like OmniMedVQA can be skewed if the test split is imbalanced across imaging types.

Evidence (verbatim from paper)

We employ a robust suite of evaluation metrics, ranging from accuracy for classification tasks to semantic metrics, including BLEU@4, ROUGE-L, BERTScore, and Semantic Textual Similarity (STS), and LLM-as-a-judge using GPT-4o for open-ended generation.

Citation

@misc{chen2025m3llm,
  title={From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validation},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.22232}
}

• arXiv: 2511.22232