Smmile Eval

smmile-eval

SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning — Rieff et al. (2025) (arXiv:2506.21355, 2025)

What this evaluates

Evaluates the ability of multimodal large language models (MLLMs) to perform in-context learning (ICL) in medical domains. It probes how effectively models leverage provided image-question-answer demonstrations to answer new clinical queries, while also measuring robustness to irrelevant examples, recency bias, and the gap between automated and expert clinical judgment.

Datasets

• SMMILE — total 111; splits: test (111)
• SMMILE++ — total 1038; splits: test (1038)

Metrics

• Exact Match (EM) — range: percent
  • Binary score (0 or 100) indicating whether the model's generated answer exactly matches the ground-truth answer after normalization for formatting, punctuation, and capitalization.
• LLM-as-a-Judge (primary) — range: percent
  • A text-only LLM (Llama3.3 70B) evaluates the model's generation against the ground-truth answer, outputting a binary judgment (0 for incorrect, 1 for correct). The final score is the percentage of outputs judged correct.
• MCQA Accuracy — range: percent
  • Percentage of multiple-choice questions where the model correctly selects the ground-truth option.
• Expert Rating — range: percent
  • Binary rating (correct/incorrect) assigned by clinical experts to model responses. Score is the percentage of responses rated correct.

Input / output format

Input: A system message, a set of multimodal in-context examples (image-question-answer triplets), and a query consisting of a medical image and a corresponding question.

Output: Free-text response (up to 512 tokens) for open-ended tasks, or a selected option letter/text for multiple-choice questions.

Scoring recipe

def compute_em(pred, gold):
    pred_norm = normalize_text(pred) # handles formatting, punctuation, capitalization
    return 100 if pred_norm == normalize_text(gold) else 0

def compute_llm_judge(pred, gold):
    prompt = f'Ground truth: {gold}\nModel output: {pred}\nJudge accuracy (0/1):'
    judge = llama3_3_70b.generate(prompt)
    return 100 if judge.strip() == '1' else 0

def compute_mcqa(pred, options):
    return 100 if pred in options else 0

Common pitfalls

• LLM-as-a-Judge can be overly lenient in ICL settings, accepting responses that merely match the format/phrasing of in-context examples without ensuring clinical adequacy.
• In-context learning can degrade performance if irrelevant examples are included or due to recency bias, where placing the most relevant example last significantly boosts scores.
• Exact Match (EM) is extremely strict and often yields much lower scores than LLM-as-a-Judge or MCQA, masking actual clinical reasoning capability.

Evidence (verbatim from paper)

For open-ended evaluations, we evaluate MLLM-generated outputs using two metrics. First, the Exact Match (EM) metric counts a model generation as correct (score of 100) if it exactly matches the ground-truth answer, and incorrect (score of 0) otherwise. During evaluation, answers are normalized to account for minor variations in formatting, punctuation, and capitalization before comparison. Second, the LLM-as-a-Judge approach provides a text-only LLM (Llama3.3 70B) with both the model generation and the ground-truth answer; the model is then prompted to evaluate accuracy. The LLM provides a binary judgment (0 for incorrect, 1 for correct) for each generated output, and the final score represents the percentage of outputs judged as correct.

Citation

@misc{rieff2025smmile,
  title={SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning},
  author={Rieff et al. (2025)},
  year={2025},
  note={arXiv:2506.21355}
}

• arXiv: 2506.21355