Medmax Eval

medmax-eval

MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants — Bansal et al. (2024) (arXiv:2412.12661, 2024)

What this evaluates

Evaluates biomedical multimodal foundation models across visual question answering, image captioning, image generation, visual chat, and interleaved text-image generation. It probes the model's ability to understand medical images, reason over clinical reports, and generate clinically grounded multimodal responses.

Datasets

• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• PathVQA — total ?; splits: test (-1)
• QuiltVQA — total ?; splits: test (-1)
• PMC-VQA — total ?; splits: test (-1)
• PathMMU — total ?; splits: validation (-1)
• ProbMed — total ?; splits: test (-1)
• OmniMedVQA — total 1000; splits: hidden (1000)
• PMC-OA — total 800; splits: test (800)
• MIMIC-CXR — total 400; splits: test (400)
• Quilt-1M — total 400; splits: unseen (400)
• LLaVA-Med — total 193; splits: test (193)
• MedMax-Instruct — total 500; splits: hidden (500)

Metrics

• Accuracy (EM) (primary) — range: [0, 1]
  • Exact match between the model's predicted answer and the ground-truth answer. Returns 1 if they match exactly, 0 otherwise. Averaged over all instances.
• Accuracy (LLM) — range: [0, 1]
  • An LLM (GPT-4o-mini) compares the predicted answer against the ground-truth answer and assigns a score of 0 or 1 based on reliability. Averaged over open-ended questions.
• BioMedCLIPScore — range: [0, 1]
  • Computes the similarity score from the BioMedCLIP model between the input image and the predicted caption (or generated image).
• LLM score — range: [0, 1]
  • An LLM scores the predicted answer out of 10 conditioned on the question and image context. The final metric is the average relative prediction score: score(predicted) / score(reference).
• Image-Image BioMedCLIPScore — range: [0, 1]
  • Computes the image-image similarity score from the BioMedCLIP model between the generated image and the reference image.

Input / output format

Input: Multimodal input consisting of a medical image paired with a text prompt (question, captioning instruction, or generation query).

Output: Text response (answer, caption, or chatbot reply) or interleaved text-image response for generation tasks.

Scoring recipe

def score_vqa(preds, golds, mode):
    scores = []
    for p, g in zip(preds, golds):
        if mode == 'EM':
            scores.append(1.0 if p.strip() == g.strip() else 0.0)
        elif mode == 'LLM':
            scores.append(llm_compare(p, g)) # GPT-4o-mini returns 0 or 1
    return sum(scores) / len(scores)

def score_caption_gen(image, pred_caption):
    return biomedclip_similarity(image, pred_caption)

def score_chatbot(pred_ans, ref_ans, question, image):
    pred_score = llm_score(pred_ans, question, image) # out of 10
    ref_score = llm_score(ref_ans, question, image) # out of 10
    return pred_score / ref_score

Common pitfalls

• Open-ended VQA answers are subjective; using exact match instead of LLM evaluation unfairly penalizes semantically correct but differently phrased answers.
• The LLM score for chatbot/generation is a relative ratio (predicted/reference), not an absolute score, which can amplify reference quality differences.
• Contamination must be explicitly checked: ensure no exact matches between instruction-tuning data and evaluation splits, as noted in the paper's contamination analysis.

Evidence (verbatim from paper)

Subsequently, we compute the BioMedCLIPScore to assess the closeness between the input image (caption) and predicted caption (image).

Citation

@misc{bansal2024medmax,
  title={MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants},
  author={Bansal et al. (2024)},
  year={2024},
  note={arXiv:2412.12661}
}

• arXiv: 2412.12661