Medebench Eval

medebench-eval

MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing — Liu et al. (2025) (arXiv:2506.01921, 2025)

What this evaluates

Evaluates the reliability and clinical appropriateness of text-guided medical image editing models. It probes anatomical localization precision, preservation of surrounding clinical context, and overall visual realism across diverse medical imaging modalities and anatomical regions.

Datasets

• MedEBench — total 1182; splits: test (-1)

Metrics

• GPT-4o Editing Accuracy (primary) — range: [0, 1]
  • LLM-based scoring where GPT-4o evaluates whether the edited image correctly implements the text instruction relative to the original. Scores are scaled from 0–10 to 0–1.
• Masked SSIM (primary) — range: [0, 1]
  • Structural Similarity Index computed only on the region-of-interest (ROI) mask to measure how well the surrounding clinical context is preserved during editing. Scaled to [0, 1].
• GPT-4o Visual Quality (primary) — range: [0, 1]
  • LLM-based scoring where GPT-4o rates the photorealism and clinical plausibility of the edited image. Scores are scaled from 0–10 to 0–1.
• FID — range: other
  • Fréchet Inception Distance measuring the distributional distance between real and generated medical images in a feature space. Lower values indicate better distributional match.

Input / output format

Input: A medical image (e.g., X-ray, CT, dermoscopy) paired with a natural language editing instruction (e.g., 'remove polyp', 'straighten spine').

Output: A single edited medical image generated by the model.

Scoring recipe

def evaluate_model(model, dataset):
    ea_scores, cp_scores, vq_scores = [], [], []
    for img, prompt, roi_mask in dataset:
        edited = model.generate(img, prompt)
        # Context Preservation
        cp = masked_ssim(img, edited, roi_mask)
        # Editing Accuracy & Visual Quality (LLM-based)
        ea = normalize_llm_score(gpt4o.evaluate(edited, img, prompt, expected_change), 0, 10)
        vq = normalize_llm_score(gpt4o.evaluate(edited, prompt, quality_prompt), 0, 10)
        ea_scores.append(ea)
        cp_scores.append(cp)
        vq_scores.append(vq)
    return {
        'EA': mean(ea_scores),
        'CP': mean(cp_scores),
        'VQ': mean(vq_scores)
    }

Common pitfalls

• FID is computed at the distribution level and cannot be used for instance-level correlation with human ratings or per-sample metric reporting.
• Models often trade off visual quality for context preservation; high VQ scores may indicate over-editing or repainting of large regions rather than precise localized changes.
• In-context learning with few-shot demonstrations frequently degrades performance on fine-grained medical edits due to model confusion between examples and test inputs.

Evidence (verbatim from paper)

EA = GPT-4o Editing Accuracy_Desc, CP = Masked SSIM, VQ = GPT-4o Visual Quality_Desc. EA and VQ scaled from 0–10 to 0–1.

Citation

@misc{liu2025medebench,
  title={MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.01921}
}

• arXiv: 2506.01921