Medlaybench V Eval

medlaybench-v-eval

MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models — Jang et al. (2026) (arXiv:2604.05738, 2026)

What this evaluates

Evaluates the ability of medical vision-language models to align expert clinical terminology with patient-accessible layman language while preserving diagnostic accuracy. It measures lexical overlap, readability, clinical factuality, and zero-shot image-text retrieval performance.

Datasets

• MedLayBench-V — total 79789; splits: train (-1), val (-1), test (9927); repo https://github.com/janghana/MedLayBench-V

Metrics

• Recall@K (R@1, R@5, R@10) (primary) — range: percent
  • Measures retrieval accuracy by checking if the ground-truth match appears within the top-K ranked candidates based on cosine similarity of L2-normalized embeddings.
• BLEU-4, ROUGE-L, METEOR — range: [0, 1]
  • Standard n-gram and sequence overlap metrics measuring structural similarity and lexical overlap between expert and layman captions.
• LENS, RaTEScore, GREEN — range: [0, 1]
  • LENS is a learnable metric for text simplification. RaTEScore and GREEN are model-based metrics designed to detect hallucinations and ensure clinical correctness in radiology reports.
• FKGL, CLI, DCRS, SMOG, FRE — range: other
  • Standard readability formulas quantifying text accessibility, grade level, and reading ease.

Input / output format

Input: Image and text pair (for retrieval) or image with prompt (for captioning).

Output: Ranked list of candidate texts/images (retrieval) or generated natural language caption (captioning).

Scoring recipe

def compute_recall_at_k(sim_matrix, k):
    # sim_matrix: [N, N] cosine similarities between images and texts
    top_k_indices = np.argsort(sim_matrix, axis=1)[:, -k:]
    correct = np.sum(np.diag(top_k_indices) == np.arange(len(sim_matrix)))
    return correct / len(sim_matrix) * 100

Common pitfalls

• Confusing expert vs. layman caption performance, as the benchmark reports both side-by-side.
• Applying fine-tuning or prompt engineering, which violates the zero-shot evaluation protocol.
• Ignoring domain adaptation effects, as general-domain VLMs significantly underperform medical-specific models on this benchmark.

Evidence (verbatim from paper)

To assess whether the simplified text preserves essential semantic information for automated analysis, we evaluate zero-shot text-to-image retrieval performance. We report Recall@K (R@1, R@5, R@10) to measure retrieval accuracy using the generated captions.

Citation

@misc{jang2026medlaybenchv,
  title={MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2604.05738}
}

• arXiv: 2604.05738