# Medlaybench V Eval

> Evaluates the ability of medical vision-language models to align expert clinical terminology with patient-accessible layman language while preserving diagnostic accuracy. It measures lexical overlap, readability, clinical factuality, and zero-shot image-text retrieval performance. Use when the user wants to benchmark on MedLayBench-V, or asks about evaluating this task. Reports Recall@K (R@1, R@5, R@10).

- Skill: `qhjqhj00/medlaybench-v-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/medlaybench-v-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/medlaybench-v-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/medlaybench-v-eval

---


# medlaybench-v-eval

> MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models — Jang et al. (2026) (arXiv:2604.05738, 2026)

## What this evaluates

Evaluates the ability of medical vision-language models to align expert clinical terminology with patient-accessible layman language while preserving diagnostic accuracy. It measures lexical overlap, readability, clinical factuality, and zero-shot image-text retrieval performance.

## Datasets

- **MedLayBench-V** — total 79789; splits: train (-1), val (-1), test (9927); repo https://github.com/janghana/MedLayBench-V

## Metrics

- `Recall@K (R@1, R@5, R@10)` **(primary)** — range: percent
  - Measures retrieval accuracy by checking if the ground-truth match appears within the top-K ranked candidates based on cosine similarity of L2-normalized embeddings.
- `BLEU-4, ROUGE-L, METEOR` — range: [0, 1]
  - Standard n-gram and sequence overlap metrics measuring structural similarity and lexical overlap between expert and layman captions.
- `LENS, RaTEScore, GREEN` — range: [0, 1]
  - LENS is a learnable metric for text simplification. RaTEScore and GREEN are model-based metrics designed to detect hallucinations and ensure clinical correctness in radiology reports.
- `FKGL, CLI, DCRS, SMOG, FRE` — range: other
  - Standard readability formulas quantifying text accessibility, grade level, and reading ease.

## Input / output format

**Input**: Image and text pair (for retrieval) or image with prompt (for captioning).

**Output**: Ranked list of candidate texts/images (retrieval) or generated natural language caption (captioning).

## Scoring recipe

```python
def compute_recall_at_k(sim_matrix, k):
    # sim_matrix: [N, N] cosine similarities between images and texts
    top_k_indices = np.argsort(sim_matrix, axis=1)[:, -k:]
    correct = np.sum(np.diag(top_k_indices) == np.arange(len(sim_matrix)))
    return correct / len(sim_matrix) * 100
```

## Common pitfalls

- Confusing expert vs. layman caption performance, as the benchmark reports both side-by-side.
- Applying fine-tuning or prompt engineering, which violates the zero-shot evaluation protocol.
- Ignoring domain adaptation effects, as general-domain VLMs significantly underperform medical-specific models on this benchmark.

## Evidence (verbatim from paper)

> To assess whether the simplified text preserves essential semantic information for automated analysis, we evaluate zero-shot text-to-image retrieval performance. We report Recall@K (R@1, R@5, R@10) to measure retrieval accuracy using the generated captions.

## Citation

```bibtex
@misc{jang2026medlaybenchv,
  title={MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2604.05738}
}
```

- arXiv: 2604.05738