Dermabench Eval

dermabench-eval

DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning — Yilmaz et al. (2026) (arXiv:2601.14084, 2026)

What this evaluates

Evaluates vision-language models on dermatological visual question answering and clinical reasoning. It probes the model's ability to understand skin lesions across diverse Fitzpatrick skin types, answer structured diagnostic questions, and reason about morphology and distribution.

Datasets

• DermaBench — total 14474; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard VQA exact-match accuracy: percentage of questions where the model's predicted answer exactly matches the ground truth answer (case-insensitive).

Input / output format

Input: A clinical dermatology image paired with a structured question from a hierarchical schema (Q0–Q21) covering diagnosis, morphology, distribution, and clinical reasoning.

Output: A text answer corresponding to the question, formatted as a single choice, multiple choice, or open-ended response depending on the question type.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(gold_answers)

Common pitfalls

• The dataset spans Fitzpatrick skin types I–VI, so performance may vary significantly by skin tone; models must be evaluated with fairness-aware stratification.
• Questions follow a hierarchical branching logic; models must handle conditional sub-questions correctly rather than answering all 22 questions uniformly.
• The benchmark emphasizes clinical reasoning and morphology grounding, so superficial visual matching without diagnostic justification may yield high accuracy but fail clinical validity checks.

Evidence (verbatim from paper)

All data follow a VQA-style JSON schema that includes image path, modality tag, Fitzpatrick phototype, diagnostic and morphological labels, question category, answer text, annotator identifier, and additional metadata fields supporting fairness and reasoning analyses. The dataset is released for research use under a permissive license and is accompanied by documentation detailing the annotation protocol, consensus process, and recommended evaluation workflows.

Citation

@misc{yilmaz2026dermabench,
  title={DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning},
  author={Yilmaz et al. (2026)},
  year={2026},
  note={arXiv:2601.14084}
}

• arXiv: 2601.14084