Dermx Benchmark Eval

dermx-benchmark-eval

Dermatological Diagnosis Explainability Benchmark for Convolutional Neural Networks — Jalaboi et al. (2023) (arXiv:2302.12084, 2023)

What this evaluates

Evaluates the diagnostic accuracy and explainability of various Convolutional Neural Network architectures on dermatological image classification. It specifically probes how well different architectures localize clinically relevant skin characteristics using Grad-CAM heatmaps compared to human dermatologists.

Datasets

• DermXDB — total ?; splits: test (-1); repo https://github.com/ralucaj/dermx-benchmark

Metrics

• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for the predicted diagnosis class across multiple skin conditions.
• image-level Grad-CAM F1 score (primary) — range: [0, 1]
  • F1 score computed by thresholding the Grad-CAM heatmap and measuring overlap with ground-truth lesion masks.
• characteristic-level sensitivity — range: [0, 1]
  • Proportion of clinically relevant skin features (e.g., scale, plaque) correctly highlighted by the model's heatmap.

Input / output format

Input: RGB photographic images of skin lesions.

Output: Predicted diagnosis class and a Grad-CAM heatmap highlighting diagnostically relevant regions.

Scoring recipe

def compute_f1(pred, gold):
    tp = sum(p == g == 1 for p, g in zip(pred, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(pred, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(pred, gold))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_gradcam_f1(model, image, gt_mask):
    heatmap = generate_gradcam(model, image)
    pred_mask = (heatmap > threshold).astype(int)
    return compute_f1(pred_mask.flatten(), gt_mask.flatten())

Common pitfalls

• Models may overfit to non-clinical artifacts like watermarks instead of learning lesion features.
• Expert-level diagnostic performance is only achieved for specific conditions, not universally across all classes.
• Grad-CAM heatmaps often focus on small, localized areas in older architectures, potentially missing broader diagnostic context.

Evidence (verbatim from paper)

Table 4 shows the image-level explainability results for each of the benchmarked ConvNets, while Figure 4 shows the relationship between ConvNet diagnosis performance, image-level explainability, and number of parameters. Xception scores the highest on the image-level Grad-CAM F1 score, while InceptionResNetV2, ResNet50, and VGG16 have the lowest performance.

Citation

@misc{jalaboi2023dermxbenchmark,
  title={Dermatological Diagnosis Explainability Benchmark for Convolutional Neural Networks},
  author={Jalaboi et al. (2023)},
  year={2023},
  note={arXiv:2302.12084}
}

• arXiv: 2302.12084