Dice Coefficient

dice-coefficient

DWARF: Disease-weighted network for attention map refinement — Luo et al. (2024) (arXiv:2406.17032, 2024)

What this evaluates

Evaluates a disease-weighted attention refinement framework for medical image analysis. It probes the model's ability to align cross-attention maps with radiologist annotations (grounding) while maintaining diagnostic accuracy on chest X-ray classification tasks.

Datasets

• (no dataset; pure metric skill)

Metrics

• Dice Coefficient (primary) — range: [0, 1]
  • Assesses spatial alignment between predicted regions ($\hat{R}$) and expert annotations ($R$). Formula: $\text{Dice}=\frac{2|\hat{R}\cap R|}{|\hat{R}|+|R|}$. Emphasizes both precision and recall, and is more sensitive to small mismatches than IoU.
• Hit-rate — range: [0, 1]
  • Evaluates the extent to which algorithmically identified key areas overlap with expert annotations. A higher value indicates better alignment between model focus and expert annotations.
• AUC-ROC — range: [0, 1]
  • Measures the model's ability to distinguish between classes across different probability thresholds. Values close to 1 indicate excellent discriminative power, while 0.5 indicates random chance.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient evaluates binary classification quality. Provides a balanced score even with imbalanced classes, ranging from -1 (total disagreement) to +1 (perfect prediction), with 0 indicating random guessing.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall. Ranges from 0 to 1, with 1 indicating perfect precision and recall. Performs well under uneven class distribution.

Input / output format

Input: Chest X-ray images paired with disease labels and expert radiologist region annotations.

Output: Predicted disease labels, refined cross-attention maps (binary or continuous regions $\hat{R}$), and computed metric scores.

Scoring recipe

def compute_metrics(pred_maps, pred_labels, gold_maps, gold_labels):
    dice = 2 * np.sum(pred_maps & gold_maps) / (np.sum(pred_maps) + np.sum(gold_maps))
    hit_rate = np.mean((pred_maps & gold_maps).any(axis=1))
    auc = roc_auc_score(gold_labels, pred_labels)
    mcc = matthews_corrcoef(gold_labels, pred_labels)
    f1 = f1_score(gold_labels, pred_labels, average='weighted')
    return {'dice': dice, 'hit_rate': hit_rate, 'auc': auc, 'mcc': mcc, 'f1': f1}

Common pitfalls

• Dice and IoU are frequently confused; Dice yields higher values for partial overlaps and is more sensitive to small mismatches than IoU.
• MCC is preferred over accuracy for imbalanced clinical datasets, but many baselines still report accuracy or macro-F1 without addressing class imbalance.
• Hit-rate requires a threshold to binarize continuous attention maps before computing overlap with annotations; the threshold choice significantly impacts the score.

Evidence (verbatim from paper)

The Dice Coefficient assesses the alignment between the predicted regions ($\hat{R}$) and expert annotations ($R$). It is defined as: $\text{Dice}=\frac{2|\hat{R}\cap R|}{|\hat{R}|+|R|}$ This metric emphasizes both precision and recall, capturing spatial overlap effectively. Compared to the Intersection over Union (IoU), defined as: $\text{IoU}=\frac{|\hat{R}\cap R|}{|\hat{R}\cup R|}$ the Dice Coefficient generally yields higher values for partial overlaps and is more sensitive to small mismatches.

Citation

@misc{luo2024dwarf,
  title={DWARF: Disease-weighted network for attention map refinement},
  author={Luo et al. (2024)},
  year={2024},
  note={arXiv:2406.17032}
}

• arXiv: 2406.17032