Fairdomain Eval

fairdomain-eval

FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification — Tian et al. (2024) (arXiv:2407.08813, 2024)

What this evaluates

Evaluates cross-domain medical image segmentation and classification performance while measuring demographic fairness across gender, race, and ethnicity. It probes whether models maintain equitable accuracy across demographic groups when domain shifts occur between imaging modalities (En face vs. SLO fundus images).

Datasets

• FairDomain-Segmentation — total ?; splits: source (-1), target (-1)

Metrics

• ESP (Equity-Scaled Performance) (primary) — range: [0, 1]
  • ESP = M / (1 + Δ), where M is a base metric (Dice, IoU, or AUC) and Δ = Σ_{A∈Demographics} |M(overall) - M(group=A)|. It balances accuracy and fairness by penalizing performance disparity across demographic groups.
• Dice — range: [0, 1]
  • Standard Dice coefficient measuring overlap between predicted and ground truth segmentation masks.
• IoU — range: [0, 1]
  • Intersection over Union measuring overlap between predicted and ground truth segmentation masks.
• AUC — range: [0, 1]
  • Area under the receiver operating characteristic curve for binary classification (normal vs. glaucoma).

Input / output format

Input: Fundus images (En face or SLO modality) paired with demographic attributes (gender, race, or ethnicity). Ground truth includes segmentation masks (optic cup/rim) or binary classification labels.

Output: Predicted segmentation masks (cup and rim boundaries) or binary classification labels (normal/glaucoma).

Scoring recipe

def compute_esp(preds, gold, demographics, base_metric):
    overall = base_metric(preds, gold)
    delta = 0.0
    for group in demographics.unique():
        mask = demographics == group
        group_score = base_metric(preds[mask], gold[mask])
        delta += abs(overall - group_score)
    return overall / (1 + delta)

Common pitfalls

• Reformulating cup/disc segmentation into cup/rim is required to avoid misrepresented performance due to the large overlapped area between the cup and disc borders.
• Evaluating only on overall metrics without stratifying by demographic attributes (gender, race, ethnicity) will miss the fairness tradeoff that ESP is designed to capture.
• Domain shift is explicitly controlled by using paired images from the same patient cohort across En face and SLO modalities, so results should not be conflated with demographic distribution shifts.

Evidence (verbatim from paper)

Inspired by [39, 69], to address the potential tradeoff between model performance and fairness in medical imaging, we use novel equity-scaled performance (ESP) metrics to assess both performance and fairness for both segmentation and classification tasks. ... The ESP metric can then be formulated as follows: ESP = M / (1 + Δ), which ensures a balanced evaluation of model accuracy and fairness.

Citation

@misc{tian2024fairdomain,
  title={FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2407.08813}
}

• arXiv: 2407.08813