Harvard Eye Fairness Eval

harvard-eye-fairness-eval

FairVision: Equitable Deep Learning for Eye Disease Screening via Fair Identity Scaling — Luo et al. (2023) (arXiv:2310.02492, 2023)

What this evaluates

Evaluates deep learning models for eye disease screening (AMD, DR, glaucoma) on 2D fundus and 3D OCT images. It measures both overall diagnostic performance and demographic fairness across race, gender, and ethnicity to assess equitable model behavior.

Datasets

• Harvard-EF30k — total 30000; splits: train (6000), val (1000), test (3000)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve, computed overall and per demographic group.
• DPD — range: [0, 1]
  • Disparate Prediction Difference: absolute difference in positive prediction rates between protected and unprotected demographic groups.
• DEOdds — range: other
  • Demographic Equality of Odds: sum of absolute differences in True Positive Rate and False Positive Rate between demographic groups.
• Mean PSD — range: percent
  • Performance-Scaled Disparity: mean of absolute differences between group-wise AUCs and overall AUC, scaled by overall AUC.
• Max PSD — range: percent
  • Performance-Scaled Disparity: maximum of absolute differences between group-wise AUCs and overall AUC, scaled by overall AUC.

Input / output format

Input: 2D SLO fundus images or 3D OCT B-scans paired with demographic labels (race, gender, ethnicity, age, language, marital status).

Output: Disease prediction probabilities per image, used to compute group-wise AUCs and fairness metrics.

Scoring recipe

def compute_metrics(y_true, y_pred_prob, groups):
    auc = roc_auc_score(y_true, y_pred_prob)
    p_pos_0 = mean(y_pred_prob[groups == 0])
    p_pos_1 = mean(y_pred_prob[groups == 1])
    dpd = abs(p_pos_0 - p_pos_1)
    tpr_0, fpr_0 = tpr_fpr(y_true[groups==0], y_pred_prob[groups==0])
    tpr_1, fpr_1 = tpr_fpr(y_true[groups==1], y_pred_prob[groups==1])
    deo = abs(tpr_0 - tpr_1) + abs(fpr_0 - fpr_1)
    group_aucs = [roc_auc_score(y_true[g], y_pred_prob[g]) for g in unique(groups)]
    mean_psd = mean(abs(g_auc - auc) / auc) * 100
    max_psd = max(abs(g_auc - auc) / auc) * 100
    return auc, dpd, deo, mean_psd, max_psd

Common pitfalls

• DPD and DEOdds may not consistently align with overall AUC improvements, as noted in the results.
• PSD scores can worsen even when overall and group-wise AUCs improve, highlighting the need for comprehensive fairness metrics.
• Random split is used for each model run, so results should be averaged over multiple seeds (paper uses 3).

Evidence (verbatim from paper)

To facilitate the model fairness assessment, we use both overall and group-wise AUCs to compare model performance. In addition, we will use traditional fairness metrics of DPD and DEOdds to assess model fairness. Furthermore, we propose to use our performance-scaled disparity (Mean and Max PSDs) scores to evaluate model fairness in the context of overall model performance.

Citation

@misc{luo2023fairvision,
  title={FairVision: Equitable Deep Learning for Eye Disease Screening via Fair Identity Scaling},
  author={Luo et al. (2023)},
  year={2023},
  note={arXiv:2310.02492}
}

• arXiv: 2310.02492