Chestxray14 Bias Eval

chestxray14-bias-eval

Hybrid Vision Transformer_GAN Attribute Neutralizer for Mitigating Bias in Chest X_Ray Diagnosis — Jobeal Solomon, Ali Mohammed Mansoor Alsahag, Seyed Sahand Mohammadi Ziabari (2026) (arXiv:2601.15490, 2026)

What this evaluates

Evaluates the effectiveness of attribute-neutralization models in removing demographic bias (sex and age) from chest X-ray images while preserving diagnostic utility for 15 disease findings. It probes the trade-off between demographic leakage suppression and clinical performance across varying edit intensities.

Datasets

• ChestX-ray14 — total ?; splits: test (1000)

Metrics

• AI-Judge AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve computed between the ground-truth demographic attribute (sex or age) and the AI-Judge's predicted probability on the neutralized image. Measures attribute leakage.
• Macro ROC-AUC — range: [0, 1]
  • Mean of the ROC-AUC scores computed independently for each of the 15 disease findings. Measures overall diagnostic performance on neutralized images.
• Worst-case subgroup AUC — range: [0, 1]
  • Minimum ROC-AUC across demographic subgroups (e.g., male/female, <60/≥60) for each disease finding, then median across findings. Measures fairness and subgroup performance.
• SSIM — range: [0, 1]
  • Structural Similarity Index measuring pixel-level similarity between original and neutralized images. Used to quantify the magnitude of image edits.

Input / output format

Input: Chest X-ray radiographs annotated with patient sex, age, and 15 disease findings. Images are processed through attribute neutralizers at varying edit intensities ($\alpha$ from 0.0 to 1.0).

Output: Neutralized chest X-ray images, AI-Judge predictions (probability of sex/age), and disease diagnosis predictions (15 binary findings).

Scoring recipe

def compute_metrics(original_img, neutralized_img, ai_judge, diagnosis_model, y_true_attr, y_true_disease):
    y_pred_attr = ai_judge.predict(neutralized_img)
    y_pred_disease = diagnosis_model.predict(neutralized_img)
    
    # Leakage
    leakage_auc = roc_auc_score(y_true_attr, y_pred_attr)
    
    # Diagnosis
    macro_auc = np.mean([roc_auc_score(y_true_disease[:, i], y_pred_disease[:, i]) for i in range(15)])
    
    # Fairness
    subgroup_aucs = [roc_auc_score(y_true_disease, y_pred_disease) for group in demographic_groups]
    worst_auc = min(subgroup_aucs)
    
    return leakage_auc, macro_auc, worst_auc

Common pitfalls

• Confusing edit-intensity parameter $\alpha$ with model capability; higher $\alpha$ does not linearly improve neutrality and can collapse diagnostic performance.
• Overlooking class imbalance when interpreting per-finding ROC-AUC distributions, which can mask significant performance drops on rare findings.
• Comparing CNN and ViT neutralizers without noting the CNN baseline showed almost no leakage reduction across all $\alpha$ levels, making direct $\alpha$-sweep comparisons misleading.

Evidence (verbatim from paper)

Leakage was assessed at three $\alpha$-levels (0.0, 0.5, 1.0) across three generator variants. Table 2 shows that at the turning-point edit ($\alpha=0.5$), the ViT neutralizer lowers the AI-Judge’s AUC by $\geq 0.15$ relative to both Hu et al.’s U-Net and our own CNN retrain, while the CNN variant hardly changes.

Citation

@misc{solomon2026hybrid,
  title={Hybrid Vision Transformer_GAN Attribute Neutralizer for Mitigating Bias in Chest X_Ray Diagnosis},
  author={Jobeal Solomon, Ali Mohammed Mansoor Alsahag, Seyed Sahand Mohammadi Ziabari (2026)},
  year={2026},
  note={arXiv:2601.15490}
}

• arXiv: 2601.15490