Fairness Aware Graph Learning Eval

fairness-aware-graph-learning-eval

A Benchmark for Fairness-Aware Graph Learning — Dong et al. (2024) (arXiv:2407.12112, 2024)

What this evaluates

This benchmark evaluates the trade-offs between predictive utility and fairness across various graph learning algorithms. It probes how well different methods balance accuracy with demographic parity and equal opportunity constraints on real-world graph-structured data.

Datasets

• 7 real-world datasets — total ?; splits: test (-1)

Metrics

• Δ_SP (primary) — range: [-1, 1]
  • Statistical Parity Difference: the absolute difference in positive prediction rates between protected and unprotected groups. Lower values indicate better group fairness.
• Δ_EO — range: [-1, 1]
  • Equal Opportunity Difference: the absolute difference in true positive rates between protected and unprotected groups. Lower values indicate better group fairness.
• AUC-ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes across all classification thresholds.
• running time — range: seconds
  • Wall-clock time required to train or evaluate the model on a given dataset.

Input / output format

Input: Graph-structured data containing node features, adjacency matrices, sensitive attributes, and ground-truth labels.

Output: Predicted node labels (or probabilities) and computed fairness/utility scores per dataset.

Scoring recipe

def compute_metrics(y_true, y_pred, sensitive_attr):
    # Group Fairness (Δ_SP)
    pred_rates = {g: mean(y_pred[sensitive_attr == g]) for g in unique(sensitive_attr)}
    delta_sp = abs(pred_rates[protected] - pred_rates[unprotected])
    # Equal Opportunity (Δ_EO)
    tpr = {g: mean((y_pred == 1) & (y_true == 1) & (sensitive_attr == g)) for g in unique(sensitive_attr)}
    delta_eo = abs(tpr[protected] - tpr[unprotected])
    # Utility
    auc_roc = roc_auc_score(y_true, y_pred)
    return delta_sp, delta_eo, auc_roc

Common pitfalls

• Confusing group fairness metrics (Δ_SP, Δ_EO) with individual fairness metrics, as the benchmark evaluates both separately.
• Assuming higher utility always correlates with better fairness; the benchmark explicitly highlights method-specific trade-offs where utility sacrifices are necessary.
• Overlooking computational efficiency, as fairness-aware methods often incur significant runtime overhead compared to baseline GNNs.

Evidence (verbatim from paper)

According to the comprehensive empirical results, we have the consistent observation that different fairness-aware graph learning methods show different advantages in balancing utility and fairness. Specifically, we observe that GNN-based algorithms are among the highest-ranking methods (e.g., most top-ranked results come from GNN-based methods), which verifies the advantage of GNNs in achieving both utility and fairness objectives due to their exceptional fitting ability. In addition, fairness-aware shallow embedding methods achieve the best performance with respect to fairness, especially on the traditional fairness metrics Δ_SP and Δ_EO .

Citation

@misc{dong2024fairnessawaregraphlearning,
  title={A Benchmark for Fairness-Aware Graph Learning},
  author={Dong et al. (2024)},
  year={2024},
  note={arXiv:2407.12112}
}

• arXiv: 2407.12112