Fairness Aware Gnn Eval

fairness-aware-gnn-eval

Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphs — Sasaki (2025) (arXiv:2510.18473, 2025)

What this evaluates

Evaluates the trade-off between prediction accuracy and statistical fairness for graph neural networks on node classification tasks. It probes how different in-processing and preprocessing methods, backbone architectures, and early stopping conditions affect both standard performance metrics and fairness constraints across synthetic, social, and knowledge graph datasets.

Datasets

• Credit — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• Bail — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• Pokec-n — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• Pokec-z — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• Pokec-n-Large — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• Pokec-z-Large — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• DBpedia — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• YAGO — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
• Wikidata — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase

Metrics

• ACC (primary) — range: [0, 1]
  • Standard classification accuracy: fraction of correctly predicted labels out of total nodes.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall.
• ΔSP — range: [0, 1]
  • Statistical Parity difference: |P(ŷ=1|s=0) - P(ŷ=1|s=1)|. Smaller is better.
• ΔEop — range: [0, 1]
  • Equal Opportunity difference: |P(ŷ=1|y=1,s=0) - P(ŷ=1|y=1,s=1)|. Smaller is better.

Input / output format

Input: Graph adjacency structure, node feature matrix, node-level sensitive attribute vector (s ∈ {0,1}), and node-level ground truth label vector (y ∈ {0,1}) for a node classification task.

Output: Binary predicted label vector (ŷ ∈ {0,1}) for each node in the dataset.

Scoring recipe

def compute_metrics(y_true, y_pred, sensitive):
    acc = (y_true == y_pred).mean()
    f1 = f1_score(y_true, y_pred, average='binary')
    auc = roc_auc_score(y_true, y_pred)
    sp = abs((y_pred[sensitive == 0].mean()) - (y_pred[sensitive == 1].mean()))
    eop = abs((y_pred[(y_true == 1) & (sensitive == 0)].mean()) - 
              (y_pred[(y_true == 1) & (sensitive == 1)].mean()))
    return {'ACC': acc, 'F1': f1, 'AUC': auc, 'ΔSP': sp, 'ΔEop': eop}

Common pitfalls

• Early stopping conditions drastically alter the accuracy-fairness trade-off, yet many baseline implementations use default or inconsistent stopping criteria.
• Fairness-aware methods can paradoxically degrade fairness metrics while improving accuracy, contradicting their design goals.
• Certain methods (e.g., FairGB, BIND) are incompatible with specific backbones or fail to converge within practical time/memory limits on larger graphs.

Evidence (verbatim from paper)

We use three prediction accuracy metrics: Accuracy (ACC), AUC, and F1. According to prior studies*([beutel2017data,] ; [TheVariationalFairAutoencoder,] ), we use $\Delta SP$([dwork2012fairness,] )* and $\Delta Eop$([hardt2016equality,] ) to quantitatively evaluate statistical parity and equal opportunity. $\Delta SP=|P(\hat{y}=1\mid s=0)-P(\hat{y}=1\mid s=1)|$ and $\Delta Eop=|P(\hat{y}=1\mid y=1,s=0)-P(\hat{y}=1\mid y=1,s=1)|$, where $y\in{0,1}$ represents the label, variable $s\in{0,1}$ represents the sensitive attribute, and $\hat{y}\in{0,1}$ indicates prediction output. Both $\Delta SP$ and $\Delta Eop$ are “smaller-is-better” metrics ( $\downarrow$), and ACC, AUC, and F1 are “larger-is-better” metrics ( $\uparrow$).

Citation

@misc{sasaki2025benchmarkingfairness,
  title={Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphs},
  author={Sasaki (2025)},
  year={2025},
  note={arXiv:2510.18473}
}

• arXiv: 2510.18473