Pokec N Fairness Eval

pokec-n-fairness-eval

Improving Fairness in Graph Neural Networks via Counterfactual Debiasing — Zengyi Wo et al. (2025) (arXiv:2508.14683, 2025)

What this evaluates

Evaluates the ability of Graph Neural Networks to perform node classification while mitigating bias related to a protected attribute (Region). It probes the trade-off between predictive accuracy and group fairness across different GNN architectures.

Datasets

• Pokec-n — total 66569; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for the downstream node classification task.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified nodes out of the total number of nodes.
• DP (Demographic Parity) — range: [0, 1]
  • Group fairness metric measuring disparity in positive prediction rates across sensitive groups. Lower values indicate higher fairness.
• EO (Equalized Odds) — range: [0, 1]
  • Group fairness metric measuring disparity in true positive and false positive rates across sensitive groups. Lower values indicate higher fairness.

Input / output format

Input: Graph-structured data containing node features, adjacency matrix, node labels (Working field), and a sensitive attribute (Region) for each node.

Output: Predicted class label for each node in the graph.

Scoring recipe

def compute_metrics(y_true, y_pred, sensitive_attr):
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred, average='macro')
    # DP and EO are computed per sensitive group; lower values indicate better fairness
    dp = compute_demographic_parity(y_true, y_pred, sensitive_attr)
    eo = compute_equalized_odds(y_true, y_pred, sensitive_attr)
    return {'accuracy': acc, 'f1': f1, 'dp': dp, 'eo': eo}

Common pitfalls

• DP and EO fairness metrics are inversely scaled: lower values indicate better fairness, contrary to standard performance metrics.
• The EDITS baseline frequently fails with Out-Of-Memory (OOM) errors on certain GNN backbones, which may artificially inflate the relative performance of other methods if not explicitly handled.
• All reported results are averaged over five independent runs with different random seeds; single-run evaluations will not match the paper's reported means.

Evidence (verbatim from paper)

We evaluate the performance of downstream classification tasks using F1 score and accuracy metrics. When assessing group fairness, we consider DP and EO based on previous research. It is essential to understand that lower values of DP and EO indicate a higher level of fairness in a model.

Citation

@misc{wo2025improvingfairness,
  title={Improving Fairness in Graph Neural Networks via Counterfactual Debiasing},
  author={Zengyi Wo et al. (2025)},
  year={2025},
  note={arXiv:2508.14683}
}

• arXiv: 2508.14683