Graph Counterfactual Fairness Eval

graph-counterfactual-fairness-eval

Learning Fair Node Representations with Graph Counterfactual Fairness — Ma et al. (2022) (arXiv:2201.03662, 2022)

What this evaluates

Evaluates graph neural networks for node classification fairness by measuring prediction accuracy alongside statistical fairness metrics (demographic parity, equal opportunity) and a novel graph counterfactual fairness metric that quantifies how much node predictions change when sensitive attributes of the node and its neighbors are perturbed.

Datasets

• Synthetic — total 2000; splits: train (-1), val (-1), test (-1)
• Bail — total 18876; splits: train (-1), val (-1), test (-1)
• Credit — total 30000; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy — range: [0, 1]
  • Fraction of correctly predicted labels out of total nodes.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for the positive class.
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring ranking quality of prediction probabilities.
• △_DP — range: [0, 1]
  • Demographic parity difference: |P(Ŷ_i|S_i=0) - P(Ŷ_i|S_i=1)|.
• △_EO — range: [0, 1]
  • Equal opportunity difference: |P(Ŷ_i|Y_i=1, S_i=0) - P(Ŷ_i|Y_i=1, S_i=1)|.
• δ_CF (primary) — range: [0, 1]
  • Graph counterfactual fairness: |P((Ŷ_i){S←s'}|X=X, A=A) - P((Ŷ_i){S←s''}|X=X, A=A)|. Estimated by averaging the ratio of nodes whose predicted labels flip when 0%, 50%, and 100% of nodes' sensitive attributes are perturbed.
• R^2 — range: [0, 1]
  • R-square of a linear regression predicting Ŷ_i from the mean-aggregated sensitive attribute values of a node and its one-hop neighbors.

Input / output format

Input: Graph-structured data containing an adjacency matrix, node feature matrix, binary sensitive attribute vector, and node labels. Each evaluation instance corresponds to a full-graph node classification task.

Output: Predicted label Ŷ_i or prediction probability P(Ŷ_i) for each node in the graph.

Scoring recipe

accuracy = mean(preds == labels)
f1 = f1_score(labels, preds)
auroc = roc_auc_score(labels, probs)
dp = abs(mean(preds[sensitive == 0]) - mean(preds[sensitive == 1]))
eo = abs(mean(preds[(labels == 1) & (sensitive == 0)]) - mean(preds[(labels == 1) & (sensitive == 1)]))
flip_rates = []
for ratio in [0.0, 0.5, 1.0]:
    cf_s = perturb_sensitive(sensitive, ratio)
    cf_graph = generate_cf_graph(X, A, cf_s, causal_model)
    cf_preds = model.predict(cf_graph)
    flip_rates.append(mean(preds != cf_preds))
delta_cf = mean(flip_rates)
neighbor_s = mean_aggregate(sensitive, neighbors)
r2 = r2_score(neighbor_s, preds)

Common pitfalls

• The counterfactual graphs for real-world datasets are generated using an assumed Naïve Bayes + cosine similarity causal model, which may not reflect true causal dependencies in the data.
• δ_CF is not computed over all 2^n possible counterfactuals but estimated by averaging label flip rates across only three perturbation levels (0%, 50%, 100% of nodes).
• Sensitive attributes in real-world datasets (race, age) are treated as binary for metric computation, requiring discretization that is not explicitly detailed in the text.

Evidence (verbatim from paper)

To measure the fairness of the representations, we first use two metrics which are commonly used in statistical fairness: △_SP=|P(Ŷ_i|S_i=0)-P(Ŷ_i|S_i=1)|, and △_EO=|P(Ŷ_i|Y_i=1,S_i=0)-P(Ŷ_i|Y_i=1,S_i=1)|. To evaluate graph counterfactual fairness, we design a metric δ_CF: ... As there are too many different counterfactuals (e.g., there are 2^n cases for a graph with n nodes), it is difficult to evaluate the difference of predictions under all these counterfactuals. Therefore, we evaluate the graph counterfactual fairness of the proposed model in the following way: on each dataset, we control the rate of sensitive subgroup population and randomly perturb the sensitive attribute of all nodes. More specifically, we randomly select 0%,50%,100% nodes, and set their sensitive attribute values to be 1, while set the sensitive attribute of other nodes to be 0. With such perturbations, we generate counterfactual data for the whole graph with different ratios of sensitive subgroup, based on the causal model described in Section 4.1. Intuitively, these perturbations implicitly control the distribution of the sensitive attribute in each node’s neighborhood, and we take the averaged ratio of n

Citation

@misc{ma2022learningfair,
  title={Learning Fair Node Representations with Graph Counterfactual Fairness},
  author={Ma et al. (2022)},
  year={2022},
  note={arXiv:2201.03662}
}

• arXiv: 2201.03662