Counterfactual Fairness Eval

counterfactual-fairness-eval

Achieving Counterfactual Fairness with Imperfect Structural Causal Model — Tri Dung Duong et al. (2023) (arXiv:2303.14665, 2023)

What this evaluates

Evaluates the trade-off between predictive accuracy and counterfactual fairness on real-world datasets. It measures how well a model's predictions remain invariant to sensitive attributes (race, gender) while maintaining performance on regression or classification tasks.

Datasets

• LSAC — total ?; splits: train (-1), test (-1)
• Compas — total 6167; splits: train (-1), test (-1)
• Adult — total ?; splits: train (-1), test (-1)

Metrics

• Balanced Accuracy (primary) — range: [0, 1]
  • Defined as (TPR + TNR) / 2, where TPR is the true positive rate and TNR is the true negative rate. Higher values indicate better performance.
• RMSE — range: other
  • Root mean squared error between predicted and true values. Lower values indicate better regression performance.
• MAE — range: other
  • Mean absolute error between predicted and true values. Lower values indicate better regression performance.
• R2score — range: other
  • Coefficient of determination measuring the proportion of variance in the dependent variable predictable from the independent variables. Higher values are better.
• Wasserstein — range: other
  • Wasserstein distance between the distribution of model predictions across different sensitive attribute groups. Lower values indicate better fairness.
• MMD (Gaussian) — range: other
  • Maximum Mean Discrepancy with a Gaussian kernel measuring distributional divergence of predictions across sensitive groups. Lower values indicate better fairness.
• CV — range: other
  • Coefficient of Variation, a generalized entropy index with alpha=2 computed on prediction errors. Lower values indicate better fairness.
• TI — range: other
  • Theil Index, a generalized entropy index with alpha=1 computed on prediction errors. Lower values indicate better fairness.

Input / output format

Input: Feature vectors containing both sensitive attributes (e.g., race, gender) and non-sensitive attributes (e.g., LSAT scores, GPA, prior convictions, age, income indicators).

Output: Continuous numerical prediction for regression tasks, or binary class label for classification tasks.

Scoring recipe

def compute_metrics(y_true, y_pred, sensitive_attr):
    rmse = np.sqrt(np.mean((y_true - y_pred)**2))
    mae = np.mean(np.abs(y_true - y_pred))
    r2 = 1 - np.sum((y_true - y_pred)**2) / np.sum((y_true - np.mean(y_true))**2)
    tpr = np.mean(y_pred[sensitive_attr==1] == y_true[sensitive_attr==1])
    tnr = np.mean(y_pred[sensitive_attr==0] == y_true[sensitive_attr==0])
    bal_acc = (tpr + tnr) / 2
    b = y_pred - y_true + 1
    mu = np.mean(b)
    cv = (1 / (len(b) * 2 * 1)) * np.sum(((b / mu)**2 - 1))
    ti = (1 / len(b)) * np.sum((b / mu) * np.log(b / mu))
    return {'RMSE': rmse, 'MAE': mae, 'R2score': r2, 'Balanced Acc': bal_acc, 'CV': cv, 'TI': ti}

Common pitfalls

• Counterfactual fairness cannot be directly evaluated on observational data without ground truth counterfactuals; the protocol approximates it using individual fairness criteria (distributional similarity across sensitive groups).
• Datasets are highly imbalanced, making standard accuracy misleading; Balanced Accuracy must be used instead.
• Lower values are better for all metrics except Precision, Recall, F1 score, and R2score, which requires careful interpretation when comparing fairness-accuracy trade-offs.

Evidence (verbatim from paper)

We emphasize that since the Adult and Compas datasets are highly imbalanced, we use the Balanced Accuracy instead of the traditional accuracy, which is defined as Balanced Acc=(TPR+TNR)/2 where TPR and TNR are true positive rate, and true negative rate, respectively. For the fairness performance, we use Wasserstein distance and maximum mean discrepancy (MMD) with Gaussian kernel in the regression task. On the other hand, we utilize generalized entropy index to evaluate the performance in the classification task.

Citation

@misc{duong2023counterfactualfairness,
  title={Achieving Counterfactual Fairness with Imperfect Structural Causal Model},
  author={Tri Dung Duong et al. (2023)},
  year={2023},
  note={arXiv:2303.14665}
}

• arXiv: 2303.14665