Fairness Algorithms Eval

fairness-algorithms-eval

Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms — Gareth P. Jones et al. (2020) (arXiv:2010.03986, 2020)

What this evaluates

This evaluation probes the trade-off between predictive performance and group fairness across various machine learning pipelines. It systematically compares fairness-unaware baselines against preprocessing and in-training fairness interventions, measuring how well algorithms maintain accuracy while satisfying demographic parity and equalized odds constraints.

Datasets

• Titanic — total 1300; splits: train (-1), test (-1)
• German — total 1000; splits: train (-1), test (-1)
• Adult — total 48800; splits: train (-1), test (-1)
• S-D — total 10000; splits: train (-1), test (-1)
• S-P — total 10000; splits: train (-1), test (-1)
• I-D — total 10000; splits: train (-1), test (-1)

Metrics

• Fair Efficiency (Theta_AUC_DI / Theta_AUC_EO) (primary) — range: [0, 1]
  • A policy-agnostic metric that integrates predictive performance (AUC) and fairness (DI or EO) over all classification thresholds and fairness regularization parameters (lambda). It is computed as the harmonic mean of the performance integral K_AUC and the fairness integral K_DI or K_EO.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring ranking quality across all thresholds.
• DI — range: [0, 1]
  • Demographic Inequality (or Disparate Impact ratio), measuring the ratio of positive prediction rates between protected and unprotected groups.
• EO — range: [0, 1]
  • Equalized Odds, measuring the difference in true positive and false positive rates between protected and unprotected groups.

Input / output format

Input: Tabular dataset with numerical/categorical features (x), a binary protected attribute (z), and a binary target label (y).

Output: Continuous probability score or decision boundary distance, thresholded at τ to produce a binary prediction.

Scoring recipe

def fair_efficiency(predictions, gold, protected_attr, lambdas, taus):
    perf_int, di_int, eo_int = 0.0, 0.0, 0.0
    for lam in lambdas:
        for tau in taus:
            preds = (predictions >= tau).astype(int)
            perf_int += compute_auc(gold, predictions)
            di_int += compute_di(gold, preds, protected_attr)
            eo_int += compute_eo(gold, preds, protected_attr)
    n = len(lambdas) * len(taus)
    k_auc, k_di, k_eo = perf_int/n, di_int/n, eo_int/n
    theta_di = 2 * (k_auc * k_di) / (k_auc + k_di)
    theta_eo = 2 * (k_auc * k_eo) / (k_auc + k_eo)
    return theta_di, theta_eo

Common pitfalls

• Standard metrics like accuracy and precision are highly sensitive to the classification threshold tau, making cross-algorithm comparisons unfair without threshold normalization or integration.
• The lambda parameter has different semantic meanings across algorithms (e.g., regularization strength vs. reweighing state vs. tolerance), requiring careful interpretation of the fairness-performance trade-off curve.
• Synthetic datasets encode specific correlation structures between features and protected attributes, which can create 'intrinsic unfairness' that confounds algorithmic fairness claims if not accounted for.

Evidence (verbatim from paper)

The aim of this evaluation is to compare a selection of practical fairness approaches across a diverse range of benchmark datasets. We will use standard metrics such as EO and DI, precision, accuracy, the area under the Receiver Operating Characteristic curve (AUC), as well as our policy-agnostic fair efficiency.

Citation

@misc{jones2020fairnesscomparison,
  title={Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms},
  author={Gareth P. Jones et al. (2020)},
  year={2020},
  note={arXiv:2010.03986}
}

• arXiv: 2010.03986