Fairness Comparison Eval

fairness-comparison-eval

A comparative study of fairness-enhancing interventions in machine learning — Friedler et al. (2018) (arXiv:1802.04422, 2018)

What this evaluates

Evaluates the comparative performance of fairness-enhancing machine learning interventions across multiple datasets. It probes how different algorithmic strategies trade off predictive accuracy against a comprehensive set of fairness metrics under standardized preprocessing and fixed train-test splits.

Datasets

• Standardized benchmark datasets (unspecified in excerpt) — total ?; splits: train (-1), test (-1); repo https://github.com/algofairness/fairness-comparison

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions on the test set.
• fairness measures — range: [0, 1]
  • Aggregate evaluation across multiple fairness metrics (e.g., demographic parity, equal opportunity) computed on test predictions.

Input / output format

Input: Preprocessed dataset instances with features, protected attributes, and ground-truth labels, partitioned into fixed training and test splits.

Output: Predicted labels for test instances, followed by computed accuracy and fairness metric scores for each evaluated algorithm.

Scoring recipe

def compute_metrics(predictions, labels, protected_attr):
    accuracy = (predictions == labels).mean()
    # Compute fairness metrics using standard definitions on the test split
    fairness_scores = compute_fairness_metrics(predictions, labels, protected_attr)
    return {'accuracy': accuracy, 'fairness': fairness_scores}

Common pitfalls

• Combining dataset-specific preprocessing directly into algorithm code, which prevents fair cross-algorithm comparison.
• Analyzing algorithms under only one or two fairness measures instead of a comprehensive suite.
• Failing to account for sensitivity to training-test split variability and preprocessing dependencies.

Evidence (verbatim from paper)

ensure that each algorithm is compared using the same dataset (including the same preprocessing), the same set of training / test splits, and all desired fairness and accuracy measures.

Citation

@misc{friedler2018fairnesscomparison,
  title={A comparative study of fairness-enhancing interventions in machine learning},
  author={Friedler et al. (2018)},
  year={2018},
  note={arXiv:1802.04422}
}

• arXiv: 1802.04422