Ffb Eval

ffb-eval

FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods — Han et al. (2023) (arXiv:2306.09468, 2023)

What this evaluates

Evaluates in-processing group fairness methods by measuring the trade-off between model utility (acc) and various fairness metrics across multiple datasets and hyperparameter settings.

Datasets

• Adult — total ?; splits: train (-1), test (-1); repo https://github.com/ahxt/fair_fairness_benchmark

Metrics

• acc (primary) — range: [0, 1]
  • Proportion of correctly predicted labels out of total instances. Computed as mean(predictions == labels).
• fairness metric — range: [0, 1]
  • Measures statistical parity or equalized odds between groups defined by a sensitive attribute. Specific formula varies by method.

Input / output format

Input: Tabular feature vectors, ground-truth labels, and a designated sensitive attribute column.

Output: Predicted labels or probabilities, along with computed utility and fairness scores.

Scoring recipe

def compute_metrics(predictions, labels, sensitive_attr):
    acc = (predictions == labels).mean()
    # Fairness metric depends on the specific group fairness definition used
    fairness = compute_fairness_score(predictions, sensitive_attr)
    return acc, fairness

Common pitfalls

• Utility-fairness trade-offs are highly sensitive to the choice of fairness control hyperparameters.
• Fairness performance on one dataset (e.g., Adult) does not guarantee similar results on other datasets due to varying inherent biases.

Evidence (verbatim from paper)

We plot the utility-fairness trade-offs for the Adult dataset with gender as the sensitive attribute and present the results in Figures 7 and 8. Figure 7: The Utility-Fairness Trade-offs with acc as utility metric.

Citation

@misc{han2023ffb,
  title={FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods},
  author={Han et al. (2023)},
  year={2023},
  note={arXiv:2306.09468}
}

• arXiv: 2306.09468