Facet Fairness Eval

facet-fairness-eval

FACET: Fairness in Computer Vision Evaluation Benchmark — Gustafson et al. (2023) (arXiv:2309.00035, 2023)

What this evaluates

This benchmark probes the intersectional fairness of computer vision models by evaluating their performance across diverse demographic attributes (e.g., skin tone, gender presentation, hair type) and person-related categories (e.g., occupations, hobbies). It measures whether models exhibit systematic performance disparities when detecting, classifying, or segmenting individuals with different attribute combinations.

Datasets

• FACET — total 32000; splits: test (-1)

Metrics

• accuracy / mAP / mIoU (primary) — range: [0, 1]
  • Standard computer vision metrics are computed for classification, detection, and segmentation tasks respectively. Performance is then stratified by 13 sensitive demographic attributes and their intersections to quantify disparity.

Input / output format

Input: RGB images containing multiple people, each annotated with person-related categories (52 classes) and 13 sensitive demographic attributes, along with ground-truth bounding boxes and segmentation masks.

Output: Per-image predictions including class labels, bounding box coordinates, and/or segmentation masks for each detected person.

Scoring recipe

def compute_fairness_metrics(preds, gold, attrs):
    results = {}
    for attr, values in attrs.items():
        for val in values:
            subset_gold = [g for g in gold if attrs[g['id']][attr] == val]
            subset_pred = [p for p in preds if attrs[p['id']][attr] == val]
            results[f'{attr}_{val}'] = calculate_metric(subset_pred, subset_gold)
    return results

Common pitfalls

• Evaluating only single demographic attributes instead of intersectional combinations, which masks compounded biases.
• Ignoring the long-tailed distribution of the 52 person categories, which can skew aggregate performance metrics.
• Using image-level annotations instead of person-level bounding boxes/masks, leading to incorrect per-instance fairness calculations.

Evidence (verbatim from paper)

The goal of our benchmark is to evaluate and analyze how vision models perform across different demographic and additional attributes for different categories of people. This analysis requires (1) images that contain people with a diverse set of attributes and (2) images that contain people matching a variety of person-related categories.

Citation

@misc{gustafson2023facet,
  title={FACET: Fairness in Computer Vision Evaluation Benchmark},
  author={Gustafson et al. (2023)},
  year={2023},
  note={arXiv:2309.00035}
}

• arXiv: 2309.00035