Flair Eval

flair-eval

FLAIR: Federated Learning Annotated Image Repository — Song et al. (2022) (arXiv:2207.08869, 2022)

What this evaluates

Evaluates federated learning models on real-world, non-IID image data with user-level heterogeneity and long-tailed label distributions. It probes how model convergence and multi-label classification performance degrade under privacy constraints (differential privacy) and distributed training compared to centralized baselines.

Datasets

• FLAIR — total 429078; splits: train (-1), val (-1), test (-1); repo https://github.com/apple/ml-flair

Metrics

• averaged precision (AP) (primary) — range: percent
  • Standard multi-label averaged precision computed over predicted and ground-truth binary vectors. Reported as both overall (micro-averaged across all examples) and per-class (macro-averaged across classes).

Input / output format

Input: Single image resized to 224×224 pixels. In federated settings, images are partitioned by user ID, creating non-IID distributions.

Output: Binary multi-label vector indicating presence/absence of each class in the coarse-grained (17 classes) or fine-grained (1,628 classes) taxonomy.

Scoring recipe

def compute_ap(y_true, y_pred, average='micro'):
    if average == 'micro':
        tp = (y_true & y_pred).sum()
        fp = (~y_true & y_pred).sum()
        fn = (y_true & ~y_pred).sum()
        prec = tp / (tp + fp + 1e-8)
        rec = tp / (tp + fn + 1e-8)
        return 2 * prec * rec / (prec + rec + 1e-8)
    else:  # macro
        class_scores = []
        for c in range(y_true.shape[1]):
            tp = ((y_true[:, c] == 1) & (y_pred[:, c] == 1)).sum()
            fp = ((y_true[:, c] == 0) & (y_pred[:, c] == 1)).sum()
            fn = ((y_true[:, c] == 1) & (y_pred[:, c] == 0)).sum()
            prec = tp / (tp + fp + 1e-8)
            rec = tp / (tp + fn + 1e-8)
            class_scores.append(2 * prec * rec / (prec + rec + 1e-8))
        return np.mean(class_scores)

Common pitfalls

• Confusing micro-averaged (overall) vs macro-averaged (per-class) metrics, which show drastically different performance gaps and are reported separately in tables.
• The differential privacy experiments simulate large cohort noise using a small cohort; failing to account for this SNR scaling trick leads to incorrect noise-level interpretation.
• Ignoring the inherent non-IID user-level partitioning, which is the core challenge the benchmark is designed to test.

Evidence (verbatim from paper)

Evaluation metrics. We use standard multi-label classification metrics for the benchmark, including precision (percentage of predicted objects that are actually in the images), recall (percentage of objects in the images are predicted), F1 score, and averaged precision (AP) score. We report overall (micro-averaged) metrics, obtained by averaging over all examples, and per-class (macro-averaged) metrics, obtained by taking the average over classes of the average over examples restricted to a specific class.

Citation

@misc{song2022flair,
  title={FLAIR: Federated Learning Annotated Image Repository},
  author={Song et al. (2022)},
  year={2022},
  note={arXiv:2207.08869}
}

• arXiv: 2207.08869