Leaf Federated Eval

leaf-federated-eval

LEAF: A Benchmark for Federated Settings — Caldas et al. (2018) (arXiv:1812.01097, 2018)

What this evaluates

Evaluates federated learning algorithms under realistic constraints including device-level data skew, heterogeneous data distributions, and communication bottlenecks. It measures model accuracy after federated training across multiple simulated devices.

Datasets

• Shakespeare — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
• Sent140 — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
• FEMNIST — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
• CelebA — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
• Synthetic — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
• Reddit — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf

Metrics

• AccuracyTop1 (primary) — range: [0, 1]
  • Fraction of correctly predicted tokens or classes out of the total number of predictions on the test set. For text tasks, predictions of unknown and padding tokens are explicitly counted as incorrect.

Input / output format

Input: Per user/device, sequences of tokens (text) or images (FEMNIST/CelebA) or feature vectors (Synthetic). Data is partitioned per user with a 60/20/20 train/val/test split.

Output: Predicted class label or token for each input instance.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Data is inherently non-IID and partitioned per user, not globally shuffled.
• Evaluation must be performed on held-out test sets per user or a global test set, never on training data.
• Federated protocols require tracking communication rounds and client sampling ratios, not just final model accuracy.

Evidence (verbatim from paper)

We evaluate using AccuracyTop1. ... evaluate using AccuracyTop1 and consider all predictions of the unknown and padding tokens as incorrect.

Citation

@misc{caldas2018leaf,
  title={LEAF: A Benchmark for Federated Settings},
  author={Caldas et al. (2018)},
  year={2018},
  note={arXiv:1812.01097}
}

• arXiv: 1812.01097