Fedhpo B Eval

fedhpo-b-eval

FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization — Wang et al. (2022) (arXiv:2206.03966, 2022)

What this evaluates

Evaluates the effectiveness of federated hyperparameter optimization (FedHPO) methods across diverse FL tasks. It measures how well optimizers can find high-performing hyperparameter configurations under distributed, communication-constrained, and heterogeneous data settings.

Datasets

• FedHPO-B — total 20; splits: train (-1), val (-1), test (-1); repo https://github.com/alibaba/FederatedScope

Metrics

• best-seen validation loss — range: [0, inf)
  • The minimum validation loss achieved during the HPO search process for a given task and optimizer. Lower values indicate better optimization performance.
• mean rank (primary) — range: other
  • For each of the 20 FL tasks, optimizers are ranked by their best-seen validation loss (lower is better). The mean rank is computed across all tasks. Lower values indicate better overall performance.

Input / output format

Input: A federated learning task specification including model architecture, data distribution, system parameters (e.g., sample_rate, max rounds), and a configurable hyperparameter search space.

Output: A hyperparameter configuration proposed by the optimizer, along with the corresponding validation loss and rank.

Scoring recipe

def compute_mean_rank(results_per_task):
    # results_per_task: dict mapping task_id -> list of (optimizer_id, best_val_loss)
    ranks_per_task = {}
    for task_id, runs in results_per_task.items():
        sorted_runs = sorted(runs, key=lambda x: x[1])
        for rank, (opt_id, _) in enumerate(sorted_runs, start=1):
            ranks_per_task.setdefault(opt_id, []).append(rank)
    mean_ranks = {opt_id: sum(ranks)/len(ranks) for opt_id, ranks in ranks_per_task.items()}
    return mean_ranks

Common pitfalls

• Failing to standardize the total communication budget (number of rounds) across single-fidelity and multi-fidelity optimizers.
• Ignoring FL-specific system constraints like client sampling rates (sample_rate) when comparing against centralized HPO baselines.
• Not repeating experiments with different random seeds, leading to high variance in rank comparisons.

Evidence (verbatim from paper)

The best-seen validation loss is monitored for each optimizer (for multi-fidelity optimizers, higher fidelity results are preferred over lower ones). We sort the optimizers by their best-seen results and compare their mean ranks on these 20 FL tasks.

Citation

@misc{wang2022fedhpo,
  title={FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2206.03966}
}

• arXiv: 2206.03966