# Fedhpo B Eval

> Evaluates the effectiveness of federated hyperparameter optimization (FedHPO) methods across diverse FL tasks. It measures how well optimizers can find high-performing hyperparameter configurations under distributed, communication-constrained, and heterogeneous data settings. Use when the user wants to benchmark on FedHPO-B, or asks about evaluating this task. Reports mean rank.

- Skill: `qhjqhj00/fedhpo-b-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/fedhpo-b-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/fedhpo-b-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/fedhpo-b-eval

---


# fedhpo-b-eval

> FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization — Wang et al. (2022) (arXiv:2206.03966, 2022)

## What this evaluates

Evaluates the effectiveness of federated hyperparameter optimization (FedHPO) methods across diverse FL tasks. It measures how well optimizers can find high-performing hyperparameter configurations under distributed, communication-constrained, and heterogeneous data settings.

## Datasets

- **FedHPO-B** — total 20; splits: train (-1), val (-1), test (-1); repo https://github.com/alibaba/FederatedScope

## Metrics

- `best-seen validation loss` — range: [0, inf)
  - The minimum validation loss achieved during the HPO search process for a given task and optimizer. Lower values indicate better optimization performance.
- `mean rank` **(primary)** — range: other
  - For each of the 20 FL tasks, optimizers are ranked by their best-seen validation loss (lower is better). The mean rank is computed across all tasks. Lower values indicate better overall performance.

## Input / output format

**Input**: A federated learning task specification including model architecture, data distribution, system parameters (e.g., sample_rate, max rounds), and a configurable hyperparameter search space.

**Output**: A hyperparameter configuration proposed by the optimizer, along with the corresponding validation loss and rank.

## Scoring recipe

```python
def compute_mean_rank(results_per_task):
    # results_per_task: dict mapping task_id -> list of (optimizer_id, best_val_loss)
    ranks_per_task = {}
    for task_id, runs in results_per_task.items():
        sorted_runs = sorted(runs, key=lambda x: x[1])
        for rank, (opt_id, _) in enumerate(sorted_runs, start=1):
            ranks_per_task.setdefault(opt_id, []).append(rank)
    mean_ranks = {opt_id: sum(ranks)/len(ranks) for opt_id, ranks in ranks_per_task.items()}
    return mean_ranks
```

## Common pitfalls

- Failing to standardize the total communication budget (number of rounds) across single-fidelity and multi-fidelity optimizers.
- Ignoring FL-specific system constraints like client sampling rates (sample_rate) when comparing against centralized HPO baselines.
- Not repeating experiments with different random seeds, leading to high variance in rank comparisons.

## Evidence (verbatim from paper)

> The best-seen validation loss is monitored for each optimizer (for multi-fidelity optimizers, higher fidelity results are preferred over lower ones). We sort the optimizers by their best-seen results and compare their mean ranks on these 20 FL tasks.

## Citation

```bibtex
@misc{wang2022fedhpo,
  title={FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2206.03966}
}
```

- arXiv: 2206.03966