Flower Framework Eval

flower-framework-eval

Flower: A Friendly Federated Learning Research Framework — Beutel et al. (2020) (arXiv:2007.14390, 2020)

What this evaluates

Evaluates the scalability, heterogeneity handling, realism, and privacy overhead of the Flower federated learning framework across various datasets and device configurations.

Datasets

• Amazon Book Reviews — total 51000000; splits: test (1000000)
• FEMNIST — total ?; splits: train (-1)
• RealWorld — total ?; splits: train (-1)
• CIFAR-10 — total ?; splits: train (-1)
• FashionMNIST — total ?; splits: train (-1)

Metrics

• training time (primary) — range: other
  • Total elapsed time in seconds or minutes to complete federated training rounds, encompassing local SGD execution, model parameter communication, and server-side aggregation.
• accuracy — range: [0, 1]
  • Classification accuracy measured on a held-out test set after each training round.
• framework overhead latency — range: other
  • Mean time in milliseconds per round spent on framework-specific operations such as gradient serialization, GRPC communication, and buffer conversion, excluding local model training.

Input / output format

Input: Federated datasets partitioned across clients (text reviews, images, or time-series sensor data), server configuration parameters (clients per round, local epochs, aggregation strategy), and hardware/network constraints (CPU/GPU specs, simulated bandwidth limits).

Output: Aggregated global model weights, per-round accuracy curves, total training time, per-device energy consumption, and framework overhead latency metrics.

Scoring recipe

def evaluate_framework(rounds, client_updates, test_data):
    total_time = 0.0
    accuracy_history = []
    for r in range(rounds):
        start = time.time()
        global_model = aggregate_updates(client_updates[r])
        end = time.time()
        total_time += (end - start)
        acc = compute_accuracy(global_model, test_data)
        accuracy_history.append(acc)
    return {'training_time': total_time, 'accuracy_curve': accuracy_history}

Common pitfalls

• Assuming more participating clients per round always accelerates convergence; empirical thresholds exist where extra clients slow it down due to data distribution heterogeneity.
• Measuring only model training time while ignoring framework communication overhead, which can dominate on resource-constrained edge devices.
• Treating simulated network or compute heterogeneity as uniform; real devices exhibit significant variance in processing speed and bandwidth that directly impacts round completion time.

Evidence (verbatim from paper)

The FL training time is aggregated over 40 rounds, and includes the time taken to perform local 10 local epochs of SGD on the client, communicating model parameters between the server and the client, and updating the global model on the server.

Citation

@misc{beutel2020flower,
  title={Flower: A Friendly Federated Learning Research Framework},
  author={Beutel et al. (2020)},
  year={2020},
  note={arXiv:2007.14390}
}

• arXiv: 2007.14390