Fedgraphnn System Eval

fedgraphnn-system-eval

FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks — He et al. (2021) (arXiv:2104.07145, 2021)

What this evaluates

Evaluates the computational efficiency and security overhead of federated graph neural network training. Measures how system-level metrics like training time, FLOPs, and parameter counts scale across diverse graph datasets under non-IID data partitioning and secure aggregation protocols.

Datasets

• SIDER — total ?; splits: train (-1)
• BACE — total ?; splits: train (-1)
• Clintox — total ?; splits: train (-1)
• BBBP — total ?; splits: train (-1)
• Tox21 — total ?; splits: train (-1)
• FreeSolv — total ?; splits: train (-1)
• ESOL — total ?; splits: train (-1)
• Lipo — total ?; splits: train (-1)
• hERG — total ?; splits: train (-1)
• QM9 — total ?; splits: train (-1)
• Ciao — total ?; splits: train (-1)
• Epinions — total ?; splits: train (-1)
• CORA — total ?; splits: train (-1)
• Citeseer — total ?; splits: train (-1)
• DBLP — total ?; splits: train (-1)
• PubMed — total ?; splits: train (-1)

Metrics

• Wall-clock Time (primary) — range: seconds
  • Total elapsed time for federated training across all clients and rounds, measured in seconds, minutes, or hours.
• Average FLOP — range: FLOPs
  • Mean floating-point operations performed per client during local training, measured in thousands or millions of FLOPs.
• Parameters — range: count
  • Total number of trainable parameters in the GNN model (GCN, GAT, or GraphSAGE).

Input / output format

Input: Graph datasets partitioned non-IID across multiple clients. Each client receives a local subgraph and trains a GNN (GCN, GAT, or GraphSAGE) using FedAvg or LightSecAgg aggregation.

Output: System performance logs reporting wall-clock training time, average FLOPs per client, and model parameter counts. For security evaluation, total running time of secure aggregation protocols.

Scoring recipe

def compute_metrics(predictions, gold):
    # System metrics are logged directly by the FedGraphNN profiler, not derived from predictions/gold
    wall_clock_time = log_wall_clock_time()
    avg_flops = log_average_flops()
    param_count = count_parameters()
    return {'wall_clock_time': wall_clock_time, 'avg_flops': avg_flops, 'params': param_count}

Common pitfalls

• Confusing system efficiency metrics (time, FLOPs) with model predictive performance (accuracy, F1).
• Assuming standard IID data splits; the benchmark explicitly uses realistic non-IID partitioning.
• Comparing wall-clock times across different hardware configurations without normalization, as the benchmark specifies 8x NVIDIA Quadro RTX 5000 GPUs.

Evidence (verbatim from paper)

Table 6: System-Level Performance Metrics for Graph-Level FedGraphNN tasks with FedAvg (Hardware: 8 x NVIDIA Quadro RTX 5000 GPU (16GB/GPU); RAM: 512G; CPU: Intel Xeon Gold 5220R 2.20GHz). ... Wall-clock Time ... Average FLOP ... Parameters

Citation

@misc{he2021fedgraphnn,
  title={FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks},
  author={He et al. (2021)},
  year={2021},
  note={arXiv:2104.07145}
}

• arXiv: 2104.07145