Openfgl Eval

openfgl-eval

OpenFGL: A Comprehensive Benchmark for Federated Graph Learning — Li et al. (2024) (arXiv:2408.16288, 2024)

What this evaluates

Evaluates federated graph learning (FGL) algorithms across effectiveness, robustness, and efficiency dimensions. It probes how well distributed GNN training handles data heterogeneity, local noise/sparsity, low client participation, and privacy constraints across diverse graph topologies and simulation scenarios.

Datasets

• MUTAG — total ?; splits: test (-1)
• BZR — total ?; splits: test (-1)
• COX2 — total ?; splits: test (-1)
• ENZYMES — total ?; splits: test (-1)
• DD — total ?; splits: test (-1)
• PROTEINS — total ?; splits: test (-1)
• COLLAB — total ?; splits: test (-1)
• BINARY — total ?; splits: test (-1)
• MULTI — total ?; splits: test (-1)
• Cora — total ?; splits: test (-1)
• CiteSeer — total ?; splits: test (-1)
• PubMed — total ?; splits: test (-1)
• Photo — total ?; splits: test (-1)
• Computers — total ?; splits: test (-1)
• Products — total ?; splits: test (-1)
• Chameleon — total ?; splits: test (-1)
• Actor — total ?; splits: test (-1)
• Ratings — total ?; splits: test (-1)
• DHFR — total ?; splits: test (-1)
• Squirrel — total ?; splits: test (-1)
• Questions — total ?; splits: test (-1)
• AIDS — total ?; splits: test (-1)
• NCI1 — total ?; splits: test (-1)
• CS — total ?; splits: test (-1)
• Physics — total ?; splits: test (-1)

Metrics

• test accuracy (primary) — range: percent
  • Percentage of correctly predicted node labels on the held-out test set, calculated as (number of correct predictions / total test samples) * 100.

Input / output format

Input: Graph-structured data (node features, adjacency matrices/edges, and labels) partitioned across multiple clients according to federated simulation strategies (Graph-FL or Subgraph-FL), with optional injected noise, sparsity, or privacy constraints.

Output: Predicted node labels for each test node, aggregated into a test accuracy percentage per dataset and algorithm configuration.

Scoring recipe

def compute_test_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Federated collaboration yields diminishing returns on small-scale datasets (e.g., MUTAG, BZR, COX2) due to insufficient data to support effective multi-client training.
• Topology-driven FGL algorithms are highly sensitive to edge noise, which severely disrupts local updates and server-side collaboration more than feature or label noise.
• Some baselines fail on large-scale datasets (e.g., ogbn-products) due to out-of-memory (OOM) errors, limiting fair scalability comparisons in real-world deployments.

Evidence (verbatim from paper)

To answer Q1, in addition to the federated multi-client collaboration, we introduce "Local" to represent solely local training for analyzing the advantages and potential limitations of FGL. Based on this, to answer Q2, we present the end-to-end performance in Table 6 and Table 7. Table 6: Graph-FL test accuracy (%) . The best result is bold.

Citation

@misc{li2024openfgl,
  title={OpenFGL: A Comprehensive Benchmark for Federated Graph Learning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2408.16288}
}

• arXiv: 2408.16288