Gnn Design Eval

gnn-design-eval

Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Models — Wang et al. (2024) (arXiv:2408.06717, 2024)

What this evaluates

Evaluates an LLM-guided framework's ability to automatically propose and refine Graph Neural Network architectures for node classification across diverse graph datasets, including out-of-distribution and heterophilic graphs, without requiring extensive training or search.

Datasets

• NAS-Bench-Graph & OOD Graphs (Cora, Citeseer, PubMed, CS, Physics, Photo, Computer, ogbn-arXiv, DBLP, Flickr, Actor) — total ?; splits: train/val/test (-1)

Metrics

• accuracy (primary) — range: percent
  • Mean node classification accuracy over 10 independent runs with different random seeds. Standard deviation is also reported alongside the mean.
• hit_rate — range: [0, 1]
  • Proportion of times the empirically top-performing benchmark graphs are included in the Top-Ns retrieved benchmarks by the similarity module.
• Kendall's tau — range: [-1, 1]
  • Rank correlation coefficient measuring the agreement between predicted task similarities and empirical performance similarities.

Input / output format

Input: Graph dataset properties (e.g., node/edge counts, feature dimensions, label distribution) and optional textual descriptions.

Output: A GNN architecture proposal (layer types, connections, hyperparameters) from a predefined unified search space.

Scoring recipe

def compute_accuracy(proposals, dataset):
    accs = []
    for run in range(10):
        model = propose_architecture(dataset.properties)
        train_and_validate(model, dataset)
        accs.append(model.test_accuracy)
    return np.mean(accs), np.std(accs)

def compute_hit_rate(retrieved_top_n, empirical_top_n):
    return len(set(retrieved_top_n) & set(empirical_top_n)) / len(empirical_top_n)

Common pitfalls

• Data leakage: Unseen datasets must be completely excluded from the knowledge retrieval process and anonymized during evaluation.
• Efficiency metric confusion: The paper uses '# proposal validations' as the basic time unit for LLM API traffic, not wall-clock time or FLOPs.
• Reporting format: Results must report mean accuracy ± standard deviation over exactly 10 runs, not single-run results.

Evidence (verbatim from paper)

To ensure reliability, we report the average accuracy and standard deviation over 10 runs. For automated methods, we also analyze the best-so-far accuracy after validating 1-30 model proposals to measure short-run efficiency.

Citation

@misc{wang2024designgnn,
  title={Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2408.06717}
}

• arXiv: 2408.06717