Gnn Ak Eval

gnn-ak-eval

From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awareness — Zhao et al. (2021) (arXiv:2110.03753, 2021)

What this evaluates

Evaluates the expressiveness and practical performance of GNN-AK, a framework that replaces star-shaped neighbor aggregation with subgraph-based encoding in Message Passing Neural Networks. It probes the model's ability to distinguish complex graph structures (e.g., strongly regular graphs, substructures) and predict graph-level properties on standard benchmarks.

Datasets

• ZINC-12K — total 12000; splits: train/val/test (-1)
• CIFAR10 — total ?; splits: train/val/test (-1)
• PATTERN — total ?; splits: train/val/test (-1)
• MolHIV — total ?; splits: train/val/test (-1)
• MolPCBA — total ?; splits: train/val/test (-1)
• EXP — total 600; splits: train/val/test (-1)
• SR25 — total 15; splits: train/val/test (-1)

Metrics

• accuracy (ACC) (primary) — range: [0, 1]
  • Fraction of correctly classified graphs out of the total number of graphs in the dataset.
• mean absolute error (MAE) (primary) — range: [0, inf)
  • Average of absolute differences between predicted and true graph-level regression values. For graph property tasks, log10(MAE) is reported.
• ROC-AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates for binary classification.
• Average Precision (AP) — range: [0, 1]
  • Area under the Precision-Recall curve, summarizing the precision-recall trade-off for multi-label classification.

Input / output format

Input: Graph-structured data consisting of nodes, edges, and node/edge features. Tasks include graph classification and graph property regression.

Output: For classification: a discrete class label. For regression: a continuous scalar value. Metrics are computed over the entire dataset.

Scoring recipe

def compute_metrics(predictions, labels, task_type):
    if task_type == 'classification':
        return sum(1 for p, l in zip(predictions, labels) if p == l) / len(labels)
    elif task_type == 'regression':
        return sum(abs(p - l) for p, l in zip(predictions, labels)) / len(labels)
    elif task_type == 'binary':
        return roc_auc_score(labels, predictions)
    elif task_type == 'multilabel':
        return average_precision_score(labels, predictions)

Common pitfalls

• Graph property regression metrics are reported as log10(MAE), not raw MAE.
• PPGN-AK+ frequently runs out of memory (OOM) on larger real-world datasets due to quadratic complexity.
• SubgraphDrop sampling (R parameter) trades off performance for runtime/memory, requiring careful selection of R for fair comparison.

Evidence (verbatim from paper)

Table 1: Simulation dataset performance: $GNN-AK^{(+)}$ boosts base GNN across tasks, empirically verifying expressiveness lift. (ACC: accuracy, MAE: mean absolute error, OOM: out of memory)

Table 3: Real-world dataset performance: $GNN-AK^{+}$ achieves SOTA performance for ZINC-12K, CIFAR10, and PATTERN. (OOM: out of memory, -: missing values from literature)

Citation

@misc{zhao2021gnnak,
  title={From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awareness},
  author={Zhao et al. (2021)},
  year={2021},
  note={arXiv:2110.03753}
}

• arXiv: 2110.03753