Shake Gnn Eval

shake-gnn-eval

SHAKE-GNN: Scalable Hierarchical Kirchhoff-Forest Graph Neural Network — Cui et al. (2025) (arXiv:2509.22100, 2025)

What this evaluates

Evaluates the predictive accuracy and training efficiency of a hierarchical graph neural network that uses Kirchhoff Forest-based stochastic coarsening for graph classification. The benchmark probes whether multi-resolution graph decomposition can maintain competitive performance while significantly reducing computational costs across molecular and social network domains.

Datasets

• MolHIV — total ?; splits: train (-1), test (-1)
• MolPPA — total ?; splits: train (-1), test (-1)
• COLLAB — total ?; splits: train (-1), test (-1)
• DD — total ?; splits: train (-1), test (-1)
• REDDIT-MULTI-12K — total ?; splits: train (-1), test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Receiver Operating Characteristic Area Under Curve; computed by integrating the area under the curve plotting true positive rate against false positive rate at various classification thresholds.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.

Input / output format

Input: Graph structures containing node features, edge features, and graph-level labels for classification.

Output: Graph-level class predictions (binary or multi-class depending on the dataset).

Scoring recipe

def compute_metric(y_true, y_pred, dataset_name):
    if dataset_name in ['MolHIV', 'MolPPA']:
        return roc_auc_score(y_true, y_pred)
    return accuracy_score(y_true, y_pred)

Common pitfalls

• Training time is reported in absolute minutes on a specific GPU (NVIDIA RTX A6000), making cross-hardware efficiency comparisons invalid.
• The resolution parameter q* is optimized via spectral evaluation on the training split, which may inadvertently leak information or overfit to the coarsening objective.
• Hyperparameters are largely fixed across diverse datasets, potentially underperforming on tasks that require different learning rates or batch sizes.

Evidence (verbatim from paper)

For each dataset, we report both the training time (in minutes) and evaluation performance (ROC-AUC for molecular datasets, accuracy for social/protein datasets). Results are given for the original graphs and for the coarsened graphs, enabling direct comparison of efficiency and predictive power.

Citation

@misc{cui2025shakegnn,
  title={SHAKE-GNN: Scalable Hierarchical Kirchhoff-Forest Graph Neural Network},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2509.22100}
}

• arXiv: 2509.22100