Pg Gnn Eval

pg-gnn-eval

Going Deeper into Permutation-Sensitive Graph Neural Networks — Huang et al. (2022) (arXiv:2205.14368, 2022)

What this evaluates

Evaluates the expressivity and predictive performance of permutation-sensitive Graph Neural Networks (PG-GNN) on synthetic substructure counting tasks and real-world graph classification/regression benchmarks. It probes the model's ability to capture pairwise node correlations and higher-order substructures (triangles, 4-cliques) compared to standard permutation-invariant GNNs.

Datasets

• Erdős-Rényi random graphs — total ?; splits: test (-1)
• Random regular graphs — total ?; splits: test (-1)
• TUDataset (PROTEINS, NCI1, IMDB-B, IMDB-M, COLLAB) — total ?; splits: test (-1)
• MNIST — total ?; splits: test (-1)
• ZINC — total ?; splits: test (-1)

Metrics

• Mean Absolute Error (MAE) — range: other
  • The average of the absolute differences between predicted and ground-truth substructure counts or regression targets. Computed as the mean of |y_pred - y_true| across all test instances.
• Accuracy (%) (primary) — range: percent
  • The percentage of correctly classified graphs out of the total test set. Computed as (number of correct predictions / total number of predictions) * 100.

Input / output format

Input: Graphs represented by adjacency matrices and node feature matrices. For real-world datasets, node features are one-hot encoded categorical labels (bioinformatics) or node degrees (social networks). For synthetic tasks, graphs are Erdős-Rényi or random regular graphs with ground-truth incidence substructure counts.

Output: Predicted counts for incidence triangles/4-cliques (synthetic), or predicted graph class labels (TUDataset, MNIST) / regression values (ZINC).

Scoring recipe

def compute_mae(predictions, targets):
    return np.mean(np.abs(np.array(predictions) - np.array(targets)))

def compute_accuracy(predictions, targets):
    return np.mean(np.array(predictions) == np.array(targets)) * 100

Common pitfalls

• Failing to replicate the exact data splits referenced in Xu et al. (for TUDataset) and Dwivedi et al. (for MNIST/ZINC), which are not fully detailed in the main text.
• Not averaging results over multiple random seeds (5 for synthetic, 4 for real-world) and reporting standard deviations as required by the protocol.
• Confusing the node-level incidence substructure counting task on synthetic graphs with the standard graph-level classification/regression tasks on real-world benchmarks.

Evidence (verbatim from paper)

Table 2 summarizes the results measured by Mean Absolute Error (MAE, lower is better) for incidence triangle counting. We report the average and standard deviation of testing MAEs over 5 runs with 5 different seeds. ... Table 3: Results (measured by accuracy: %) on TUDataset.

Citation

@misc{huang2022going,
  title={Going Deeper into Permutation-Sensitive Graph Neural Networks},
  author={Huang et al. (2022)},
  year={2022},
  note={arXiv:2205.14368}
}

• arXiv: 2205.14368