Gnn Internet Routing Eval

gnn-internet-routing-eval

Benchmarking Graph Neural Networks for Internet Routing Data — Giakatos et al. (2022) (arXiv:2210.14189, 2022)

What this evaluates

Evaluates Graph Neural Networks and traditional ML models on a compiled Internet routing dataset for link prediction and node classification. It probes the ability to infer missing AS-AS connections and predict voluntary PeeringDB attributes under conditions of graph sparsity and label imbalance.

Datasets

• Internet Routing AS Graph Dataset — total ?; splits: train (-1), test (-1)

Metrics

• binary cross entropy (primary) — range: other
  • Standard binary cross-entropy loss used to optimize and evaluate link prediction tasks. Measures the difference between predicted link probabilities and actual link existence.
• cross entropy loss — range: other
  • Multi-class cross-entropy loss used for node classification tasks. Measures the difference between predicted attribute distributions and true labels.

Input / output format

Input: Node embeddings generated by GNNs or graph-embedding models (or raw node features for Random Forest). Link prediction uses node pairs; node classification uses individual node features.

Output: Binary prediction for link existence (link prediction) or multi-class label for AS attributes (node classification).

Scoring recipe

def compute_metric(predictions, gold, task):
    if task == 'link_prediction':
        return -mean(gold * log(predictions) + (1 - gold) * log(1 - predictions))
    else:
        return -mean(sum(gold_one_hot * log(predictions), axis=1))

Common pitfalls

• Only ~25% of ASes have PeeringDB labels, requiring masking of unlabeled nodes during MLP training/testing.
• Severe class imbalance and graph sparsity force balancing of train sets and merging of rare classes (<500 samples).
• The protocol specifies loss functions for evaluation rather than standard accuracy or AUC metrics.

Evidence (verbatim from paper)

We use a binary cross entropy as a loss function for the link prediction task and the cross entropy loss (due to the many classes) for the node classification tasks. Since labels are not known for all nodes, for the classification task, while we consider all nodes in the generation of the embeddings (GNN), we train/test the MLP by using only the values for the nodes with labels (i.e., by masking the nodes without labels).

Citation

@misc{giakatos2022benchmarking,
  title={Benchmarking Graph Neural Networks for Internet Routing Data},
  author={Giakatos et al. (2022)},
  year={2022},
  note={arXiv:2210.14189}
}

• arXiv: 2210.14189