Tgbsseq Eval

tgbsseq-eval

TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics — Lu Yi et al. (2025) (arXiv:2502.02975, 2025)

What this evaluates

Evaluates temporal graph neural networks on future link prediction tasks, specifically probing their ability to generalize to unseen edges and capture complex sequential dynamics rather than memorizing repeated interactions.

Datasets

• ML-20M — total ?; splits: train (-1), val (-1), test (-1)
• Taobao — total ?; splits: train (-1), val (-1), test (-1)
• Yelp — total ?; splits: train (-1), val (-1), test (-1)
• GoogleLocal — total ?; splits: train (-1), val (-1), test (-1)
• Wikipedia — total ?; splits: train (-1), val (-1), test (-1)
• Reddit — total ?; splits: train (-1), val (-1), test (-1)
• Flickr — total ?; splits: train (-1), val (-1), test (-1)
• YouTube — total ?; splits: train (-1), val (-1), test (-1)
• Patent — total ?; splits: train (-1), val (-1), test (-1)
• WikiLink — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: averaged over all queries, computes the reciprocal of the rank of the correct answer (1/rank).

Input / output format

Input: Timestamped sequence of graph interactions (edges) up to a cutoff time, used to predict the next edge in the sequence.

Output: Ranking of candidate next edges or nodes, evaluated via MRR score.

Scoring recipe

def compute_mrr(ranks):
    return sum(1.0 / r for r in ranks) / len(ranks)
# ranks: list of integer positions (1-indexed) of the ground-truth edge in the predicted ranking

Common pitfalls

• Methods marked 'OOT' (Out of Time) are excluded from performance comparison, which can skew efficiency vs. accuracy trade-offs.
• Batch size is fixed at 200 for GoogleLocal but increased to 400 for other datasets, potentially affecting convergence and final scores across datasets.
• Temporal graphs require strict time-aware train/val/test splits; using random splits would leak future information and invalidate MRR scores.

Evidence (verbatim from paper)

Table 3: MRR of eight popular temporal GNN methods and SGNN-HN on four recommendation datasets and two previously established datasets (e.g., Wikipedia and Reddit). “OOT” denotes that the method failed to complete one epoch of training within 24 hours. The first, second, and third place rankings are highlighted accordingly. Each result is the average of three runs with different random seeds with reported standard deviation.

Citation

@misc{lu2025tgbsseq,
  title={TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics},
  author={Lu Yi et al. (2025)},
  year={2025},
  note={arXiv:2502.02975}
}

• arXiv: 2502.02975