Benchtemp Eval

benchtemp-eval

BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks — Huang et al. (2023) (arXiv:2308.16385, 2023)

What this evaluates

Evaluates the effectiveness and efficiency of Temporal Graph Neural Networks (TGNNs) on link prediction and node classification tasks. It probes model performance across transductive and inductive settings (New-Old/New-New) to ensure fair cross-model comparisons.

Datasets

• BenchTemp (15 datasets) — total ?; splits: test (-1); repo https://github.com/qianghuangwhu/benchtemp

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Computed as the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• AP — range: [0, 1]
  • Average Precision, the area under the precision-recall curve.
• Runtime — range: other
  • Seconds per epoch required for training.
• Epochs — range: other
  • Number of epochs until early stopping triggers.
• RAM — range: other
  • Maximum CPU RAM usage during training.
• GPU Memory — range: other
  • Maximum GPU memory usage during training.

Input / output format

Input: Temporal graph data (nodes, edges, timestamps) formatted for link prediction or node classification under transductive or inductive (New-Old/New-New) settings.

Output: Predicted link/node probabilities. Efficiency logs: runtime per epoch, epochs to convergence, peak RAM/GPU memory.

Scoring recipe

def compute_auc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)

def compute_ap(y_true, y_pred):
    from sklearn.metrics import average_precision_score
    return average_precision_score(y_true, y_pred)

# Run 3 times, report mean ± std. Early stop: patience=3, tol=1e-3, timeout=48h.

Common pitfalls

• Early stopping uses a patience of 3 and tolerance of 1e-3, which may cause models to terminate prematurely if validation metrics fluctuate.
• A 48-hour timeout is enforced; models that do not converge or encounter runtime errors are marked with '—' or '*' and excluded from averages.
• Standard deviations may be reported as zero if a model fails to complete multiple epochs within the timeout, leading to only one valid run.

Evidence (verbatim from paper)

Evaluation Metrics. We use the Evaluator module, choosing AUC and AP for the link prediction task, and AUC for the node classification task, following the prior works [1-7]. In addition, we report efficiency metrics, as shown in Table 4. Protocol. We run each job three times (unless timed out) and report the mean and standard deviation. We use an EarlyStopMonitor with a patience of 3 and tolerance of 10^{-3}, and set a timeout (48 hours).

Citation

@misc{huang2023benchtemp,
  title={BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2308.16385}
}

• arXiv: 2308.16385