Relbench Eval

relbench-eval

RelBench: A Benchmark for Deep Learning on Relational Databases — Robinson et al. (2024) (arXiv:2407.20060, 2024)

What this evaluates

Evaluates the ability of graph neural networks to learn from relational databases by predicting entity attributes (classification and regression) and discovering relationships between entities (link prediction) using primary-foreign key graph structures.

Datasets

• RELBENCH — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/snap-stanford/relbench

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Computed by varying the classification threshold and plotting the true positive rate against the false positive rate.
• MAE — range: other
  • Mean Absolute Error: (1/N) * Σ|y_true - y_pred|.
• MAP — range: [0, 1]
  • Mean Average Precision: average of precision values at each rank where a relevant item is retrieved, averaged over all queries.

Input / output format

Input: Relational database tables represented as a heterogeneous graph where entities are nodes and primary-foreign key relationships are edges. Node features include categorical/numerical attributes and text embeddings (GloVe/BERT), along with temporal information relative to a query time.

Output: For classification: predicted class probabilities or labels. For regression: predicted continuous values. For link prediction: ranked list of candidate linked entities.

Scoring recipe

def score(predictions, golds, task_type):
    if task_type == 'classification':
        return roc_auc_score(golds, predictions)
    elif task_type == 'regression':
        return mean_absolute_error(golds, predictions)
    elif task_type == 'link_prediction':
        return mean_average_precision(golds, predictions)

Common pitfalls

• Using randomly permuted edges instead of primary-foreign key relationships severely degrades performance.
• Ignoring temporal information (time embeddings) leads to significant performance drops across tasks.
• Masking node features or text embeddings results in much worse GNN performance compared to using rich features.

Evidence (verbatim from paper)

Tables 6, 7 and 8 show mean and standard deviations over 5 runs for the entity classification, entity regression and link prediction results respectively. Table 6: Entity classification results (AUROC mean±std over 5 runs, higher is better) on RELBENCH.

Citation

@misc{robinson2024relbench,
  title={RelBench: A Benchmark for Deep Learning on Relational Databases},
  author={Robinson et al. (2024)},
  year={2024},
  note={arXiv:2407.20060}
}

• arXiv: 2407.20060