qhjqhj00 elliptic-fraud-eval
A Systematic Evaluation Protocol of Graph-Derived Signals for Tabular Machine Learning — Heidrich et al. (2026) (arXiv:2603.13998, 2026)
What this evaluates
Evaluates the utility, robustness, and interpretability of graph-derived signals for tabular machine learning on a binary node classification task. It compares graph-augmented models against tabular baselines using statistical hypothesis testing and graph perturbation analysis to ensure reproducibility.
Datasets
- Elliptic — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graph-eval/graph-eval-protocol
Metrics
F1-score(primary) — range: [0, 1]- Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Aggregated across random seeds using trimmed aggregation to reduce outlier influence.
McNemar's test— range: [0, 1]- Statistical hypothesis test applied to paired predictions on identical test splits to assess significance of performance differences between baseline and graph-augmented models. Results with p ≤ 0.05 are considered statistically significant.
Input / output format
Input: Transaction-level features concatenated with graph-derived signals (or original features only for baseline). Binary node classification task.
Output: Binary prediction (licit or illicit) per node.
Scoring recipe
def compute_f1(y_true, y_pred):
tp = sum((y_true == 1) & (y_pred == 1))
fp = sum((y_true == 0) & (y_pred == 1))
fn = sum((y_true == 1) & (y_pred == 0))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
def mcnemar_pval(y_true, y_pred_a, y_pred_b):
b = sum((y_true != y_pred_a) & (y_true == y_pred_b))
c = sum((y_true == y_pred_a) & (y_true != y_pred_b))
if b + c == 0: return 1.0
chi2 = (abs(b - c) - 1)**2 / (b + c)
return 1 - chi2_cdf(chi2, df=1)
Common pitfalls
- Using an inductive split instead of the specified transductive setting (full graph available during training), which breaks the paired comparison assumption and alters structural information flow.
- Averaging results across seeds without using trimmed aggregation, which masks outlier runs and inflates perceived robustness.
- Reporting raw performance differences without applying McNemar's test on paired predictions, leading to false claims of statistical significance.
Evidence (verbatim from paper)
As the primary comparison metric, we report the F1-score, which balances precision and recall and enables direct comparison across classifiers and graph signal configurations. To assess whether observed performance differences between baseline and graph-augmented models are statistically significant, we apply McNemar’s test to paired predictions on identical test splits, considering results with p≤ 0.05 as statistically significant.
Citation
@misc{heidrich2026grapheval,
title={A Systematic Evaluation Protocol of Graph-Derived Signals for Tabular Machine Learning},
author={Heidrich et al. (2026)},
year={2026},
note={arXiv:2603.13998}
}
- arXiv: 2603.13998