qhjqhj00 ethereum-fraud-detection-eval
Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision — Jin et al. (2024) (arXiv:2408.00641, 2024)
What this evaluates
Evaluates a model's ability to detect fraudulent Ethereum accounts by analyzing transaction interaction graphs and behavioral patterns. It specifically probes the model's capacity to handle imbalanced label distributions and distinguish between Ponzi schemes and phishing scams using self-supervised feature learning.
Datasets
- Ponzi Scheme Dataset — total 1342; splits: train (-1), val (-1), test (-1); repo https://github.com/GSec-Team/Meta-IFD
- Phish Scam Dataset — total 2763; splits: train (-1), val (-1), test (-1); repo https://github.com/GSec-Team/Meta-IFD
Metrics
Precision— range: percent- Ratio of correctly predicted fraudulent accounts to all accounts predicted as fraudulent. Calculated as TP / (TP + FP).
Recall— range: percent- Ratio of correctly predicted fraudulent accounts to all actual fraudulent accounts. Calculated as TP / (TP + FN).
Binary-F1(primary) — range: percent- Harmonic mean of Precision and Recall for binary classification. Calculated as 2 * (Precision * Recall) / (Precision + Recall).
Micro-F1— range: percent- F1 score calculated globally by counting total true positives, false negatives, and false positives across all classes.
Macro-F1— range: percent- F1 score calculated as the unweighted mean of F1 scores for each class, treating all classes equally regardless of support.
Input / output format
Input: Graph/interaction features derived from first- and second-order transaction objects of labeled Ethereum accounts, constructed into a HEIG.
Output: Binary classification label indicating whether an account is fraudulent (Ponzi/Phishing) or legitimate.
Scoring recipe
def compute_metrics(y_true, y_pred):
tp = ((y_true == 1) & (y_pred == 1)).sum()
fp = ((y_true == 0) & (y_pred == 1)).sum()
fn = ((y_true == 1) & (y_pred == 0)).sum()
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
binary_f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
return precision * 100, recall * 100, binary_f1 * 100
Common pitfalls
- Datasets exhibit significant label imbalance, making Macro-F1 and Micro-F1 report drastically different performance; readers must check which F1 variant is prioritized.
- Standard deviation is reported over multiple runs, so single-run results cannot be directly compared to the paper's reported values.
- Graph construction relies on first- and second-order transaction objects; omitting this preprocessing step will break reproducibility.
Evidence (verbatim from paper)
We collect a certain amount of labeled data from the Xblock and Etherscan platforms, covering 191 Ponzi accounts and 1151 non-Ponzi accounts, as well as 1206 phishing accounts and 1557 non-phishing accounts. ... TABLE IV: The results of fraud detection in terms of Precision(%, Recall(%, Binary-F1(%, Micro-F1(%, Macro-F1(%) and Standard Deviation(%).
Citation
@misc{jin2024metaifd,
title={Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision},
author={Jin et al. (2024)},
year={2024},
note={arXiv:2408.00641}
}
- arXiv: 2408.00641