Eth Phishing Detection Eval

eth-phishing-detection-eval

TTAGN: Temporal Transaction Aggregation Graph Network for Ethereum Phishing Scams Detection — Sijia Li et al. (arXiv:2204.13442, 2022)

What this evaluates

Evaluates a model's ability to detect phishing addresses on the Ethereum blockchain by analyzing temporal transaction dynamics and graph topology. It probes whether the model can effectively fuse edge-level temporal patterns with node-level structural and statistical features to distinguish malicious accounts from legitimate ones.

Datasets

• $D_1$ — total 30000; splits: train (24000), test (6000)
• $D_2$ — total 40000; splits: train (32000), test (8000)
• $D_3$ — total 50000; splits: train (40000), test (10000)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve, calculated across multiple classification thresholds. Measures the model's ability to rank positive (phishing) instances higher than negative ones.
• Recall — range: [0, 1]
  • True Positive Rate: TP / (TP + FN). Represents the percentage of known phishing nodes correctly detected by the model.
• Precision — range: [0, 1]
  • TP / (TP + FP). Represents the percentage of accounts flagged as suspicious that are actually phishing addresses.
• F1-score — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Provides a single comprehensive score balancing false positives and false negatives.

Input / output format

Input: Graph-structured data containing Ethereum addresses (nodes) and transaction records (edges). Each node is represented by statistical features (e.g., 219-dimensional node attributes) and temporal transaction sequences. The model processes these to generate node embeddings for classification.

Output: Binary classification label per node: 'phishing' or 'non-phishing'.

Scoring recipe

def compute_metrics(y_true, y_pred_proba, threshold=0.5):
    y_pred = (y_pred_proba >= threshold).astype(int)
    tp = np.sum((y_pred == 1) & (y_true == 1))
    fn = np.sum((y_pred == 0) & (y_true == 1))
    fp = np.sum((y_pred == 1) & (y_true == 0))
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    auc = roc_auc_score(y_true, y_pred_proba)
    return {'AUC': auc, 'Recall': recall, 'Precision': precision, 'F1-score': f1}

Common pitfalls

• The dataset is highly imbalanced; the authors explicitly upsample the minority (phishing) class with a ratio of 50 during training, which must be replicated for fair comparison.
• Data cleaning strictly removes transactions before 2016-08-02 and filters nodes with <5 or >1000 transactions, altering the original graph distribution and node degree statistics.
• Performance is sensitive to the maximum temporal sequence length; variable-length sequences outperform fixed-length ones, and shorter sequences sometimes yield better results due to information redundancy.

Evidence (verbatim from paper)

In this paper, we use the following four metrics to have a comprehensive evaluation of the performance of different methods in terms of Ethereum phishing scam detection: (1) Area Under Curve (AUC). The AUC metric is to calculate the area under the ROC curve formed by TPRs and FPRs with multiple thresholds, which is frequently used in binary classification tasks. (2) Recall. The recall rate means the percentage of known phishing nodes samples detected. (3) Precision. The precision rate means the percentage of real phishing nodes are in the accounts that are judged to be suspicious. (4) F1-score. F1-score is a comprehensive evaluation of the Precision and Recall score.

Citation

@misc{li2022ttagn,
  title={TTAGN: Temporal Transaction Aggregation Graph Network for Ethereum Phishing Scams Detection},
  author={Sijia Li et al.},
  year={2022},
  note={arXiv:2204.13442}
}

• arXiv: 2204.13442