Phishing Detection Eval

phishing-detection-eval

Towards Benchmark Datasets for Machine Learning Based Website Phishing Detection: An experimental study — Hannousse et al. (2020) (arXiv:2010.12847, 2020)

What this evaluates

This benchmark evaluates machine learning classifiers and feature selection strategies for detecting phishing websites. It probes the ability of models to distinguish between legitimate and malicious web pages using content-based, external service, and hybrid feature sets, while measuring classification accuracy and macro F1-score.

Datasets

• Collected Phishing Dataset — total 11430; splits: 10-fold cross-validation (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Ratio of correct predictions to total samples: (TP + TN) / (TP + TN + FP + FN).
• Macro F1-score — range: [0, 1]
  • Mean of class-wise F1-scores: (1/N) * Σ F1_i, where F1_i = 2 * (Precision_i * Recall_i) / (Precision_i + Recall_i).

Input / output format

Input: Feature vectors representing website attributes (content, external service, or hybrid combinations) fed into machine learning classifiers.

Output: Binary class label indicating whether the website is 'phishing' or 'legitimate'.

Scoring recipe

def score(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    acc = (tp + tn) / (tp + tn + fp + fn)
    prec_pos = tp / (tp + fp) if (tp + fp) else 0
    rec_pos = tp / (tp + fn) if (tp + fn) else 0
    f1_pos = 2 * prec_pos * rec_pos / (prec_pos + rec_pos) if (prec_pos + rec_pos) else 0
    prec_neg = tn / (tn + fn) if (tn + fn) else 0
    rec_neg = tn / (tn + fp) if (tn + fp) else 0
    f1_neg = 2 * prec_neg * rec_neg / (prec_neg + rec_neg) if (prec_neg + rec_neg) else 0
    macro_f1 = (f1_pos + f1_neg) / 2
    return acc, macro_f1

Common pitfalls

• The dataset is custom-collected and not publicly hosted under a standard benchmark name, so replication requires re-scraping or using the authors' provided data if available.
• Feature extraction runtime is evaluated separately in Experiment V and is not part of the classification metrics, yet it critically impacts real-world deployment feasibility.
• Experiments use default Weka parameters without hyperparameter tuning, which may not represent optimal performance for these specific feature sets.

Evidence (verbatim from paper)

For performance evaluation, we use two main metrics: 1. Accuracy: represents the ratio of correct predicted samples to the total number of samples. Accuracy metric works well for balanced datasets which is the case of the dataset used in this study. The accuracy of a model is calculated using the following formula: Accuracy = (TP+TN)/(TP+TN+FP+FN) ... 2. Macro F1-score: captures the mean of class-wise F1-scores. Macro F1-score is obtained by averaging F1-scores computed for each class i.

Citation

@misc{hannousse2020phishingbenchmark,
  title={Towards Benchmark Datasets for Machine Learning Based Website Phishing Detection: An experimental study},
  author={Hannousse et al. (2020)},
  year={2020},
  note={arXiv:2010.12847}
}

• arXiv: 2010.12847