Phreshphish Eval

phreshphish-eval

PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark — Dalton et al. (2025) (arXiv:2507.10854, 2025)

What this evaluates

Evaluates phishing website detection models on temporally disjoint, real-world data with realistic base rates, while mitigating training-to-test leakage and varying difficulty levels.

Datasets

• PhreshPhish — total ?; splits: train (-1), test (-1), benchmark (-1)

Metrics

• Precision-Recall (primary) — range: [0, 1]
  • Area under the Precision-Recall curve computed across classification thresholds. Precision is the fraction of predicted phishing sites that are actually phishing, and Recall is the fraction of actual phishing sites correctly identified.
• Accuracy — range: [0, 1]
  • Fraction of correctly classified instances (phishing and benign) out of the total dataset size.

Input / output format

Input: URL string and/or full HTML content of the webpage.

Output: Binary classification label (phishing/benign) or classification probability.

Scoring recipe

def compute_metrics(y_true, y_pred_proba, threshold=0.5):
    y_pred = (y_pred_proba >= threshold).astype(int)
    accuracy = np.mean(y_true == y_pred)
    precision, recall, _ = precision_recall_curve(y_true, y_pred_proba)
    pr_auc = auc(recall, precision)
    return {'accuracy': accuracy, 'PR-AUC': pr_auc}

Common pitfalls

• Using publicly available datasets with unrealistically high phishing base rates (e.g., ~33%), which artificially inflate model performance.
• Failing to apply temporal partitioning or leakage filtering, allowing models to memorize similar train/test points and overestimate generalization.
• Ignoring HTML similarity when pruning datasets, as dissimilar URLs can still host nearly identical phishing pages.

Evidence (verbatim from paper)

Precision-recall curves on the benchmark datasets. Right: Precision-recall curves on benchmark datasets with and without applying the difficulty filter

Citation

@misc{dalton2025phreshphish,
  title={PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark},
  author={Dalton et al. (2025)},
  year={2025},
  note={arXiv:2507.10854}
}

• arXiv: 2507.10854