Wikipedia Vandal Detection Eval

wikipedia-vandal-detection-eval

One-Class Adversarial Nets for Fraud Detection — Panpan Zheng et al. (2018) (arXiv:1803.01798, 2018)

What this evaluates

Evaluates a model's ability to detect malicious Wikipedia editors (vandals) using only benign user data for training. It probes one-class anomaly detection and sequential behavior modeling by measuring how well the system distinguishes benign from malicious users based on edit sequences.

Datasets

• UMDWikipedia — total 22023; splits: train (7000), test (6000); repo https://github.com/PanpanZheng/OCAN

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• Precision — range: [0, 1]
  • Ratio of correctly predicted vandals to all predicted vandals: TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of correctly predicted vandals to all actual vandals: TP / (TP + FN).
• Accuracy — range: [0, 1]
  • Ratio of correct predictions to total predictions: (TP + TN) / Total.

Input / output format

Input: Sequence of user edits (actions on Wikipedia pages) or concatenated raw feature vectors per edit. Features include: meta-page edit flag, consecutive edits <1 min flag, page previously edited flag, edit reverted flag. For sequence models, the LSTM hidden state at each step is used.

Output: Binary classification label (benign or vandal) or probability score from the discriminator. A threshold (e.g., 5-quantile of benign probabilities) is applied to the score to make the final prediction.

Scoring recipe

def compute_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    accuracy = (tp + tn) / len(gold)
    return {'precision': precision, 'recall': recall, 'f1': f1, 'accuracy': accuracy}

Common pitfalls

• Baseline models require a small validation set of vandals (5%) to tune the detection threshold, whereas OCAN requires zero malicious data for training or validation.
• GAN training instability causes higher standard deviations in results, so reporting mean±std over multiple runs (10 in this paper) is critical.
• Early detection evaluates performance at each edit step using LSTM hidden states, not just the final user representation, which changes how metrics are aggregated over time.

Evidence (verbatim from paper)

To evaluate the performance of vandal detection, we randomly select 7000 benign users as the training dataset and 3000 benign users and 3000 vandals as the testing dataset. We report the mean value and standard deviation based on 10 different runs. Table 1 shows the means and standard deviations of the precision, recall, F1 score and accuracy for vandal detection.

Citation

@misc{zheng2018oneclassadversarialnets,
  title={One-Class Adversarial Nets for Fraud Detection},
  author={Panpan Zheng et al. (2018)},
  year={2018},
  note={arXiv:1803.01798}
}

• arXiv: 1803.01798