PDF Malware Poisoning Eval

pdf-malware-poisoning-eval

Is feature selection secure against training data poisoning? — Huang Xiao et al. (2018) (arXiv:1804.07933, 2018)

What this evaluates

Evaluates the robustness of embedded feature selection methods (LASSO, Ridge, Elastic Net) against training data poisoning attacks in a PDF malware detection setting. It measures how injected malicious samples manipulate feature selection stability and degrade classification performance.

Datasets

• Contagio + Web Benign PDFs — total 11944; splits: train (300), test (5000)

Metrics

• classification error (primary) — range: percent
  • Fraction of misclassified samples in the test set.
• Kuncheva's Stability Index — range: other
  • I_C(A,B) = (rd - k^2) / (k(d-k)), where r is the intersection size of two feature subsets A and B of size k, and d is the total feature dimension. Ranges from -1 to +1.
• number of selected features — range: other
  • Count of features with the highest absolute weight values selected by the model.

Input / output format

Input: PDF files represented as a 114-dimensional vector of keyword occurrence counts, normalized to [0, 1] by dividing by 20.

Output: Binary label (malicious/benign) and a ranked list of selected features based on absolute weight values.

Scoring recipe

# Classification Error
preds = model.predict(X_test)
error = np.mean(preds != y_test)

# Kuncheva's Stability Index
# A: features selected without poisoning, B: features selected with poisoning
# k: number of top features to compare (e.g., 30 or 50), d: total features (114)
r = len(set(A) & set(B))
stability = (r * d - k**2) / (k * (d - k))
# Average over all pairwise combinations of runs

Common pitfalls

• Poisoning percentage is relative to the 300-sample training set, so 20% injection equals only 60 samples.
• Stability index requires averaging over all pairwise combinations of feature subsets across the 5 independent runs, not just a single comparison.
• Feature normalization caps keyword counts at 20 to limit attacker manipulation capability, which significantly affects attack success rates.

Evidence (verbatim from paper)

We evaluate our results by reporting the classification error as a function of the percentage of injected poisoning samples, which was increased from 0% to 20% (where 20% corresponds to adding 75 poisoning samples to the initial data). Furthermore, to understand how feature selection and ranking are affected by the attack, we also evaluate the consistency index originally defined by Kuncheva (2007) to evaluate the stability of feature selection under random perturbations of the training data.

Citation

@misc{xiao2018feature,
  title={Is feature selection secure against training data poisoning?},
  author={Huang Xiao et al. (2018)},
  year={2018},
  note={arXiv:1804.07933}
}

• arXiv: 1804.07933