Clam Adversarial Eval

clam-adversarial-eval

Exploring the Vulnerabilities of Machine Learning and Quantum Machine Learning to Adversarial Attacks using a Malware Dataset: A Comparative Analysis — Akter et al. (2023) (arXiv:2305.19593, 2023)

What this evaluates

Evaluates the robustness of classical neural networks and quantum neural networks against adversarial attacks by measuring performance degradation on a malware classification task after injecting random noise into input features.

Datasets

• ClaMP_Integrated — total 5210; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Ratio of correctly classified instances to the total number of instances.
• precision — range: [0, 1]
  • Ratio of true positive predictions to the total number of positive predictions.
• recall — range: [0, 1]
  • Ratio of true positive predictions to the total number of actual positives.
• F-score — range: [0, 1]
  • Harmonic mean of precision and recall.

Input / output format

Input: 68-dimensional feature vector extracted from PE headers (raw, expanded, derived features), or 16-dimensional PCA-reduced vector for QNN models. Binary labels mapped to -1 and 1.

Output: Binary classification prediction (malware or benign), typically output as logits or probabilities.

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = np.sum((y_true == 1) & (y_pred == 1))
    tn = np.sum((y_true == -1) & (y_pred == -1))
    fp = np.sum((y_true == -1) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == -1))
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return accuracy, precision, recall, f1

Common pitfalls

• The adversarial perturbation uses np.random.randint without explicit bounds, which may generate large integers depending on the NumPy version, drastically altering feature scales.
• PCA dimensionality reduction to 16 components is strictly required for the QNN due to simulator qubit limits, making direct comparison with the full 68D feature space impossible without this step.
• Labels are mapped to -1 and 1 for hinge loss training, which requires careful thresholding (e.g., >0 for positive) during metric computation.

Evidence (verbatim from paper)

In order to design effective adversarial attacks, we define the experimental settings, where we use accuracy [24], precision [25], recall [26], and F-score metrics to evaluate the robustness of models to attacks.

Citation

@misc{akter2023adversarial,
  title={Exploring the Vulnerabilities of Machine Learning and Quantum Machine Learning to Adversarial Attacks using a Malware Dataset: A Comparative Analysis},
  author={Akter et al. (2023)},
  year={2023},
  note={arXiv:2305.19593}
}

• arXiv: 2305.19593