Certified Malware Detection Eval

certified-malware-detection-eval

Towards Certified Malware Detection: Provable Guarantees Against Evasion Attacks — Giri et al. (2026) (arXiv:2604.20495, 2026)

What this evaluates

Evaluates the robustness of malware classifiers against metamorphic evasion attacks and synthetic feature-space perturbations. It probes whether randomized smoothing and majority voting can maintain detection accuracy and recall when executables are structurally altered or corrupted.

Datasets

• EMBER — total ?; splits: test (-1)

Metrics

• Recall (primary) — range: percent
  • True positives divided by the sum of true positives and false negatives. Calculated exclusively on malware-only test samples for evasion attacks to measure false negative rates.
• Accuracy — range: percent
  • Ratio of correct predictions (both malware and benign) to total predictions.
• Precision — range: percent
  • True positives divided by the sum of true positives and false positives.
• F1-Score — range: percent
  • Harmonic mean of precision and recall.

Input / output format

Input: Feature vectors extracted via the EMBER pipeline (for tree-based models) or raw byte sequences truncated to 2MB and embedded into continuous vectors (for MalConv).

Output: Binary classification label (malware or benign) determined via majority voting over N=50 stochastic forward passes.

Scoring recipe

def compute_metrics(predictions, labels):
    tp = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 1)
    fp = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 0)
    fn = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 1)
    tn = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 0)
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    accuracy = (tp + tn) / (tp + fp + fn + tn)
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Evasion attacks are evaluated exclusively on malware-only test samples, so precision and false positive rates are not meaningful for the primary robustness claim.
• The Smoothed Classifier requires N=50 stochastic forward passes per sample for majority voting, significantly increasing inference latency compared to the Base Classifier.
• Synthetic noise is applied directly to feature groups or embeddings rather than the raw executable, which may not perfectly mirror real-world metamorphic transformations.

Evidence (verbatim from paper)

Since only malicious instances are considered, recall is the primary metric. Mutated binaries were generated and processed through the EMBER pipeline to extract adversarial feature vectors.

Citation

@misc{giri2026certified,
  title={Towards Certified Malware Detection: Provable Guarantees Against Evasion Attacks},
  author={Giri et al. (2026)},
  year={2026},
  note={arXiv:2604.20495}
}

• arXiv: 2604.20495