Drsm Certified Robustness Eval

drsm-certified-robustness-eval

DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness — Saha et al. (2023) (arXiv:2303.13372, 2023)

What this evaluates

Evaluates the standard classification accuracy and certified robustness of a malware detector against adversarial byte perturbations. It measures how well the model maintains correct predictions under a bounded perturbation budget using a de-randomized smoothing defense with window ablation.

Datasets

• PACE — total ?; splits: train (-1), validation (-1), test (-1); repo https://github.com/ShoumikSaha/DRSM

Metrics

• Standard Accuracy (primary) — range: percent
  • Percentage of files correctly classified. A file is correctly classified if the winning class from majority voting across n ablated windows matches the true label. Ties are broken in favor of 'malware'.
• Certified Accuracy — range: percent
  • Percentage of files where the certified robustness condition holds: n_m(x) >= n_b(x) + 2Δ, where n_m and n_b are votes for malware and benign, and Δ is the perturbation budget. Represents a theoretical lower bound on accuracy under adversarial perturbation.

Input / output format

Input: Raw binary executable files, processed into n overlapping ablated windows/sequences for classification.

Output: Binary classification label (benign or malware) derived from majority voting across n ablated windows.

Scoring recipe

def compute_metrics(votes_malware, votes_benign, labels, delta):
    # votes_malware, votes_benign: lists of length n (number of windows)
    # labels: ground truth binary labels
    n_files = len(labels)
    
    # Standard prediction: malware if n_m > n_b, else benign (ties -> malware)
    preds = ['malware' if m >= b else 'benign' for m, b in zip(votes_malware, votes_benign)]
    standard_acc = sum(1 for p, l in zip(preds, labels) if p == l) / n_files * 100
    
    # Certified accuracy: check robustness condition for budget delta
    certified_correct = sum(1 for m, b in zip(votes_malware, votes_benign) if m >= b + 2 * delta)
    certified_acc = certified_correct / n_files * 100
    
    return standard_acc, certified_acc

Common pitfalls

• Tie-breaking differs between evaluations: standard accuracy breaks ties in favor of 'malware', while certified accuracy uses a stricter threshold (n_m >= n_b + 2Δ) to maintain consistency with the robustness bound.
• The perturbation budget Δ starts at 2, not 1, because perturbations smaller than the window size can overlap at most 2 ablated sequences.
• Certified accuracy reports a theoretical lower bound; empirical robustness against actual adversarial attacks is typically higher.

Evidence (verbatim from paper)

Besides standard accuracy, we also evaluate the certified accuracy for DRSM-n models. Recall that – ‘certified accuracy’ is the percentage of files for which the inequality [1] holds true for DRSM-n models. In short, it denotes the lower bound of model performance even when the attacker can perturb bytes in Δ number of ablated windows and alter predictions for all of them.

Citation

@misc{saha2023drsm,
  title={DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness},
  author={Saha et al. (2023)},
  year={2023},
  note={arXiv:2303.13372}
}

• arXiv: 2303.13372