Open Set Malware Eval

open-set-malware-eval

CNS-Net: Conservative Novelty Synthesizing Network for Malware Recognition in an Open-set Scenario — Jingcai Guo et al. (2023) (arXiv:2305.01236, 2023)

What this evaluates

Evaluates a model's ability to classify malware into known families while simultaneously detecting instances belonging to novel, unseen families in an open-set scenario.

Datasets

• BIG 2015 — total 10868; splits: train (6900), test (3968)
• Mailing — total 9339; splits: train (6400), test (2939)
• MAL-100 — total 56481; splits: train (31523), test (24958)

Metrics

• classification accuracy ($C_{Acc}$) (primary) — range: percent
  • Calculated as $N_{correct} / N_{instance}$, where $N_{correct}$ is the number of correctly classified known malware instances and $N_{instance}$ is the total number of testing known instances.
• detection accuracy ($D_{Acc}$) — range: percent
  • Calculated as $(TPR_{(K)} + TNR_{(U)}) / 2$, where $TPR_{(K)}$ is the true positive rate for known families and $TNR_{(U)}$ is the true negative rate for unknown families.

Input / output format

Input: Malware instances represented as 25×25 grayscale images (or 1D vectors for some baselines) derived from 8 groups of raw characteristics including PE header info, COFF header, imported/exported functions, section info, byte histogram, byte-entropy histogram, and printable-string info.

Output: Predicted malware family label for known instances, and a binary known/unknown decision for detection.

Scoring recipe

def compute_C_Acc(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    return correct / len(golds)

def compute_D_Acc(preds, golds, is_unknown):
    tp_k = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p == g and not u)
    fn_k = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p != g and not u)
    tpr_k = tp_k / (tp_k + fn_k) if (tp_k + fn_k) > 0 else 0
    tn_u = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p == 'unknown' and u)
    fp_u = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p != 'unknown' and u)
    tnr_u = tn_u / (tn_u + fp_u) if (tn_u + fp_u) > 0 else 0
    return (tpr_k + tnr_u) / 2

Common pitfalls

• Data splitting is not standardized across baselines; some competitors use different train/test ratios or only report results on known families.
• The $D_{Acc}$ metric averages TPR for knowns and TNR for unknowns, which can mask severe imbalance in detection performance between known and unknown classes.
• Resizing malware feature vectors to fixed 25×25 or 32×32 images may discard fine-grained structural details present in the original 622-dimensional feature space.

Evidence (verbatim from paper)

The performance of open-set malware recognition can be evaluated by two parallel tasks including the multi-families classification and the unknown detection. The performance is evaluated by the classification accuracy $C_{Acc}$ defined as: $C_{Acc} = N_{correct} / N_{instance}$... The detection accuracy $D_{Acc}$ is then calculated as: $D_{Acc} = (TPR_{(K)} + TNR_{(U)}) / 2$.

Citation

@misc{guo2023cnsnet,
  title={CNS-Net: Conservative Novelty Synthesizing Network for Malware Recognition in an Open-set Scenario},
  author={Jingcai Guo et al. (2023)},
  year={2023},
  note={arXiv:2305.01236}
}

• arXiv: 2305.01236