Big2015 Eval

big2015-eval

Random CapsNet Forest Model for Imbalanced Malware Type Classification Task — Çayır et al. (2019) (arXiv:1912.10836, 2019)

What this evaluates

Evaluates the ability of deep learning models to classify malware binaries into their respective family types using image-based representations, specifically probing performance on imbalanced class distributions.

Datasets

• BIG2015 — total 10868; splits: train (8151), val (1359), test (1358)

Metrics

• accuracy — range: [0, 1]
  • Ratio of correct predictions to total instances: (TP+TN)/(TP+TN+FP+FN).
• F-Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: (2×TP)/(2×TP+FN+FP). The paper provides the binary formula but applies it to multi-class malware families; averaging strategy is not explicitly specified.

Input / output format

Input: Image representation of malware binaries.

Output: Predicted malware family class label.

Scoring recipe

def compute_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == g)
    accuracy = tp / len(gold)
    # F-Score formula given is binary; multi-class averaging unspecified
    # Standard implementation for multi-class:
    f1 = f1_score(gold, preds, average='macro')
    return accuracy, f1

Common pitfalls

• The F-Score formula provided is strictly for binary classification, but the task is multi-class; the paper does not specify whether macro, micro, or weighted averaging is used.
• Stratified splits are mentioned, but random seeds and exact shuffling procedures are not reported, making exact replication of train/val/test partitions difficult.

Evidence (verbatim from paper)

Model evaluation has been done in terms of accuracy, F-Score, and the number of parameters of deep neural nets. These performance metrics are defined as follows: accuracy=(TP+TN)/(TP+TN+FP+FN) ... F-Score=(2×TP)/(2×TP+FN+FP) where true positive (TP) and false positive (FP) are the numbers of instances correctly and wrongly classified as positive respectively.

Citation

@misc{cayir2019randomcapsnet,
  title={Random CapsNet Forest Model for Imbalanced Malware Type Classification Task},
  author={Çayır et al. (2019)},
  year={2019},
  note={arXiv:1912.10836}
}

• arXiv: 1912.10836