Android Malware Classification Eval

android-malware-classification-eval

Graph Neural Network-based Android Malware Classification with Jumping Knowledge — Lo et al. (2022) (arXiv:2201.07537, 2022)

What this evaluates

Evaluates the ability of Graph Neural Networks to classify Android applications as benign or malicious, and to identify specific malware families or categories, by learning topological patterns from function call graphs.

Datasets

• Malnet-Tiny — total 5000; splits: train (3500), val (500), test (1000)
• Drebin — total 18246; splits: train (12772), test (5474)

Metrics

• Accuracy (primary) — range: [0, 1]
  • The proportion of correctly classified samples out of the total. Formula: (TP + TN) / (TP + FP + TN + FN).
• Precision — range: [0, 1]
  • The proportion of true positive predictions among all positive predictions. Formula: TP / (TP + FP).
• Recall — range: [0, 1]
  • The proportion of true positive predictions among all actual positives. Also called Detection Rate. Formula: TP / (TP + FN).
• F1-Score — range: [0, 1]
  • The harmonic mean of Precision and Recall. Formula: 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Function Call Graphs (FCGs) extracted from Android APKs, representing inter-procedural call relationships between functions.

Output: Discrete classification label: binary (malicious/benign) or multiclass (specific malware family/category).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * (recall * precision) / (recall + precision) if (recall + precision) > 0 else 0
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Drebin originally contains only malware labels; benign samples must be sourced externally (e.g., AndroZoo) to form a binary classification task.
• Malnet-Tiny uses a fixed 70/10/20 split, while Drebin uses a random 70/30 split; mixing these up will cause non-reproducible results.
• Drebin multiclass evaluation is restricted to the top 24 malware families by sample count, not all 179 families.

Evidence (verbatim from paper)

For evaluating the performance of the different GNN models, the standard metrics listed in Table I are used, where TP, TN, FP and FN represent the number of True Positives, True Negatives, False Positives and False Negatives, respectively. Malnet-Tiny is an Android malware FCG dataset that was created by Scott et al. from Georgia Tech University and the Microsoft APT team. The dataset consists of 4,500 malicious FCGs, belonging to four different malware categories, and 500 benign FCGs. We follow the approach for training, validation, and testing split (70/10/20), as specified by the authors. Drebin is an Android malware dataset that includes 5,560 APKs from 179 different malware families. Since Drebin only consists of malware samples, we also download 12,686 benign APK files from AndroZoo, a large-scale APK repository, with samples collected from 2013 to 2019 for the malware detection experiments. For these experiments, we randomly split the combined and Drebin datasets into 70% for training and 30% for testing.

Citation

@misc{lo2022gnnandroidmalwareclassification,
  title={Graph Neural Network-based Android Malware Classification with Jumping Knowledge},
  author={Lo et al. (2022)},
  year={2022},
  note={arXiv:2201.07537}
}

• arXiv: 2201.07537