Big 15 Malware Detection Eval

big-15-malware-detection-eval

Revisiting Concept Drift in Windows Malware Detection: Adaptation to Real Drifted Malware with Minimal Samples — Li et al. (2024) (arXiv:2407.13918, 2024)

What this evaluates

Evaluates a graph-based adversarial domain adaptation model's ability to classify Windows malware and benign binaries under concept drift using minimal labeled samples. It probes the model's robustness to real-world malware evolution by comparing performance against baselines on the Big-15 dataset.

Datasets

• Big-15 — total ?; splits: train (-1), test (-1); repo https://github.com/gloryer/malware-detection-concept-drift

Metrics

• performance metrics (primary) — range: percent
  • Classification performance metrics (e.g., accuracy, F1-score) computed on the Big-15 dataset after training the domain adaptation model. The exact formula is not specified in this section, but results are reported in Figure 12 and Table II.

Input / output format

Input: Assembly files (malware/benign binaries) processed into Control Flow Graphs (CFGs), with node-level embeddings generated via a pre-trained PalmTree model.

Output: Cluster labels (via weighted consensus clustering) and classification predictions for malware vs. benign binaries.

Scoring recipe

# 1. Extract CFGs from assembly files
cfgs = extract_cfgs(assembly_files)
# 2. Generate node embeddings using PalmTree
embeddings = generate_embeddings(cfgs, model="PalmTree")
# 3. Perform graph-based clustering to generate cluster labels
cluster_labels = consensus_clustering(embeddings)
# 4. Train domain adaptation model on Big-15 dataset
model = train_domain_adaptation(cfgs, cluster_labels, epochs=low_value_for_demo)
# 5. Evaluate on test set and compute performance metrics
predictions = model.predict(test_cfgs)
metric_value = compute_performance_metrics(predictions, gold_labels)

Common pitfalls

• The provided notebooks use a low number of epochs by default to minimize runtime for artifact testing, which may not reproduce final paper results without adjustment.
• Experiments E1-E4 demonstrate functionality using only five example files, but the code is designed to scale to the full dataset.
• Original labels in Big-15 are noted as poorly separated; the evaluation relies on newly generated cluster labels from the graph-based clustering algorithm.

Evidence (verbatim from paper)

Experiment (E5): [Domain Adaptation] [2 human-hours + 4 compute-hours]: The following experiment is to generate our models' performance metrics on the Big-15 dataset. Follow these instructions to train our model:

• Run the following notebooks in any order
  • Train_origin_label_graph.ipynb
  • Train_cluster_label_graph.ipynb

Train_cluster_label_image.ipynb

Train_cluster_label_content.ipynb

• The first two notebooks will generate the metrics for our approach, as shown in Figure 12. The third and fourth notebooks will generate the metrics for our approach using image representations and content-based representations, respectively, corresponding to Table II.

Citation

@misc{li2024revisitingconceptdrift,
  title={Revisiting Concept Drift in Windows Malware Detection: Adaptation to Real Drifted Malware with Minimal Samples},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2407.13918}
}

• arXiv: 2407.13918