Malnet Tiny Eval

malnet-tiny-eval

Mitigating Distribution Shift in Graph-Based Android Malware Classification via Function Metadata and LLM Embeddings — Ngoc N. Tran et al. (arXiv:2508.06734, 2025)

What this evaluates

Evaluates graph neural networks for Android malware family classification under intra-family and cross-family distribution shifts. It probes how semantic feature enrichment (function metadata and LLM embeddings) and test-time/domain adaptation methods mitigate performance degradation when models encounter unseen malware families.

Datasets

• MalNet-Tiny — total ?; splits: test (-1)
• MalNet-Tiny-Common — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly predicted malware family labels out of the total number of test samples. Reported as mean accuracy over 3 independent runs with standard deviation.

Input / output format

Input: Function call graphs (FCGs) representing Android applications, optionally augmented with feature collations such as function metadata, LLM-derived code embeddings, and LDP features.

Output: A single predicted malware family label per application graph.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Performance is highly sensitive to the feature collation strategy (Trim, Zero, Prune) used to handle missing semantic features in the graphs.
• Results must be averaged over 3 independent runs and reported with standard deviation; single-run evaluations are not comparable to the paper's benchmarks.
• Cross-family (distribution shift) accuracy drops significantly compared to intra-family baselines, so evaluating only on the common split masks the model's generalization capability.

Evidence (verbatim from paper)

Tab. 2: Models' accuracy on MalNet-Tiny and MalNet-Tiny-Common across different feature configurations. Top 5 highest values are highlighted in green, darker green represents higher accuracy; values lower than baseline are highlighted in red. Subscript denotes standard deviation over 3 independent runs.

Citation

@misc{tran2025mitigating,
  title={Mitigating Distribution Shift in Graph-Based Android Malware Classification via Function Metadata and LLM Embeddings},
  author={Ngoc N. Tran et al.},
  year={2025},
  note={arXiv:2508.06734}
}

• arXiv: 2508.06734