Droidspan Eval

droidspan-eval

A Preliminary Study On the Sustainability of Android Malware Detection — Cai (2018) (arXiv:1807.08221, 2018)

What this evaluates

Evaluates the sustainability and robustness of a dynamic behavioral profiling approach (DroidSpan) for Android malware detection over time and against code obfuscation, compared to a static baseline (MamaDroid).

Datasets

• all-data — total ?; splits: train (-1), test (-1)
• oldBen+oldMal — total ?; splits: train (-1)
• MalObf — total 220; splits: test (220)

Metrics

• Precision — range: [0, 1]
  • P = TP / (TP + FP), computed with respect to the positive class L = MALICIOUS.
• Recall — range: [0, 1]
  • R = TP / (TP + FN), computed with respect to the positive class L = MALICIOUS.
• F1-measure (primary) — range: [0, 1]
  • F1 = 2 * (P * R) / (P + R). The paper notes that F1 scores in cross-validation are averaged across folds, not recomputed from averaged P and R.

Input / output format

Input: Runtime dynamic traces of Android apps, specifically the Sensitive Access Distribution (SAD) profile capturing patterns of sensitive data and operation accesses.

Output: Binary classification label: BENIGN or MALICIOUS.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'MALICIOUS' and g == 'MALICIOUS')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'MALICIOUS' and g != 'MALICIOUS')
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p != 'MALICIOUS' and g == 'MALICIOUS')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• F1 scores in cross-validation are averaged across folds, not recomputed from averaged precision and recall.
• MamaDroid's static analysis (FlowDroid) fails on obfuscated apps, making direct comparison on the MalObf dataset impossible.
• Evaluation splits are strictly time-based (training on older malware, testing on newer) to measure sustainability, not random splits.

Evidence (verbatim from paper)

In our evaluation, we gauged the performance of DroidSpan versus the baseline in terms of three metrics: precision, recall, and F1-measure (accuracy), as defined below. ... Precision (P) = TP/(TP + FP), Recall (R) = TP/(TP + FN), and F1 = 2 * (P*R)/(P+R). For evaluating malware detection, these metrics are computed concerning L = MALICIOUS only.

Citation

@misc{cai2018preliminary,
  title={A Preliminary Study On the Sustainability of Android Malware Detection},
  author={Cai (2018)},
  year={2018},
  note={arXiv:1807.08221}
}

• arXiv: 1807.08221