Elsa Ramd Eval

elsa-ramd-eval

DeepTrust: Multi-Step Classification through Dissimilar Adversarial Representations for Robust Android Malware Detection — Pulido-Cortázar et al. (2025) (arXiv:2510.12310, 2025)

What this evaluates

Evaluates the robustness of Android malware detection models against feature-space and problem-space adversarial attacks, as well as temporal concept drift. It measures detection accuracy under increasing perturbation budgets while enforcing a strict false positive rate constraint on benign applications.

Datasets

• ELSA-RAMD Benchmark — total ?; splits: train (75000), test_track1_2 (6250), test_track3 (100000); repo https://github.com/danielPulidoCortazar/deeptrust

Metrics

• TPR 100-FSA (primary) — range: [0, 1]
  • True Positive Rate (TP / (TP + FN)) evaluated under a perturbation budget of 100 feature-space modifications. Serves as the primary winning metric for Track 1.
• TPR 100-PSA — range: [0, 1]
  • True Positive Rate evaluated under a perturbation budget of 100 problem-space APK modifications. Serves as the primary winning metric for Track 2.
• AUT — range: other
  • Area Under Time curve computed by integrating F1 scores across four temporally distinct test rounds spanning 2020–2022. Serves as the primary winning metric for Track 3.
• TNR — range: [0, 1]
  • True Negative Rate (TN / (TN + FP)), measuring specificity on benign applications. Used as a tie-breaker and prerequisite constraint.

Input / output format

Input: Sparse binary feature vectors (1,461,078 dimensions from DREBIN extraction) for Tracks 1 & 2; raw APK files for Track 2 & 3.

Output: Binary classification probability/logit for malware vs. benign, thresholded to satisfy FPR ≤ 1% constraint on benign test set.

Scoring recipe

def compute_metrics(y_true, y_pred, temporal_f1_scores=None):
    tp = np.sum(y_true & y_pred)
    fn = np.sum(y_true & ~y_pred)
    tn = np.sum(~y_true & ~y_pred)
    fp = np.sum(~y_true & y_pred)
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
    if temporal_f1_scores is not None:
        return np.trapz(temporal_f1_scores)  # AUT for Track 3
    return tpr  # Primary metric for Tracks 1 & 2

Common pitfalls

• Models must first satisfy the strict FPR ≤ 1% constraint on the clean benign test set before robustness metrics (TPR under attack) are evaluated or considered for winning.
• Perturbation budgets are discrete (25, 50, 100 modified features); metrics are evaluated at these exact attack strengths rather than interpolated or averaged.
• Track 3 requires models to accept raw APKs and handle temporal concept drift across four yearly test sets, unlike Tracks 1 & 2 which use fixed DREBIN feature vectors.

Evidence (verbatim from paper)

The performance of models that satisfy this criterion is then measured by the True Positive Rate (TPR) under perturbation budgets of 0, 25, 50, and 100, with the winner determined by the robustness at 100 feature modifications.

Citation

@misc{pulidocortazar2025deeptrust,
  title={DeepTrust: Multi-Step Classification through Dissimilar Adversarial Representations for Robust Android Malware Detection},
  author={Pulido-Cortázar et al. (2025)},
  year={2025},
  note={arXiv:2510.12310}
}

• arXiv: 2510.12310