Foca Malware Classification Eval

foca-malware-classification-eval

FOCA: Multimodal Malware Classification via Hyperbolic Cross-Attention — Choudhury et al. (2026) (arXiv:2601.17638, 2026)

What this evaluates

This evaluation probes a model's ability to classify Android malware by fusing audio and visual representations derived from raw APK binaries. It measures supervised classification performance across multiple malware families and benign samples using standard accuracy and macro-F1 metrics.

Datasets

• CICMalDroid-2020 — total 17341; splits: 5-fold cross-validation (-1)
• Mal-Net — total 8000; splits: 5-fold cross-validation (-1)

Metrics

• Accuracy (primary) — range: percent
  • Number of correctly predicted class labels divided by the total number of samples, reported as a percentage.
• macro-F1 — range: percent
  • Unweighted mean of the F1-score computed independently for each class, reported as a percentage.

Input / output format

Input: Multimodal feature representations of APK files: audio features (generated via binary-to-audio transformation) and image features (generated via binary-to-image transformation).

Output: A single categorical class label indicating the malware family (e.g., Adware, Banking malware, SMS malware, Riskware) or benign category.

Scoring recipe

def compute_metrics(predictions, labels, num_classes):
    accuracy = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
    f1_scores = []
    for c in range(num_classes):
        tp = sum(1 for p, l in zip(predictions, labels) if p == c and l == c)
        fp = sum(1 for p, l in zip(predictions, labels) if p == c and l != c)
        fn = sum(1 for p, l in zip(predictions, labels) if p != c and l == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / num_classes
    return accuracy * 100, macro_f1 * 100

Common pitfalls

• Using the full Mal-Net dataset (>1.2M images) instead of the paper's explicitly filtered 8,000-sample subset (800 per class).
• Reporting results from a single train/test split instead of averaging over the specified 5-fold cross-validation strategy.
• Confusing the fusion operators: $+$ (simple concatenation), $\otimes$ (Euclidean cross-attention), and $\boxplus$ (hyperbolic cross-attention/FOCA).

Evidence (verbatim from paper)

Accuracy and macro-F1 are reported in %; Here, $+$ and $\otimes$ represent concatenation and cross-modal attention in Euclidean space, and $\boxplus$ defines hyperbolic cross-attention fusion via FOCA, respectively; The scores are average of five folds;

Citation

@misc{choudhury2026foca,
  title={FOCA: Multimodal Malware Classification via Hyperbolic Cross-Attention},
  author={Choudhury et al. (2026)},
  year={2026},
  note={arXiv:2601.17638}
}

• arXiv: 2601.17638