Motif Eval

motif-eval

MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels — Joyce et al. (2021) (arXiv:2111.15031, 2021)

What this evaluates

This benchmark evaluates the accuracy of malware family classification models and antivirus-based labeling tools on a large, expert-verified dataset. It probes a model's ability to correctly assign ground-truth family labels to malware samples, including handling open-set noise and alias resolution.

Datasets

• MOTIF — total 3095; splits: train (-1), test (-1); repo https://github.com/boozallen/MOTIF

Metrics

• accuracy (primary) — range: [0, 1]
  • Correct predictions divided by total predictions. Standard classification accuracy.
• precision — range: [0, 1]
  • True positives divided by all predicted positives. Measures the proportion of predicted family labels that are correct.
• recall — range: [0, 1]
  • True positives divided by all actual positives. Measures the proportion of actual family labels correctly identified.
• f1_measure — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Malware samples provided as PE files, disarmed binaries, or pre-extracted EMBER feature vectors.

Output: A single predicted malware family label (string) per sample.

Scoring recipe

def score(predictions, gold):
    accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    precision = precision_score(gold, predictions, average='macro')
    recall = recall_score(gold, predictions, average='macro')
    f1 = f1_score(gold, predictions, average='macro')
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1_measure': f1}

Common pitfalls

• Using precision and recall as clustering metrics (grouping samples together) rather than strict label matching can artificially inflate scores and hide incorrect family assignments.
• Assuming antivirus majority voting yields high accuracy; the benchmark shows it only reaches ~62% accuracy due to open-set noise and alias mismatches.
• Including singleton families (families with only one sample) in both training and test splits causes data leakage and invalidates cross-validation results.

Evidence (verbatim from paper)

Antivirus majority voting resulted in 1,178 reports with the correct family, 719 reports with an incorrect family, and 1,198 reports where no family was the clear majority. After discarding the scan reports with no majority (as is common practice), antivirus majority voting resulted in only 62.10% accuracy.

Citation

@misc{joyce2021motif,
  title={MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels},
  author={Joyce et al. (2021)},
  year={2021},
  note={arXiv:2111.15031}
}

• arXiv: 2111.15031