Tesseract Eval

tesseract-eval

TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time — Pendlebury et al. (2018) (arXiv:1807.07838, 2018)

What this evaluates

Evaluates the robustness of Android malware classifiers against spatio-temporal experimental bias. It measures how model performance degrades when trained on past application data and tested on future data, while accounting for realistic malware-to-goodware class distributions.

Datasets

• Android malware dataset (2014-2016) — total ?; splits: train (-1), test (-1)

Metrics

• F1-Score (primary) — range: [0, 1]
  • The harmonic mean of Precision and Recall. Calculated as 2 * (Precision * Recall) / (Precision + Recall). Precision is TP/(TP+FP) and Recall is TP/(TP+FN).

Input / output format

Input: Android application samples (features or binaries) labeled as either malware or goodware.

Output: Binary classification label (malware or goodware).

Scoring recipe

def compute_f1(tp, fp, fn):
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    if (precision + recall) == 0:
        return 0
    return 2 * (precision * recall) / (precision + recall)

Common pitfalls

• Using k-fold cross-validation on malware datasets artificially inflates performance because it often places samples from the same malware family in both train and test splits, ignoring concept drift and time decay.
• Manipulating the malware-to-goodware ratio in training or testing sets (e.g., using 90% malware) artificially boosts Precision and F1-Score, failing to reflect realistic in-the-wild deployment distributions (~10% malware).

Evidence (verbatim from paper)

Since the F1-Score is the harmonic mean of Precision and Recall, it goes up with Precision. We also observe that, inversely, the Precision for the goodware (gw) class—the negative class— decreases... This example shows how considering an unrealistic testing distribution with more malware than goodware in this context (§2.2) positively inflates Precision and hence the F1-Score of the malware classifier.

Citation

@misc{pendlebury2018tesseract,
  title={TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time},
  author={Pendlebury et al. (2018)},
  year={2018},
  note={arXiv:1807.07838}
}

• arXiv: 1807.07838