Cosoco Malware Detection Eval

cosoco-malware-detection-eval

Malware Detection in Docker Containers: An Image is Worth a Thousand Logs — Nousias et al. (2025) (arXiv:2504.03238, 2025)

What this evaluates

This benchmark evaluates a model's ability to detect malware within Docker container file systems by treating the entire file system as a large-scale RGB image. It probes the capability to identify subtle, localized malicious byte patterns against a largely benign background, simulating real-world container security threats.

Datasets

• COSOCO — total 3364; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for the positive (malevolent) class: F1 = 2 * (Precision * Recall) / (Precision + Recall).
• Precision — range: [0, 1]
  • Proportion of correctly predicted positive samples out of all samples predicted as positive: TP / (TP + FP).
• Recall — range: [0, 1]
  • Proportion of actual positive samples correctly identified: TP / (TP + FN).
• Accuracy — range: [0, 1]
  • Proportion of all samples classified correctly: (TP + TN) / (TP + TN + FP + FN).

Input / output format

Input: RGB images (1024x4096) generated by converting Docker container tarball file systems into byte representations, processed in 256x256 patches.

Output: Binary classification label: benign or malevolent.

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    return {'F1': f1, 'Precision': precision, 'Recall': recall, 'Accuracy': accuracy}

Common pitfalls

• The dataset is highly imbalanced; models must use class-weighted loss (e.g., weight 256 for malevolent) to avoid bias toward the benign class.
• Overall accuracy is misleading due to class imbalance; evaluation must prioritize Precision, Recall, and F1 on the malevolent class.
• Input is not raw file data but a transformed RGB image; standard file-based evaluation pipelines will fail without the specific image generation step.

Evidence (verbatim from paper)

Classification performance is evaluated with multiple metrics. Accuracy measures the overall correctness of predictions by calculating the proportion of all samples that were classified correctly... While accuracy gives an overall sense of performance, it may be misleading when different errors has different costs. In our case, misclassifying a malevolent container as benign introduces more problems than misclassifying benign containers, as it could breach the entire security layer. For this reason, our study includes the accuracy metric, yet focuses on precision, recall, and f1 score, specifically for the positive (malevolent) class.

Citation

@misc{nousias2025malware,
  title={Malware Detection in Docker Containers: An Image is Worth a Thousand Logs},
  author={Nousias et al. (2025)},
  year={2025},
  note={arXiv:2504.03238}
}

• arXiv: 2504.03238