# Ember2024 Eval

> Evaluates malware classifiers on detection, family identification, and attribute prediction tasks across multiple file formats. It specifically probes robustness against concept drift and novel malware families using a temporally separated test set and a challenge set of evasive samples. Use when the user wants to benchmark on EMBER2024, or asks about evaluating this task. Reports ROC AUC.

- Skill: `qhjqhj00/ember2024-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/ember2024-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ember2024-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/ember2024-eval

---


# ember2024-eval

> EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers — Joyce et al. (2025) (arXiv:2506.05074, 2025)

## What this evaluates

Evaluates malware classifiers on detection, family identification, and attribute prediction tasks across multiple file formats. It specifically probes robustness against concept drift and novel malware families using a temporally separated test set and a challenge set of evasive samples.

## Datasets

- **EMBER2024** — total 3200000; splits: train (-1), test (-1), challenge (-1); repo https://github.com/FutureComputing4AI/EMBER2024

## Metrics

- `ROC AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `PR AUC` — range: [0, 1]
  - Area under the Precision-Recall curve, measuring the trade-off between precision and recall across thresholds. Preferred over ROC AUC for highly imbalanced datasets.

## Input / output format

**Input**: Malware or benign files in PE (Win32/Win64/.NET), APK, ELF, or PDF formats, represented using EMBER feature version 3.

**Output**: Binary label (malicious/benign) or multi-class/multi-label predictions for family/attributes.

## Scoring recipe

```python
def compute_roc_auc(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_pr_auc(y_true, y_scores):
    precision, recall, _ = precision_recall_curve(y_true, y_scores)
    return auc(recall, precision)
```

## Common pitfalls

- ROC AUC can be misleading on highly imbalanced datasets (e.g., the challenge set); PR AUC should be reported instead.
- The challenge set contains only malware, so it must be joined with the corresponding benign test partition to compute standard binary classification metrics.
- Temporal train/test splits introduce concept drift, making detection of novel families significantly harder than random splits.

## Evidence (verbatim from paper)

> The ROC AUC and Precision-Recall (PR) AUC scores in Table 5 indicate that the trained LightGBM classifiers are able to accurately detect malicious files in the EMBER2024 test set, which consists of files that appeared in VirusTotal for the first time 1-12 weeks after the most recent file in the training set.

## Citation

```bibtex
@misc{joyce2025ember2024,
  title={EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers},
  author={Joyce et al. (2025)},
  year={2025},
  note={arXiv:2506.05074}
}
```

- arXiv: 2506.05074