Malware Analysis Eval

malware-analysis-eval

CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning — Deason et al. (2025) (arXiv:2509.20166, 2025)

What this evaluates

This benchmark evaluates an AI system's ability to analyze low-level process execution logs and identify malicious signals from malware detonations. It probes structured data parsing, security event correlation, and malware family classification capabilities.

Datasets

• CyberSOCEval Malware Analysis — total 609; splits: test (609); repo https://github.com/CrowdStrike/cybersoceval_data

Metrics

• accuracy (primary) — range: [0, 1]
  • The share of questions for which the system selects all correct options and only the correct options. It is a strict exact-match metric for multi-label multiple-choice questions.

Input / output format

Input: JSON-formatted system log data from malware detonations, including process execution details, extracted files, and static signature detections.

Output: A set of selected multiple-choice options (up to 10 possible answers per question).

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if set(pred) == set(gold):
            correct += 1
    return correct / len(golds)

Common pitfalls

• Partial credit is not awarded; models must select exactly the correct set of options to receive a point for a question.
• The input consists of structured JSON logs rather than raw binary files or plain text, requiring the model to parse and interpret system event data.

Evidence (verbatim from paper)

Evaluation is based on accuracy: the share of questions for which the system selects all correct options and only the correct options.

Citation

@misc{deason2025cybersoceval,
  title={CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning},
  author={Deason et al. (2025)},
  year={2025},
  note={arXiv:2509.20166}
}

• arXiv: 2509.20166