Threat Intelligence Eval

threat-intelligence-eval

CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning — Deason et al. (2025) (arXiv:2509.20166, 2025)

What this evaluates

This benchmark evaluates an AI system's ability to extract actionable insights from threat intelligence reports and perform security reasoning. It probes multi-document comprehension, attack chain reconstruction, and MITRE ATT&CK framework mapping capabilities.

Datasets

• CyberSOCEval Threat Intelligence Reasoning — total 588; splits: test (588); repo https://github.com/CrowdStrike/cybersoceval_data

Metrics

• accuracy (primary) — range: [0, 1]
  • The share of questions for which the system selects all correct options and only the correct options. It is a strict exact-match metric for multi-label multiple-choice questions.

Input / output format

Input: Multiple images representing a threat intelligence report, with one image provided per report page.

Output: A set of selected multiple-choice options (up to 6 possible answers per question).

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if set(pred) == set(gold):
            correct += 1
    return correct / len(golds)

Common pitfalls

• Partial credit is not awarded; models must select exactly the correct set of options to receive a point for a question.
• The input is provided as images of report pages rather than extractable text, requiring vision-language capabilities instead of pure text processing.

Evidence (verbatim from paper)

Evaluation is based on accuracy: the share of questions for which the system selects all correct options and only the correct options.

Citation

@misc{deason2025cybersoceval,
  title={CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning},
  author={Deason et al. (2025)},
  year={2025},
  note={arXiv:2509.20166}
}

• arXiv: 2509.20166