Cybercertbench Eval

cybercertbench-eval

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge — Keppler et al. (2026) (arXiv:2604.20389, 2026)

What this evaluates

Evaluates large language models' knowledge of cybersecurity certifications across a spectrum from general IT security to specialized operational technology (OT) and vendor-specific procedural knowledge. It probes whether models can meet professional certification passing standards and identifies gaps in formal, safety-critical industrial protocols.

Datasets

• CyberCertBench — total ?; splits: test (-1); repo https://github.com/GKeppler/CyberCertBench

Metrics

• accuracy (primary) — range: percent
  • Calculated as the number of correctly answered multiple-choice questions divided by the total number of questions. Reported as a percentage or fraction.

Input / output format

Input: Multiple-choice questions (MCQA) derived from official cybersecurity certification exams, covering general IT, OT security, vendor-specific procedures, and formal standards.

Output: Model selects the single correct option from the provided choices.

Scoring recipe

correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return correct / total

Common pitfalls

• General IT security benchmarks are highly saturated, with top models achieving near-perfect accuracy, making them poor differentiators for state-of-the-art systems.
• Models may demonstrate surface-level conceptual understanding of OT security but fail on formal, standard-specific knowledge (e.g., ISA/IEC 62443) and vendor-specific procedures.
• The 'PRO' benchmark filters questions based on performance of three small baseline models, which may not perfectly correlate with human-perceived difficulty or professional certification standards.

Evidence (verbatim from paper)

The accuracy of Qwen2.5 7B falls by 25 percentage points, while the small baseline model, Ministral 3B, falls by 44 points from 62% to just 18%, indicating that it still answers some questions correctly that the other two small models cannot.

Citation

@misc{keppler2026cybercertbench,
  title={CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge},
  author={Keppler et al. (2026)},
  year={2026},
  note={arXiv:2604.20389}
}

• arXiv: 2604.20389