Secbench Eval

secbench-eval

SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity — Jing et al. (2024) (arXiv:2412.20787, 2024)

What this evaluates

Evaluates large language models' cybersecurity knowledge retention and logical reasoning capabilities across multiple subdomains, languages, and difficulty levels using multiple-choice and short-answer questions.

Datasets

• SecBench — total 47910; splits: test (47910)

Metrics

• correctness percentage (primary) — range: [0, 100] percent
  • For MCQs: (number of correctly answered questions / total MCQs) × 100. For SAQs: average score assigned by the GPT-4o-mini grading agent, normalized to a 0–100 scale based on alignment with ground truth.

Input / output format

Input: Multiple-choice question with four options, or open-ended short-answer question in a cybersecurity domain.

Output: For MCQ: selected option letter or text. For SAQ: generated natural language answer.

Scoring recipe

def evaluate_secbench(predictions, golds, q_type='mcq'):
    if q_type == 'mcq':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return (correct / len(golds)) * 100
    else:
        scores = [grade_with_gpt4o_mini(pred, gold) for pred, gold in zip(predictions, golds)]
        return sum(scores) / len(scores)

Common pitfalls

• SAQ evaluation relies on an LLM-based grading agent (GPT-4o-mini) rather than human annotators, which may introduce scoring bias or inconsistency.
• The dataset spans multiple languages (Chinese/English) and nine cybersecurity subdomains; reporting only the overall average masks significant performance disparities across categories.
• MCQ scoring uses strict exact-match correctness, ignoring partial credit or reasoning quality.

Evidence (verbatim from paper)

Table [1] presents the benchmarking results for the 44,823 MCQs. The values in each cell represent the correctness percentage for the corresponding category. Table [2] presents the benchmarking results for the 3,087 short-answer questions (SAQs). The values in each cell represent the average score, graded by the grading agent, on a percentage scale for the corresponding category.

Citation

@misc{jing2024secbench,
  title={SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity},
  author={Jing et al. (2024)},
  year={2024},
  note={arXiv:2412.20787}
}

• arXiv: 2412.20787