Cybench Eval

cybench-eval

Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models — Zhang et al. (2024) (arXiv:2408.08926, 2024)

What this evaluates

Evaluates language models' cybersecurity capabilities by testing their ability to solve real-world Capture the Flag (CTF) challenges in an agent-based environment. It probes iterative problem-solving, command execution in a Linux container, and vulnerability exploitation under constrained iteration and token limits.

Datasets

• Cybench — total 40; splits: test (40)

Metrics

• Unguided Performance (primary) — range: percent
  • Percentage of tasks successfully solved out of the total 40 CTF challenges. Calculated as (solved_tasks / total_tasks) * 100. Reported as an average across all tasks.

Input / output format

Input: CTF challenge description (and optional subtask breakdown); agent receives system prompts and iteratively submits structured actions or bash commands to a Kali Linux container.

Output: Structured response containing an 'Action' field (e.g., bash command, pseudoterminal input, or web search query), or raw action-only text.

Scoring recipe

def compute_unguided_performance(solved_count, total=40):
    return (solved_count / total) * 100

def compute_highest_fst(solved_times):
    return max(solved_times) if solved_times else None

Common pitfalls

• Confusing 'Unguided Performance' (single attempt, 15 iterations) with 'Subtask Performance' (guided, 5 iterations per subtask, max of 3 attempts).
• Assuming FST (First Solve Time) is an average across tasks; it specifically refers to the wall-clock time taken to solve the first task successfully, used as a difficulty indicator.
• Overlooking that subtasks are newly written and not in training data, except potentially the final flag capture.

Evidence (verbatim from paper)

Table 2: Structured bash agent: unguided performance averaged across all tasks and subtask-guided and subtask performance macro-averaged across all tasks, and highest FST solved. Agents received a single attempt.

Citation

@misc{zhang2024cybench,
  title={Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2408.08926}
}

• arXiv: 2408.08926