qhjqhj00 ai-red-teaming-ctf-eval
Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers — Bozdemir et al. (2025) (arXiv:2510.16005, 2025)
What this evaluates
Evaluates adversarial AI red-teaming capabilities by measuring participant success rates in bypassing LLM guardrails, manipulating model outputs, and extracting sensitive data through prompt injection and jailbreaking techniques.
Datasets
- AI Red Teaming CTF (CTF ID: 2604) — total 11; splits: test (11)
Metrics
solve_rate(primary) — range: percent- Calculated as the number of teams solving a specific challenge divided by the total number of teams (all registered or active), expressed as a percentage. The paper reports both overall and active-team completion rates.
Input / output format
Input: Adversarial challenge scenarios presented as CTF instances, requiring participants to craft prompts or multi-step attack sequences to bypass AI guardrails, manipulate outputs, or extract hidden data.
Output: A submitted flag string upon successfully completing a challenge instance.
Scoring recipe
def compute_solve_rate(challenges, teams, denominator='active'):
total = 504 if denominator == 'all' else 217
rates = {}
for c in challenges:
solves = sum(1 for t in teams if t.solved(c))
rates[c] = (solves / total) * 100
return rates
Common pitfalls
- Confusing 'active teams' (n=217) with 'all registered teams' (n=504) when interpreting completion percentages, leading to skewed difficulty assessments.
- Measuring time-to-first-solve from CTF start versus first instance launch yields drastically different distributions due to timezone and availability effects rather than pure problem-solving latency.
- Aggregating category success rates (e.g., Output Manipulation vs. Data Extraction) without accounting for the steep difficulty drop-off between early onboarding tasks and the final multi-step challenge.
Evidence (verbatim from paper)
Among active teams (n = 217), 170 (78.3%) launched the final challenge (Figure 6); 74 (34.1%) solved it; and 96 (44.2%) launched but did not solve.
Citation
@misc{bozdemir2025breaking,
title={Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers},
author={Bozdemir et al. (2025)},
year={2025},
note={arXiv:2510.16005}
}
- arXiv: 2510.16005