# AI Red Teaming Ctf Eval

> ai-red-teaming-ctf-eval

- Skill: `qhjqhj00/ai-red-teaming-ctf-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/ai-red-teaming-ctf-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ai-red-teaming-ctf-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/ai-red-teaming-ctf-eval

---


# ai-red-teaming-ctf-eval

> Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers — Bozdemir et al. (2025) (arXiv:2510.16005, 2025)

## What this evaluates

Evaluates adversarial AI red-teaming capabilities by measuring participant success rates in bypassing LLM guardrails, manipulating model outputs, and extracting sensitive data through prompt injection and jailbreaking techniques.

## Datasets

- **AI Red Teaming CTF (CTF ID: 2604)** — total 11; splits: test (11)

## Metrics

- `solve_rate` **(primary)** — range: percent
  - Calculated as the number of teams solving a specific challenge divided by the total number of teams (all registered or active), expressed as a percentage. The paper reports both overall and active-team completion rates.

## Input / output format

**Input**: Adversarial challenge scenarios presented as CTF instances, requiring participants to craft prompts or multi-step attack sequences to bypass AI guardrails, manipulate outputs, or extract hidden data.

**Output**: A submitted flag string upon successfully completing a challenge instance.

## Scoring recipe

```python
def compute_solve_rate(challenges, teams, denominator='active'):
    total = 504 if denominator == 'all' else 217
    rates = {}
    for c in challenges:
        solves = sum(1 for t in teams if t.solved(c))
        rates[c] = (solves / total) * 100
    return rates
```

## Common pitfalls

- Confusing 'active teams' (n=217) with 'all registered teams' (n=504) when interpreting completion percentages, leading to skewed difficulty assessments.
- Measuring time-to-first-solve from CTF start versus first instance launch yields drastically different distributions due to timezone and availability effects rather than pure problem-solving latency.
- Aggregating category success rates (e.g., Output Manipulation vs. Data Extraction) without accounting for the steep difficulty drop-off between early onboarding tasks and the final multi-step challenge.

## Evidence (verbatim from paper)

> Among active teams (n = 217), 170 (78.3%) launched the final challenge (Figure 6); 74 (34.1%) solved it; and 96 (44.2%) launched but did not solve.

## Citation

```bibtex
@misc{bozdemir2025breaking,
  title={Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers},
  author={Bozdemir et al. (2025)},
  year={2025},
  note={arXiv:2510.16005}
}
```

- arXiv: 2510.16005