Agent Red Teaming Eval

agent-red-teaming-eval

Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition — Zou et al. (2025) (arXiv:2507.20526, 2025)

What this evaluates

Evaluates the security and robustness of frontier AI agents against adversarial prompt injections and policy violations across multiple models and realistic deployment scenarios. It measures how effectively attacks transfer between models and whether model capability or inference compute correlates with safety.

Datasets

• Agent Red Teaming (ART) benchmark — total ?; splits: test (-1)

Metrics

• ASR (primary) — range: percent
  • Calculated as the proportion of adversarial prompts that successfully elicit a policy violation or complete the restricted task out of the total number of attacks tested on a given model or behavior.

Input / output format

Input: Adversarial single-turn prompts or attack templates targeting specific behavioral scenarios (e.g., illegal purchases, system prompt overrides, faux reasoning injections) provided to the AI agent.

Output: The model's generated response or executed action, evaluated for whether it violates the stated deployment policy or completes the restricted task.

Scoring recipe

def calculate_asr(successful_violations, total_attacks):
    return (successful_violations / total_attacks) * 100

# For each model/behavior pair, count prompts that elicit policy violations,
divide by total prompts tested for that pair, and express as a percentage.

Common pitfalls

• Assuming that higher model capability or larger parameter counts directly correlate with improved adversarial robustness.
• Assuming that increasing inference compute or enabling extended reasoning capabilities inherently mitigates prompt injection vulnerabilities.

Evidence (verbatim from paper)

Figure 5 presents a heatmap illustrating transfer attack success rates; diagonal elements understandably show the highest ASR values, reflecting the effectiveness of attacks applied directly to the models they were designed for.

Citation

@misc{zou2025securitychallenges,
  title={Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition},
  author={Zou et al. (2025)},
  year={2025},
  note={arXiv:2507.20526}
}

• arXiv: 2507.20526