AI Agent Safety Engineering
Comprehensive guide to building safe AI agents with defense-in-depth architecture.
1. Defense-in-Depth Architecture
Layer 1: Input Classification (Constitutional Classifiers / Guardrails)
↓
Layer 2: Permission Evaluation (hierarchical rules)
↓
Layer 3: Sandbox Enforcement (OS-level isolation)
↓
Layer 4: Output Classification (post-generation filters)
↓
Layer 5: Human-in-the-Loop (checkpoints, approval prompts)
2. Constitutional Classifiers (Anthropic)
Two-stage pipeline for real-time content filtering:
Stage 1 — Synthetic Data Generation:
- Define principles specifying allowed vs disallowed content
- Use Claude to generate synthetic prompts across all content classes
- Augment via translation + known jailbreak style transformation
Stage 2 — Dual Classifier Deployment:
- Input classifier: Screens prompts before reaching the model
- Output classifier: Screens completions before delivery
| Metric | Value |
|---|---|
| Unguarded jailbreak success | 86% |
| With Constitutional Classifiers | 4.4% |
| Over-refusal increase | 0.38% (not significant) |
| Compute overhead | ~23.7% → optimized to ~1% |
Most effective attack vectors found in red team (339 participants, 3,700 hours):
- Ciphers/encoding circumventing output classifiers
- Role-play via system prompts
- Keyword substitution
- Prompt injection
3. OpenAI Guardrails (8 Built-in)
| Guardrail | Purpose |
|---|---|
| Moderation | Content classification via moderation API |
| URL Filter | Domain allowlists/blocklists |
| Contains PII | Personal data detection |
| Hallucination Detection | Vector store comparison |
| Jailbreak Detection | Manipulation attempt recognition |
| NSFW Text | Inappropriate content flagging |
| Off Topic Prompts | Business scope enforcement |
| Custom Prompt Check | LLM-based custom rules |
SDK Guardrail Types
# Input guardrail -- runs before agent execution
@input_guardrail
async def safety_check(ctx, agent, input):
result = await Runner.run(checker_agent, input, context=ctx.context)
return GuardrailFunctionOutput(
output_info=result.final_output,
tripwire_triggered=result.final_output.is_unsafe,
)
# Tool guardrail -- wraps every tool invocation
@tool_input_guardrail
def block_secrets(data):
args = json.loads(data.context.tool_arguments or "{}")
if "sk-" in json.dumps(args):
return ToolGuardrailFunctionOutput.reject_content("Remove secrets first.")
return ToolGuardrailFunctionOutput.allow()
4. Permission Models
Claude Code: 4-Tier Hierarchy
MANAGED (highest) → LOCAL → PROJECT → USER (lowest)
Evaluation order: Deny → Ask → Allow (first match wins)
{
"permissions": {
"allow": ["Bash(npm run lint)", "Read(./src/**)"],
"ask": ["Bash(git push *)"],
"deny": ["Bash(curl *)", "Read(./.env)", "WebFetch"]
}
}
Permission modes: askAlways, acceptEdits, acceptAllEdits
OpenAI: 5-Level Authority
PLATFORM (non-overridable) → DEVELOPER → USER → GUIDELINE → NO AUTHORITY
Hard-coded platform rules (cannot be overridden):
- Refuse CSAM, CBRN weapons, targeted political manipulation
- Untrusted data (tool outputs, quoted text) has zero authority
5. Prompt Injection Defense
Source-Sink Framework
- Source: Where untrusted data enters (user input, tool outputs, web content)
- Sink: Where it can cause harm (tool calls, code execution, file writes)
- Defense: Minimize paths from untrusted sources to dangerous sinks
Anthropic's Layers
- Isolated context windows for web fetch
- Command injection detection (suspicious bash → manual approval)
- Fail-closed matching (unmatched commands → require approval)
- Network request approval by default
- Command blocklist (curl, wget blocked)
OpenAI's Approach
- Model-level adversarial training embedded in weights
- Automated red teaming via RL-trained attacker models
- Independent URL verification index
- Always-on classifiers + reasoning monitors
Many-Shot Jailbreaking Defense
- Attack: 256+ faux dialogues in single prompt exploiting in-context learning
- Mitigation: Prompt classification reduced success from 61% → 2%
- Larger models MORE vulnerable (better in-context learning)
6. OS-Level Sandboxing
Claude Code
| Platform | Technology | Mechanism |
|---|---|---|
| Linux | bubblewrap | Linux namespaces |
| macOS | seatbelt | sandbox-exec profiles |
| Cloud | Isolated VMs | Per-session with auto-cleanup |
Filesystem: Write restricted to working directory only Network: All traffic through Unix domain socket proxy → domain allowlist
{
"sandbox": {
"enabled": true,
"filesystem": {
"allowWrite": ["/tmp/build"],
"denyRead": ["~/.aws/credentials"]
},
"network": {
"allowedDomains": ["github.com", "*.npmjs.org"]
}
}
}
OpenAI Codex
| Mode | Description |
|---|---|
workspace-write (default) |
Read/edit within workspace only |
danger-full-access |
No restrictions (use with caution) |
7. Safety Research Key Findings
| Research | Finding |
|---|---|
| Alignment Faking | 12% strategic compliance when monitored; 97% refusal when unmonitored |
| Sleeper Agents | Backdoors persist through standard safety training; adversarial training counterproductive |
| Sleeper Detection | Linear probes achieve >99% AUROC on residual stream activations |
| Code Sabotage | Claude 3.5 Sonnet defenders blocked ~99% of sabotage attacks |
8. Safety Checklist
- Input/output classifiers deployed
- Permission rules configured (deny > ask > allow)
- OS-level sandbox enabled
- Network restricted to allowed domains
- Sensitive files in deny list (.env, credentials)
- Human approval for destructive operations
- Tool outputs treated as untrusted
- Command blocklist configured
- Iteration limits set for agent loops
- Audit logging enabled