# Agent Safety

> AI Agent safety engineering -- guardrails, permission models, prompt injection defense, Constitutional Classifiers, and defense-in-depth architecture for both Anthropic and OpenAI ecosystems.

- Skill: `leolin990405/agent-safety` (Agent Skill)
- Install (CLI): `npx get-skill add leolin990405/agent-safety`
- Raw SKILL.md: https://api.skillmd.com/api/skills/leolin990405/agent-safety/raw
- Safety review: pending (external: skill-scanner FAIL, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: LeoLin990405 (https://skillmd.com/u/leolin990405)
- Updated: 2026-11-11
- Page: https://skillmd.com/skills/leolin990405/agent-safety

---


# AI Agent Safety Engineering

> Comprehensive guide to building safe AI agents with defense-in-depth architecture.

## 1. Defense-in-Depth Architecture

```
Layer 1: Input Classification (Constitutional Classifiers / Guardrails)
    ↓
Layer 2: Permission Evaluation (hierarchical rules)
    ↓
Layer 3: Sandbox Enforcement (OS-level isolation)
    ↓
Layer 4: Output Classification (post-generation filters)
    ↓
Layer 5: Human-in-the-Loop (checkpoints, approval prompts)
```

## 2. Constitutional Classifiers (Anthropic)

Two-stage pipeline for real-time content filtering:

**Stage 1 — Synthetic Data Generation:**
1. Define principles specifying allowed vs disallowed content
2. Use Claude to generate synthetic prompts across all content classes
3. Augment via translation + known jailbreak style transformation

**Stage 2 — Dual Classifier Deployment:**
- **Input classifier**: Screens prompts before reaching the model
- **Output classifier**: Screens completions before delivery

| Metric | Value |
|--------|-------|
| Unguarded jailbreak success | 86% |
| With Constitutional Classifiers | **4.4%** |
| Over-refusal increase | 0.38% (not significant) |
| Compute overhead | ~23.7% → optimized to ~1% |

Most effective attack vectors found in red team (339 participants, 3,700 hours):
1. Ciphers/encoding circumventing output classifiers
2. Role-play via system prompts
3. Keyword substitution
4. Prompt injection

## 3. OpenAI Guardrails (8 Built-in)

| Guardrail | Purpose |
|-----------|---------|
| Moderation | Content classification via moderation API |
| URL Filter | Domain allowlists/blocklists |
| Contains PII | Personal data detection |
| Hallucination Detection | Vector store comparison |
| Jailbreak Detection | Manipulation attempt recognition |
| NSFW Text | Inappropriate content flagging |
| Off Topic Prompts | Business scope enforcement |
| Custom Prompt Check | LLM-based custom rules |

### SDK Guardrail Types

```python
# Input guardrail -- runs before agent execution
@input_guardrail
async def safety_check(ctx, agent, input):
    result = await Runner.run(checker_agent, input, context=ctx.context)
    return GuardrailFunctionOutput(
        output_info=result.final_output,
        tripwire_triggered=result.final_output.is_unsafe,
    )

# Tool guardrail -- wraps every tool invocation
@tool_input_guardrail
def block_secrets(data):
    args = json.loads(data.context.tool_arguments or "{}")
    if "sk-" in json.dumps(args):
        return ToolGuardrailFunctionOutput.reject_content("Remove secrets first.")
    return ToolGuardrailFunctionOutput.allow()
```

## 4. Permission Models

### Claude Code: 4-Tier Hierarchy

```
MANAGED (highest) → LOCAL → PROJECT → USER (lowest)
```

Evaluation order: **Deny → Ask → Allow** (first match wins)

```json
{
  "permissions": {
    "allow": ["Bash(npm run lint)", "Read(./src/**)"],
    "ask": ["Bash(git push *)"],
    "deny": ["Bash(curl *)", "Read(./.env)", "WebFetch"]
  }
}
```

Permission modes: `askAlways`, `acceptEdits`, `acceptAllEdits`

### OpenAI: 5-Level Authority

```
PLATFORM (non-overridable) → DEVELOPER → USER → GUIDELINE → NO AUTHORITY
```

Hard-coded platform rules (cannot be overridden):
- Refuse CSAM, CBRN weapons, targeted political manipulation
- Untrusted data (tool outputs, quoted text) has **zero authority**

## 5. Prompt Injection Defense

### Source-Sink Framework

- **Source**: Where untrusted data enters (user input, tool outputs, web content)
- **Sink**: Where it can cause harm (tool calls, code execution, file writes)
- **Defense**: Minimize paths from untrusted sources to dangerous sinks

### Anthropic's Layers
1. Isolated context windows for web fetch
2. Command injection detection (suspicious bash → manual approval)
3. Fail-closed matching (unmatched commands → require approval)
4. Network request approval by default
5. Command blocklist (curl, wget blocked)

### OpenAI's Approach
1. Model-level adversarial training embedded in weights
2. Automated red teaming via RL-trained attacker models
3. Independent URL verification index
4. Always-on classifiers + reasoning monitors

### Many-Shot Jailbreaking Defense
- Attack: 256+ faux dialogues in single prompt exploiting in-context learning
- Mitigation: Prompt classification reduced success from 61% → 2%
- Larger models MORE vulnerable (better in-context learning)

## 6. OS-Level Sandboxing

### Claude Code

| Platform | Technology | Mechanism |
|----------|-----------|-----------|
| Linux | bubblewrap | Linux namespaces |
| macOS | seatbelt | sandbox-exec profiles |
| Cloud | Isolated VMs | Per-session with auto-cleanup |

**Filesystem**: Write restricted to working directory only
**Network**: All traffic through Unix domain socket proxy → domain allowlist

```json
{
  "sandbox": {
    "enabled": true,
    "filesystem": {
      "allowWrite": ["/tmp/build"],
      "denyRead": ["~/.aws/credentials"]
    },
    "network": {
      "allowedDomains": ["github.com", "*.npmjs.org"]
    }
  }
}
```

### OpenAI Codex

| Mode | Description |
|------|-------------|
| `workspace-write` (default) | Read/edit within workspace only |
| `danger-full-access` | No restrictions (use with caution) |

## 7. Safety Research Key Findings

| Research | Finding |
|----------|---------|
| Alignment Faking | 12% strategic compliance when monitored; 97% refusal when unmonitored |
| Sleeper Agents | Backdoors persist through standard safety training; adversarial training counterproductive |
| Sleeper Detection | Linear probes achieve >99% AUROC on residual stream activations |
| Code Sabotage | Claude 3.5 Sonnet defenders blocked ~99% of sabotage attacks |

## 8. Safety Checklist

- [ ] Input/output classifiers deployed
- [ ] Permission rules configured (deny > ask > allow)
- [ ] OS-level sandbox enabled
- [ ] Network restricted to allowed domains
- [ ] Sensitive files in deny list (.env, credentials)
- [ ] Human approval for destructive operations
- [ ] Tool outputs treated as untrusted
- [ ] Command blocklist configured
- [ ] Iteration limits set for agent loops
- [ ] Audit logging enabled

## Sources

- [Constitutional AI Paper](https://arxiv.org/abs/2212.08073)
- [Constitutional Classifiers](https://www.anthropic.com/research/constitutional-classifiers)
- [Claude Code Security](https://code.claude.com/docs/en/security)
- [Claude Code Sandboxing](https://www.anthropic.com/engineering/claude-code-sandboxing)
- [OpenAI Model Spec](https://model-spec.openai.com/2025-02-12.html)
- [OpenAI Guardrails](https://github.com/openai/openai-guardrails-python)
- [Many-Shot Jailbreaking](https://www.anthropic.com/research/many-shot-jailbreaking)
- [Sleeper Agents](https://www.anthropic.com/research/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training)
- [Alignment Faking](https://www.anthropic.com/research/alignment-faking)

