AI Red Team
When to Use
- Red-teaming chatbots, agents, RAG systems, or copilots before launch
- Designing safety evaluation suites and adversarial test harnesses
- Reproducing reported prompt injection or jailbreak vulnerabilities
- Validating mitigations after incidents (retesting filters, hardening)
- Running multi-turn coercion, encoding, or indirect injection campaigns
- Assessing bias, harmful output, or data exfiltration risks in LLM applications
- Scoping rules of engagement and severity rubrics for AI security testing
When NOT to Use
- Writing corporate AI policy or risk governance frameworks →
ai-risk-governance
- Building production LLM features or RAG pipelines →
ai-engineer
- General network/AD/infra penetration testing →
network-pentester
- Authorized web/API OWASP testing (non-LLM) →
web-pentester
- Enterprise adversary simulation, MITRE ATT&CK campaigns, purple team →
red-team-specialist
- Binary, firmware, or protocol reverse engineering →
reverse-engineer
- CI/CD pipeline security →
devsecops
Related skills
| Need |
Skill |
| Production architecture and mitigations |
ai-engineer |
| Governance sign-off and risk tiers |
ai-risk-governance |
| Prompt design baselines |
prompt-engineer |
| CI pipeline security |
devsecops |
| Web/API OWASP pentest (non-LLM) |
web-pentester |
| Network/AD/infra pentest (non-LLM) |
network-pentester |
| Multi-domain pentest (non-LLM) |
penetration-tester |
| Enterprise red team / adversary simulation (non-LLM) |
red-team-specialist |
| Security program and pentest governance |
cybersecurity |
| Deploy/monitor safeguard inference path |
ml-infrastructure-engineer-safeguards |
| Safety benchmarks and classifier training |
ml-research-engineer-safeguards |
| Post-incident disk/memory/log forensics and chain of custody |
digital-forensics-analyst |
| Binary/protocol RE on non-LLM malware or implants |
reverse-engineer |
| Security incident coordination after AI abuse |
incident-responder |
Core Workflows
1. Scope and rules of engagement
- Define target: model, app surface, tools, data stores
- Obtain written authorization and time window
- Agree out-of-scope (e.g., no social engineering of employees unless approved)
- Define success criteria: critical findings, reproduction steps, severity rubric
- Plan safe test environment (no prod customer data)
See references/engagement_scope.md for ROE template and severity definitions.
2. Threat model for LLM applications
| Class |
Examples |
| Prompt injection |
Instructions in user/doc content override system policy |
| Jailbreak |
Role-play, encoding, multi-turn coercion |
| Tool abuse |
Unauthorized API calls, parameter injection |
| Data exfiltration |
RAG leaks other tenants' chunks, PII in logs |
| Supply chain |
Malicious tool definitions, compromised plugins |
| Denial of service |
Token burn, recursive agent loops |
See references/attack_catalog.md for technique families and test prompts (use ethically).
3. Test execution
Phases:
- Baseline — document intended refusals and allowed behaviors
- Automated sweep — harness with curated attack set + fuzz mutations
- Manual creativity — domain-specific abuse scenarios
- Tool/RAG focus — indirect injection via retrieved documents
- Regression — re-run after mitigations
Log: input, output, tool calls, latency, whether guardrail fired.
See references/testing_harness.md for harness design and datasets.
4. Reporting
Each finding includes:
- Title and severity (impact × likelihood)
- Steps to reproduce (minimal)
- Evidence (redacted transcripts)
- Affected component
- Recommended mitigation
- Retest criteria
See references/reporting.md for report template and remediation tracking.
5. Mitigation validation
| Mitigation |
Retest |
| Input/output filters |
Bypass attempts with paraphrases |
| System prompt hardening |
Injection via RAG context |
| Tool allowlists |
Confused deputy and scope creep |
| Human approval gate |
Automated agent bypass paths |
See references/mitigations.md for defense depth and known weak controls.
When to load references
- ROE and scope →
references/engagement_scope.md
- Attack types →
references/attack_catalog.md
- Harness and automation →
references/testing_harness.md
- Reports →
references/reporting.md
- Defenses →
references/mitigations.md
1---2name: ai-redteam3description: Guides adversarial testing of AI systems—prompt injection, jailbreaks, tool abuse, data exfiltration, bias and harmful output probes, multi-turn attacks, and automated red-team harnesses for LLM applications. Use when red-teaming chatbots, agents, RAG systems, or copilots before launch, designing safety eval suites, reproducing reported vulnerabilities, or validating mitigations after incidents—not for writing corporate AI policy (ai-risk-governance), building production features (ai-engineer), or general network or app penetration testing (penetration-tester, network-pentester), enterprise adversary simulation or purple-team campaigns (red-team-specialist), authorized web/API OWASP testing (web-pentester), or binary/firmware RE (reverse-engineer). Production safeguard serving and gateways: ml-infrastructure-engineer-safeguards. Safety classifier R&D and benchmarks: ml-research-engineer-safeguards.4---56# AI Red Team78## When to Use910- Red-teaming chatbots, agents, RAG systems, or copilots before launch11- Designing safety evaluation suites and adversarial test harnesses12- Reproducing reported prompt injection or jailbreak vulnerabilities13- Validating mitigations after incidents (retesting filters, hardening)14- Running multi-turn coercion, encoding, or indirect injection campaigns15- Assessing bias, harmful output, or data exfiltration risks in LLM applications16- Scoping rules of engagement and severity rubrics for AI security testing1718## When NOT to Use1920- Writing corporate AI policy or risk governance frameworks → `ai-risk-governance`21- Building production LLM features or RAG pipelines → `ai-engineer`22- General network/AD/infra penetration testing → `network-pentester`23- Authorized web/API OWASP testing (non-LLM) → `web-pentester`24- Enterprise adversary simulation, MITRE ATT&CK campaigns, purple team → `red-team-specialist`25- Binary, firmware, or protocol reverse engineering → `reverse-engineer`26- CI/CD pipeline security → `devsecops`2728## Related skills2930| Need | Skill |31|---|---|32| Production architecture and mitigations | `ai-engineer` |33| Governance sign-off and risk tiers | `ai-risk-governance` |34| Prompt design baselines | `prompt-engineer` |35| CI pipeline security | `devsecops` |36| Web/API OWASP pentest (non-LLM) | `web-pentester` |37| Network/AD/infra pentest (non-LLM) | `network-pentester` |38| Multi-domain pentest (non-LLM) | `penetration-tester` |39| Enterprise red team / adversary simulation (non-LLM) | `red-team-specialist` |40| Security program and pentest governance | `cybersecurity` |41| Deploy/monitor safeguard inference path | `ml-infrastructure-engineer-safeguards` |42| Safety benchmarks and classifier training | `ml-research-engineer-safeguards` |43| Post-incident disk/memory/log forensics and chain of custody | `digital-forensics-analyst` |44| Binary/protocol RE on non-LLM malware or implants | `reverse-engineer` |45| Security incident coordination after AI abuse | `incident-responder` |4647## Core Workflows4849### 1. Scope and rules of engagement50511. Define target: model, app surface, tools, data stores522. Obtain written authorization and time window533. Agree out-of-scope (e.g., no social engineering of employees unless approved)544. Define success criteria: critical findings, reproduction steps, severity rubric555. Plan safe test environment (no prod customer data)5657**See `references/engagement_scope.md` for ROE template and severity definitions.**5859### 2. Threat model for LLM applications6061| Class | Examples |62|---|---|63| Prompt injection | Instructions in user/doc content override system policy |64| Jailbreak | Role-play, encoding, multi-turn coercion |65| Tool abuse | Unauthorized API calls, parameter injection |66| Data exfiltration | RAG leaks other tenants' chunks, PII in logs |67| Supply chain | Malicious tool definitions, compromised plugins |68| Denial of service | Token burn, recursive agent loops |6970**See `references/attack_catalog.md` for technique families and test prompts (use ethically).**7172### 3. Test execution7374**Phases:**75761. **Baseline** — document intended refusals and allowed behaviors772. **Automated sweep** — harness with curated attack set + fuzz mutations783. **Manual creativity** — domain-specific abuse scenarios794. **Tool/RAG focus** — indirect injection via retrieved documents805. **Regression** — re-run after mitigations8182Log: input, output, tool calls, latency, whether guardrail fired.8384**See `references/testing_harness.md` for harness design and datasets.**8586### 4. Reporting8788Each finding includes:8990- Title and severity (impact × likelihood)91- Steps to reproduce (minimal)92- Evidence (redacted transcripts)93- Affected component94- Recommended mitigation95- Retest criteria9697**See `references/reporting.md` for report template and remediation tracking.**9899### 5. Mitigation validation100101| Mitigation | Retest |102|---|---|103| Input/output filters | Bypass attempts with paraphrases |104| System prompt hardening | Injection via RAG context |105| Tool allowlists | Confused deputy and scope creep |106| Human approval gate | Automated agent bypass paths |107108**See `references/mitigations.md` for defense depth and known weak controls.**109110## When to load references111112- **ROE and scope** → `references/engagement_scope.md`113- **Attack types** → `references/attack_catalog.md`114- **Harness and automation** → `references/testing_harness.md`115- **Reports** → `references/reporting.md`116- **Defenses** → `references/mitigations.md`
daemon-blockint-tech