# Promptfoo Redteam Configuration

> Skill: promptfoo-redteam-configuration

- Skill: `soludevtech/promptfoo-redteam-configuration` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds@latest add soludevtech/promptfoo-redteam-configuration`
- Raw SKILL.md: https://api.skillmd.com/api/skills/soludevtech/promptfoo-redteam-configuration/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: SoluDevTech (https://skillmd.com/u/soludevtech)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/soludevtech/promptfoo-redteam-configuration

---

# Skill: promptfoo-redteam-configuration

# Configure and Run promptfoo Red Team Scans (Fully Local, No Cloud)

Write, validate, and run `promptfooconfig.yaml` files for red team scans against LLM applications. Covers the full configuration schema, local execution without email verification, provider setup, plugin/strategy selection, HTTP/WebSocket targets, multipart uploads, cookie-based auth, and the traps that block scans in practice.

## Use this skill when
- Writing or reviewing a `promptfooconfig.yaml` for red team scanning
- Running `promptfoo redteam run` / `generate` / `eval` locally
- Configuring OpenRouter (or other non-OpenAI) as both generation and grading provider
- Hitting HTTP API targets with cookie-based auth (oauth2-proxy, Traefik)
- Uploading files (PDF/DOCX) via multipart for indirect prompt injection testing
- Debugging "Email Verification Required", "plugin requires remote generation", 401 on targets, or 0 tests generated
- Isolating redteam npm scripts from normal test suites

## Do not use this skill when
- Selecting which plugins/strategies map to which vulnerability → use `promptfoo-redteam-llm`, `promptfoo-redteam-agents`, `promptfoo-redteam-multi-input`, or `promptfoo-redteam-rag`
- Testing against a specific compliance framework → use `promptfoo-framework-*` skills
- Configuring dynamic/static/multi-turn/indirect-injection strategies → use `promptfoo-strategies-*` skills

## The #1 trap: "Email Verification Required" — bypass it with CI=true

**Symptom**: `promptfoo redteam run` or `redteam generate` prompts for a work email and blocks.

**Root cause**: promptfoo's red team generation uses cloud-hosted uncensored models by default. The email check gates access to that cloud. It triggers when `!neverGenerateRemote()` — i.e., when `PROMPTFOO_DISABLE_REMOTE_GENERATION` is NOT set.

**Wrong fix**: `PROMPTFOO_DISABLE_REMOTE_GENERATION=1` — this skips the email check BUT hard-blocks all remote-only plugins (hijacking, bola, bfla, agentic:memory-poisoning, harmful:privacy, indirect-prompt-injection) and remote-only strategies (jailbreak, goat, jailbreak:composite, jailbreak:meta). You lose 60%+ of coverage.

**Correct fix**: Set `CI=true` in the environment. When `isCI()` returns true, promptfoo uses `ci-placeholder@promptfoo.dev` as the email and returns `"ok"` immediately — no verification, no email, no cloud login. All plugins and strategies work because remote generation is not disabled.

```bash
export CI=true
export OPENROUTER_API_KEY="sk-or-v1-..."
promptfoo redteam run -c promptfooconfig.yaml --output results.yaml
```

**Why this works**: `CI=true` sets `ciMode = true` in `accounts.js`. The email check function `checkEmailStatusAndMaybeExit()` returns `"ok"` immediately when `ciMode` is true. Generation proceeds via the configured `redteam.provider` (e.g., OpenRouter). No cloud account, no email, full plugin coverage.

## The #2 trap: OpenRouter API key env var name

**Symptom**: Provider returns `401 Unauthorized` / `"Missing Authentication header"` during generation.

**Root cause**: promptfoo's `openrouter:` provider reads the API key from `OPENROUTER_API_KEY` (no underscore between OPEN and ROUTER). Many projects use `OPEN_ROUTER_API_KEY` (with underscore). The env var name mismatch silently sends no auth header.

**Fix**: Always set `OPENROUTER_API_KEY` (no underscore). If your project uses `OPEN_ROUTER_API_KEY`, alias it:

```bash
export OPENROUTER_API_KEY="$OPEN_ROUTER_API_KEY"
```

In config, reference it as:
```yaml
targets:
  - id: openrouter:anthropic/claude-haiku-4.5
    config:
      apiKey: "{{env.OPENROUTER_API_KEY}}"
```

## The #3 trap: `redteam run` doesn't accept `--provider`

**Symptom**: `error: unknown option '--provider'` when running `redteam run`.

**Root cause**: `--provider` is only valid for `redteam generate`. `redteam run` reads the provider from the config file's `redteam.provider` field.

**Fix**: Set the generation provider in the YAML, not on the CLI:

```yaml
redteam:
  provider: openrouter:anthropic/claude-haiku-4.5
```

## The #4 trap: `owasp:llm:01` collection pulls in `indirect-prompt-injection` which needs `indirectInjectionVar`

**Symptom**: `Validation failed for plugin indirect-prompt-injection: Error: Invariant failed: Indirect prompt injection plugin requires config.indirectInjectionVar to be set. If using this plugin in a plugin collection, configure this plugin separately., skipping plugin.`

**Root cause**: `owasp:llm:01` is a collection that auto-expands to include `indirect-prompt-injection`. That plugin requires `config.indirectInjectionVar` to know which input variable holds untrusted data. When used inside a collection, you can't configure it.

**Fix**: Don't use `owasp:llm:01` as a collection. Instead, list the individual plugins you want and configure `indirect-prompt-injection` separately:

```yaml
plugins:
  - id: hijacking
    numTests: 5
  - id: indirect-prompt-injection
    numTests: 8
    config:
      indirectInjectionVar: query  # the input variable holding untrusted data
  - id: prompt-extraction
    numTests: 8
```

## The #5 trap: `jailbreak` strategy auto-expands to `jailbreak:meta` which requires remote generation

**Symptom**: `Error: Failed to load redteam provider 'promptfoo:redteam:iterative:meta': jailbreak:meta strategy requires remote generation, which has been explicitly disabled.`

**Root cause**: The `jailbreak` strategy (iterative LLM-as-judge) internally uses `jailbreak:meta` which requires promptfoo's cloud or a `PROMPTFOO_REMOTE_GENERATION_URL` endpoint. With `PROMPTFOO_DISABLE_REMOTE_GENERATION=1`, it hard-fails at eval time.

**Fix**: With `CI=true` (not `PROMPTFOO_DISABLE_REMOTE_GENERATION=1`), `jailbreak` works because remote generation is not disabled — it uses your configured provider. If you still hit issues, use `jailbreak-templates` (static, always local) instead of `jailbreak` (dynamic, needs an LLM to refine attacks).

## The #6 trap: `goat` strategy requires remote grading

**Symptom**: `Error: Failed to load redteam provider 'promptfoo:redteam:goat': GOAT strategy requires remote grading to be enabled.`

**Root cause**: GOAT (Generative Offensive Agent Tester) uses promptfoo's cloud to grade multi-turn conversations. Even with `CI=true`, it may require `PROMPTFOO_REMOTE_GENERATION_URL`.

**Fix**: If GOAT fails, remove it and keep `crescendo` (which works locally with your provider). Both are multi-turn strategies; crescendo is a sufficient substitute.

## The #7 trap: HTTP target returns 401 — cookie-based auth

**Symptom**: All probes return HTTP 401. Target is behind an oauth2-proxy (Traefik, oauth2-proxy) that uses a cookie, not a JWT bearer.

**Root cause**: promptfoo's `HttpProvider` sends headers you configure. If the target expects `_oauth2_proxy_<service>` cookie, a `Bearer` token won't work.

**Fix**: Pass the cookie as a header:

```yaml
targets:
  - id: https
    config:
      url: "{{env.PICKPRO_API_URL}}/pickpro/rest/v1/profiles/search"
      method: POST
      headers:
        Content-Type: application/json
        Cookie: "_oauth2_proxy_pickpro={{env.PICKPRO_OAUTH_COOKIE}}"
      body:
        query: "{{query}}"
      transformResponse: "json"
```

**Getting the cookie**: Run the Playwright auth setup, then extract from `storage-state.json`:

```bash
npx playwright test --project=auth-setup
python3 -c "
import json
d = json.load(open('auth/storage-state.json'))
for c in d.get('cookies', []):
    if c['name'] == '_oauth2_proxy_pickpro':
        print(c['value'])
"
```

**Cookie expiry**: oauth2-proxy cookies expire (default 1h refresh, 24h expire). If you get 401 mid-scan, re-run auth setup and update the env var.

## The #8 trap: Multipart file upload config schema

**Symptom**: `Invalid options: multipart.parts: Invalid input: expected array, received undefined` or `multipart.parts.0: Invalid input`.

**Root cause**: The multipart config expects a `parts` array where each part has `kind: file` or `kind: field`, not a nested object.

**Correct schema** (verified from promptfoo source `HttpMultipartConfigSchema`):

```yaml
config:
  multipart:
    parts:
      - kind: file
        name: file                    # form field name
        filename: cv-candidate.pdf
        contentType: application/pdf
        source:
          type: generated             # generated document (PDF/PNG/JPEG)
          format: pdf
          # text: "optional custom text"  # defaults to "Promptfoo generated document..."
      - kind: field
        name: extra_param
        value: "some_value"
```

**Source types**:
- `type: generated` — promptfoo generates a PDF/image with adversarial text embedded
- `type: path` — reads a file from disk: `path: /path/to/file.pdf`

## The #9 trap: Claude refuses to generate adversarial payloads during generation

**Symptom**: `Error: openrouter:anthropic/claude-haiku-4.5 returned a refusal during inference for <Plugin> test case generation.`

**Root cause**: Claude (and other aligned models) refuse to generate harmful/adversarial content even when asked to by promptfoo's generation prompts. This is the model working as intended — it's refusing to help create attacks.

**Impact**: The plugin generates 0 tests. The scan continues with other plugins.

**Fix options**:
1. Accept it — the plugin is skipped, other plugins still run
2. Use a less-aligned model for generation (e.g., `openrouter:meta-llama/llama-3.3-70b-instruct`)
3. Set `redteam.provider` to a model that doesn't refuse (at the cost of lower-quality grading)

**Note**: This is NOT a bug. It's a tradeoff: aligned models generate fewer attacks but grade better. Unaligned models generate more attacks but may produce low-quality or nonsensical probes.

## The #10 trap: npm scripts cross-contamination

**Symptom**: `npm test` (Playwright) accidentally runs promptfoo redteam configs.

**Fix**: Isolate redteam scripts completely:

```json
{
  "scripts": {
    "test": "npx playwright test",
    "redteam:promptfoo": "npm run redteam:promptfoo:chat && npm run redteam:promptfoo:cv",
    "redteam:promptfoo:chat": "npx promptfoo@latest redteam run -c redteam/promptfoo/chat-model.yaml --output redteam/reports/promptfoo/chat-results.yaml",
    "redteam:report": "npx promptfoo@latest redteam report",
    "redteam:all": "npm run redteam:promptfoo && npm run redteam:report"
  }
}
```

**Rules**:
- Redteam configs live in `redteam/promptfoo/` — NOT in `specs/` or the default `promptfooconfig.yaml` location
- `npm test` only runs Playwright — never promptfoo
- `npm run redteam:*` only runs promptfoo — never Playwright
- Never put a `promptfooconfig.yaml` at the repo root (promptfoo auto-discovers it)

## Core workflow

1. **Set env vars** — load `references/env-setup.md` for the full env var checklist
2. **Write configs** — load `references/config-schema.md` for the complete YAML schema with examples
3. **Select plugins** — load `references/plugins-local.md` for which plugins work locally vs remote-only
4. **Configure targets** — load `references/targets.md` for HTTP, LLM API, multipart, and WebSocket target configs
5. **Run scans** — load `references/cli.md` for the exact commands
6. **Review results** — load `references/reporting.md` for interpreting pass/fail/error

## Core principles (summary)

- **Always set `CI=true`** — skips email verification without disabling remote generation. This is the single most important env var.
- **`OPENROUTER_API_KEY` not `OPEN_ROUTER_API_KEY`** — promptfoo reads the env var without the underscore.
- **`redteam.provider` in YAML, not `--provider` on CLI** — `redteam run` doesn't accept `--provider`.
- **Don't use `PROMPTFOO_DISABLE_REMOTE_GENERATION=1`** — it blocks 60%+ of plugins. Use `CI=true` instead.
- **Cookie auth for oauth2-proxy targets** — pass `Cookie: _oauth2_proxy_<service>={{env.COOKIE}}` in headers, not `Authorization: Bearer`.
- **`owasp:llm:01` breaks `indirect-prompt-injection`** — use individual plugins instead of collections when you need per-plugin config.
- **`jailbreak-templates` is always local** — `jailbreak` (iterative) may need remote. Use `jailbreak-templates` as the safe default.
- **Multipart needs `parts` array** — each part has `kind: file|field`, `name`, and `source` or `value`.
- **Isolate npm scripts** — redteam configs in `redteam/`, scripts prefixed `redteam:`, never at repo root.

## References
- `references/env-setup.md` — full env var checklist (CI, OPENROUTER_API_KEY, cookies, PROMPTFOO_DISABLE_REMOTE_GENERATION)
- `references/config-schema.md` — complete YAML schema with all fields, plugins, strategies, purpose, contexts
- `references/plugins-local.md` — which plugins work locally vs remote-only, how to configure each
- `references/targets.md` — HTTP (JSON, multipart, cookie auth), LLM API (OpenRouter), WebSocket targets
- `references/cli.md` — exact commands for generate, run, eval, report
- `references/reporting.md` — interpreting pass/fail/error, reading the web UI, exporting reports

Base directory for this skill: /Users/yohan/.config/opencode/skills/promptfoo-redteam-configuration
Relative paths in this skill (e.g., scripts/, reference/) are relative to this base directory.