Adversarial Testing

Adversarial Testing

Normal tests ask "does it do what we meant?" Adversarial tests ask "what did we fail to imagine?" Failures cluster in the input regions nobody modeled, so a suite built from expected inputs is structurally blind to them — not because it is incomplete, but because it was drawn from the wrong distribution.

This skill covers classical software and AI systems together, because they are the same problem at different severities. Both come down to one question, and that question routes the entire method.

The spine: what is your oracle?

A test oracle is whatever tells you an output is wrong. Everything below follows from how much of one you have.

Weyuker formalized the oracle problem in 1982, and it is the reason this skill exists. Where you cannot say what a single correct answer is, you can almost always still say what must stay true across answers. That shift — from values to invariants to rates — is the whole craft.

Most real targets contain more than one row. A web service has a crash-only parser, an undecidable-deterministic pricing engine, and an undecidable-probabilistic support chatbot. Classify each surface separately. Applying chatbot methodology to the parser wastes the strongest oracle you have.

When to run

• /adversarial-testing [target]
• The user wants a system probed for failures rather than verified against a spec.
• A model, agent, or tool-using system is going to production and nobody has attacked it yet.
• Coverage is high, tests are green, and bugs still reach users — a classic sign the suite samples only the expected distribution.

Do not use this skill for:

• Finding bugs by reading code → code-audit-deep.
• Design and maintainability problems → code-smells.
• Fast deploy-gate checks → smoke-test.
• Writing the ordinary test suite → tdd.

This skill assumes the ordinary suite exists. It attacks what that suite was never shaped to catch.

Step 1 — Authorization and blast radius, before anything else

This skill generates working exploits and sends hostile traffic. That is the point, and it is why scope comes first, not last.

Establish and write down, in the report header:

• What system, precisely — repo path, service name, endpoint, model and version.
• Who owns it. If it is not the user's system or their employer's, stop and ask for the engagement scope. "It's fine" is not scope.
• Which environment. Default to non-production. Say so explicitly. If the user directs you at production, name what you will not do there: no destructive payloads, no data exfiltration beyond a canary, no denial-of-service load.
• What is out of bounds — third-party APIs the target calls, shared infrastructure, other tenants, real customer records.

If you cannot fill these in, the deliverable is that gap, not a corpus of attacks. Say so and stop.

Step 2 — Classify each surface

Walk the target and place every surface in a row of the spine table. Recon that is usually enough:

git ls-files | head -50                    # shape of the repo
rg -n "def |fn |func |function " --stats   # rough entry points
rg -n "input\(|request\.|argv|stdin|json\.loads|yaml\.load"   # untrusted input
rg -n "openai|anthropic|langchain|llm|prompt|completion"      # model surfaces
rg -n "subprocess|exec|eval|os\.system|shell=True"            # tool-execution surfaces

Produce a short table of surface → oracle class → chosen method before generating a single test. Skipping this is how a session turns into undirected payload spraying.

Step 3 — Read the layer reference you actually need

Load only what applies. Each is written to be read on its own.

references/tooling.md also carries install commands and the current gotchas — repos that moved, packages that split. Check it before you tell the user to install anything, because the obvious answer is stale in at least two cases.

Step 4 — Generate families, not examples

The single most common failure in this work is hand-writing a dozen clever attacks. A dozen attacks measures a dozen attacks. What you want is a generator: a relation, a mutation operator, an attack template with slots, so that coverage grows with compute rather than with your patience.

For each surface, produce a corpus of at least a few hundred cases from a small number of generators, then keep the generators in the repo. The corpus is disposable. The generators are the deliverable.

Step 5 — Measure against a baseline

A number with no comparison is not a result. "Attack success rate 12%" means nothing alone — 12% against what?

Always produce at least one baseline: the same corpus against the undefended configuration, the previous release, or a competing implementation. For probabilistic targets, run each case several times and report the spread, because a single run of a non-deterministic system is an anecdote.

Report the pair, always. 12% (baseline 47%) is a finding. 12% is a number.

Step 6 — Triage by reachability, then severity

Adversarial methods generate volume, and volume without triage is noise that trains the team to ignore you.

For each candidate finding, in this order:

1. Reproduce it. A finding you cannot re-run is not a finding.
2. Is it reachable from an untrusted input in the deployed configuration? Unreachable code paths go in an appendix, not the report body.
3. What does it actually get an attacker? Crash, data, privilege, money, or nothing.
4. Is it equivalent to one you already filed? Fuzzers produce hundreds of inputs that hit one bug. Cluster by root cause, not by input.

Report format

Use this structure. It puts the reproducible facts first, because the first question anyone asks is "show me."

# Adversarial test report — <target>

## Scope
Target, owner, environment, out-of-bounds list, date.

## Surfaces and methods
| Surface | Oracle class | Method | Corpus size |

## Findings
For each, most severe first:
### <short name>
- **Reachable from**: <untrusted entry point>
- **Repro**: exact command or input, copy-pasteable
- **Observed**: what happened
- **Expected**: what should have
- **Impact**: what it gets an attacker
- **Fix sketch**: one or two lines

## Measured results
| Metric | Result | Baseline | Corpus |

## Not found
What you tested that held. This is the half of the report that tells the
reader what the numbers cover, and it is the half everyone omits.

## Generators
Where the corpus generators live, and how to re-run them.

The "Not found" section is not padding. A report listing only failures gives no way to distinguish a hardened system from a shallow test pass.

Failure modes to avoid

• Reporting raw fuzzer output. Hundreds of crashes are usually a handful of bugs. Cluster before reporting.
• Attack success rate with no baseline. See step 5.
• Testing the model when the bug is in the system. Most real AI incidents are ordinary application-security failures around the model — caching, auth, logging — not alignment failures inside it. Check the plumbing before concluding the model is the problem.
• One run of a probabilistic system. Non-determinism means a single pass proves nothing in either direction.
• Treating published attack rates as current. Injection results age fast — a technique reported at 95% against one model version may be under 10% against the next. Re-measure on the actual target rather than quoting a paper.
• Generating exploits for a system nobody confirmed you may test. Step 1 exists for this.

Depth over breadth

If time is short, one surface tested to exhaustion beats six surfaces sampled. A generator, a baseline, and a triaged finding on a single parser is a real result. A page of untested attack ideas across the whole system is a reading list.