Orchestrating Vulnerability Research

Orchestrating Vulnerability Research

One agent hunting one target rationalizes. It finds a "probably exploitable" path, writes a confident paragraph, and grades its own paragraph as a finding. The paragraph is not the bug. This skill is the harness that stops that: give the hunt a bar it cannot talk its way around, split the target so pieces are worked in parallel, and never let the agent that built a candidate be the one that decides it is real.

It is a loop, not a pass. You run it until findings are proven or the target is genuinely exhausted — not until the first plausible writeup appears.

When to Use

• Told to find previously-unknown vulnerabilities in a whole codebase, a binary, or a named live target, with room to run many agents
• Running a bug-bounty or research campaign where depth and novelty matter more than a one-pass coverage report
• A single audit or test pass has stalled or produced only unproven "maybe" findings, and you want independent critics to break or confirm them
• You have the budget to fan out and iterate, and want the builder/critic separation and a demonstrated-trigger bar enforced across the whole effort

When NOT to Use

• One focused review of a source tree for coverage (client audit, one pass, a deliverable coverage table) — use auditing-code-for-vulnerabilities directly; this skill dispatches it, it does not replace it
• Reversing or triaging a single binary — use analyzing-binaries
• Black-box testing one web app or API methodically — use testing-web-applications or testing-apis
• Writing up the confirmed findings — use reporting-security-findings
• Tracking the campaign's evidence, provenance, and dead ends — use maintaining-engagement-state; this skill produces that record, it does not define its format
• Hunting a webshell or backdoor someone already planted (not a latent vulnerability) — use hunting-web-backdoors
• A stateless spot check — "is this one function injectable?" is one builder call, not a campaign. The harness overhead only pays off at scale.

The Loop

Five roles, run as a loop over each slice of the target. The lead never hunts and never grades; it decomposes, dispatches, and reconciles.

decompose → build → critique → iterate → smooth
   lead      hunter   critic     hunter    lead

1. Decompose — split into the smallest independently-huntable slices

The lead breaks the target into pieces that can each be hunted and judged on their own, without cross-talk. A good slice has one entry surface and a bounded reachable set. Slice by whichever axis makes pieces independent:

Write the slice list down before dispatching. A slice carries: what it covers, the reachable sink set, and the bar a finding here must clear (below). Slices that share state are a smell — merge them, or the critics will disagree because they saw different halves.

2. Build — a hunter per slice, told the goal, not the method

Dispatch one builder agent per slice with fresh context, pointed at the right domain skill for that artifact (auditing-code-for-vulnerabilities, analyzing-binaries, testing-web-applications). Give it the slice, the assets to protect, and the bar — not a script of steps. Told how, it performs the steps and reports success; told the goal and the bar, it has to actually reach them. Each hunter returns candidates: (location, bug class, the trigger it claims, the evidence it has).

Run slices in parallel; they are independent by construction. Depth per slice beats breadth across slices — one fully triggered bug is worth twenty "suspicious" notes.

3. Critique — a separate critic, fresh context, told to refute

This is the rule the whole skill exists to enforce: the builder never grades its own work. Each candidate goes to a different agent with fresh context whose job is to refute it, and which inspects the real artifact — the running binary, the actual HTTP response, the executed test, the re-read source — never the hunter's summary of it.

The bar is a demonstrated trigger, and the critic asks only whether it was met:

• Codebase: is there a concrete input that reaches the sink with attacker control intact, past every validator on the path? Not "looks reachable."
• Binary: does it crash or corrupt state under a controlled input, with the fault understood? Not "this memcpy looks unbounded."
• Live target: did the request produce the anomalous response that proves the weakness, reproducibly? Not "the error suggests injection."

If the bar is not met, the critic names the single biggest gap between the candidate and a proven bug — the missing reachability step, the validator the hunter did not account for, the input it never actually ran. That gap is the next round's work order. A critic that says only "not proven" has failed; it must say what would prove or kill it.

Bias the critic toward refutation. A candidate that survives an agent genuinely trying to break it is worth ten a builder pronounced exploitable. For high-stakes candidates, run more than one critic with different lenses (reachability, the mitigating control, does-it-actually-run) rather than three identical ones.

4. Iterate — feed the gap back, loop the slice

The builder takes the critic's gap and closes it: builds the reachability step, writes the input that actually triggers, accounts for the validator. Then the candidate goes back to a critic. Repeat until one of three stop conditions:

• Proven — the bar is met and a critic could not refute it. It becomes a finding; hand it to reporting-security-findings.
• Dead — the critic found the control that makes it safe, or repeated rounds cannot produce a trigger. Record it as a checked-and-cleared candidate in the engagement state, with why — dead candidates are coverage, and stop you re-hunting the same path.
• Budget — you have spent the slice's allotment without convergence. Downgrade to a documented "suspected, unproven" lead and move on; do not promote it to a finding to salvage the effort.

Never let a slice loop forever. The failure mode of a loop is not stopping too early — it is a builder and a lax critic passing an unproven candidate back and forth until it sounds proven.

5. Smooth — the lead reconciles across slices

Slices were hunted blind to each other; the lead is the only one that sees all results. After the per-slice loops settle:

• Deduplicate. The same root cause surfaces in several slices — one missing auth-layer check hit from five routes is one finding with five instances, not five findings.
• Chain. A weak primitive in one slice plus a reachable sink in another is often the real, higher-severity bug. Composition is invisible to any single hunter — this is where the lead earns its keep.
• Variant-sweep confirmed bugs. Every proven finding is a template; sweep the other slices for the same mistake before closing.
• Reconcile coverage. State which slices were hunted to the bar, which were time-boxed, and which were not reached. Honest coverage is the deliverable's spine — see reporting-security-findings.

Setting the Bar

The bar is the reference standard the critic compares against — the thing the agent cannot argue with. Set it per slice before hunting, and make it a demonstration, not a description:

• Codebase — a source-to-sink path with a concrete triggering input, or a written PoC that exercises it. "Attacker controls id, no tenant scope on the query, here is the request that returns another tenant's row."
• Binary — a reproducing input plus a fault analysis: the crash, the corrupted state, and whether control is influenced. A sanitizer report (ASan/UBSan) or a fuzzer-minimized case clears the bar; a hand-wave does not.
• Live target — the observed anomalous behavior, reproduced, with the request/response captured. Blind and time-based signals count only when reproduced and controlled for the environment.

If a slice cannot express a concrete bar, it is not decomposed enough. Split it until each piece has a demonstrable pass/fail the critic can check.

Running It in Practice

In an agentic harness (Claude Code, with subagents), the loop maps directly:

• The lead is your main context: it holds the slice list and the engagement state, and dispatches.
• Each builder and each critic is a subagent with its own fresh context — a critic that shares the builder's context inherits its blind spots and its optimism, and the separation is the whole point.
• Persist the slice list, the candidate worklist, and each candidate's verdict in the engagement record (maintaining-engagement-state) — the loop can run for a long time and must survive a restart without re-hunting cleared paths.
• Scale the fan-out to the budget you were given, and log what you did not reach. A campaign that silently sampled ten of forty slices and reported clean is worse than one that hunted ten and said so.

Automated discovery tools are builders inside a slice, not a substitute for the loop. A fuzzer (AFL++, libFuzzer), a taint engine (CodeQL), or a scanner produces candidates; they still go to an independent critic and the same bar. Unverified tool output promoted straight to a finding is the exact failure this harness exists to prevent.

Scope and Authorization

A discovery campaign is more dangerous than a single test, because it fans out and iterates.

• Authorization must cover the whole target and the intensity. "You may test X" is not "you may fuzz X's production parser until it crashes." Discovery techniques — fuzzing, injection sweeps, deserialization probes — cause outages and corrupt data. Confirm the target, the environment (prefer non-production), and the permitted intensity in writing before you fan out.
• A named target pulls in third-party estate. Its login federates to an identity provider, its assets sit behind a CDN, its "subdomain" is a SaaS tenant you were not authorized to touch. Enumerate ownership before hunting, and keep every builder inside the authorized boundary — fan-out makes it easy to drift out of scope without noticing.
• Novel bugs in software you do not own carry disclosure duties. A previously-unknown vulnerability is a coordinated-disclosure obligation, not a trophy. Route it through reporting-security-findings.
• Do not weaponize past the bar. The bar is a demonstrated trigger. Proving control of a sink is the goal; a full working exploit against a live third-party system is a separate authorization you probably do not have.

Rationalizations to Reject

• "The hunter said it's exploitable, so it's a finding." The hunter is the one agent that must not decide that. Independent critic, real artifact, or it is a hypothesis.
• "It looks reachable / that buffer looks unbounded / the error suggests injection." Every one of these is a candidate, not a bug. The bar is a demonstrated trigger, not a plausible read.
• "The critic and builder can share context to save tokens." Then the critic inherits the builder's blind spots and grades its optimism. The separation is the method; collapsing it deletes the value.
• "We've looped enough; promote it so the effort isn't wasted." An unproven finding is not a smaller win — it is a false positive that discredits the real ones. Downgrade it to a documented lead; do not launder it into a finding.
• "One agent found nothing, so the slice is clean." One hunter's negative is one search strategy's negative. Clean means hunted to the bar and confirmed by a critic, not "the first pass came back empty."
• "Dead candidates aren't worth recording." A checked-and-cleared path with a reason is coverage. Drop it and the next round re-hunts it, or worse, reports it as untested.
• "Decomposition is overhead; just point one agent at the repo." Then it grades itself and stops at the first plausible paragraph. Slicing is what makes independent judgment and parallelism possible.

References

• auditing-code-for-vulnerabilities — the per-slice hunter for source code
• analyzing-binaries — the per-slice hunter for compiled targets
• testing-web-applications, testing-apis — the per-slice hunters black-box
• reporting-security-findings — where proven findings go
• maintaining-engagement-state — where the slice list, worklist, and verdicts live
• CWE and OWASP ASVS for classifying what a hunter is looking for