Customize

Customize the vuln-pipeline

This pipeline ships as an opinionated C/C++ + AddressSanitizer demo. Its real shape is more general: an agent crafts an input, runs a target in a sandbox, a detector fires, a second agent verifies, a third agent analyses exploitability. Every noun in that sentence can be swapped. Your job is to interview the user, figure out which nouns they want to swap, and rewrite the relevant files.

The existing C/C++ code is the worked example. You don't need a playbook for each domain — read what's there, understand what's generic vs. ASAN-specific, and adapt.

STEP 1 — Read the pipeline (do this BEFORE asking anything)

Skim these files so your questions are grounded:

• README.md — pipeline overview (recon → find → grade → judge → report)
• harness/cli.py — orchestration; shows how stages wire together and what lands on disk
• harness/find.py, harness/grade.py, harness/report.py — the three container-agent loops; mostly generic plumbing
• harness/prompts/find_prompt.py, harness/prompts/grade_prompt.py — the C/C++-specific parts; bug taxonomy, quality tiers, grading rubric
• harness/prompts/report_prompt.py, harness/prompts/report_grader_prompt.py — also C/C++-specific; exploitability sections (primitive, heap layout, escalation path) and the rubric that scores them
• harness/prompts/judge_prompt.py — triage prompt; keys on ASAN excerpts and memory-safety crash classes
• harness/prompts/system_prompt.py — authorization block; hard-codes "C/C++ target" and "sanitizer output"
• harness/asan.py — stack-trace parser for dedup/judge signatures; ASAN-specific regex
• harness/artifacts.py — CrashArtifact, GraderVerdict, JudgeVerdict, ReportVerdict data contracts
• harness/config.py, targets/drlibs/config.yaml — target config schema
• targets/README.md — how a target directory is structured (Dockerfile + config.yaml + entry wrapper)

You don't need agent.py, docker_ops.py, recon.py, judge.py, or novelty.py in detail — they're generic plumbing (judge/novelty domain-specificity lives in the prompts and the asan parser, not the flow).

STEP 2 — Interview the user

Use AskUserQuestion to gather requirements. Start with broad context, then narrow to technical specifics based on what they say.

Round 1 — Context (always ask these first, together)

Two open-ended questions to understand who you're talking to and what they're after. Expect most answers to come via Other as free text — the options are there to prompt thinking, not to constrain.

Question A — Operating context

• header: Context
• question: What's your operating environment? Who will run this pipeline and why?
• options: a few archetypes as inspiration — e.g. "Pentesting firm — client engagements, need reportable findings", "Internal appsec team — scan our own services in CI", "Smart-contract auditor — pre-deployment reviews", "Security researcher — hunting novel bug classes". These tell you what output format, grading rigor, and workflow integration matter.

Question B — Goal

• header: Goal
• question: Describe in your own words what you want this pipeline to find. What kind of target, what kind of bugs?
• options: 2–3 concrete examples (e.g. "Web vulnerabilities like SQLi/XSS in HTTP services", "Reentrancy and access-control bugs in Solidity contracts", "Deserialization RCE in Java microservices").

The context answer calibrates your follow-ups: a pentesting firm probably cares about CVSS scoring and SARIF output; a researcher may want differential testing and novel detection signals; an internal team likely wants CI integration and low false-positive rates.

Round 2 — Technical follow-ups (adaptive — derive from round-1 answers)

Parse their round-1 answers against the axes of variation below. For each axis left ambiguous, ask a targeted follow-up. Batch up to 4 questions per AskUserQuestion call. Common follow-ups:

• Detection signal — "How will the pipeline know it found something?" (crash, exception, canary file appears, DNS callback, differential mismatch, invariant violation)
• PoC shape — "What does a proof-of-concept look like?" (single file, HTTP request sequence, transaction list, test-pipeline code)
• Isolation — "Where does the target run?" (Docker, VM, testnet, remote sandbox, or no execution — static-only)
• Grading criteria — "What makes a finding high-quality vs. low-quality in this domain?"
• Exploitability analysis — "What sections should a report contain?" The C/C++ report has primitive · reachability · heap layout · escalation path · constraints. A web-vuln report might want injection vector · auth bypass · data exposure · chaining potential. Ask what they need, or whether they want the report stage at all.
• Novelty/upstream check — "Should the pipeline check if a finding is already fixed upstream?" The C/C++ version shallow-clones the target's GitHub and checks git log <commit>..HEAD -- <crash_file>. Only applies if targets have a canonical upstream and a sensible "crashing file" to key on — many domains won't.
• Scope — "Replace the C/C++ support entirely, or keep it alongside the new domain via a profile system?"

Keep going until you can fill in every row of the architecture map in STEP 3. If an answer is vague, ask a narrower follow-up rather than guessing.

Background — axes of variation (context for formulating follow-ups)

These are the dimensions along which customers might want to deviate from the C/C++ demo. Use this list to spot gaps in the user's description and generate follow-up questions — do not present it as a menu.

Vulnerability class: memory safety · web/API (SQLi, XSS, SSRF, XXE, path traversal, IDOR) · deserialization RCE · logic/race (TOCTOU, privilege escalation) · crypto (weak RNG, timing, nonce reuse) · DoS (ReDoS, hash flooding) · smart contracts (reentrancy, access control, front-running) · ML/AI (prompt injection, jailbreaks, data extraction) · protocol parsing

Target shape: CLI binary + file · HTTP service · library via test harness · network daemon · smart contract · browser extension · mobile app

Detection mechanism: crash/abort · uncaught exception · sanitizer hooks (Jazzer/Atheris) · outcome-based (canary file, DNS callback, shell spawn) · differential testing · invariant violation · taint tracking

Input modality: single file · HTTP request chain · multi-file archive · stdin stream · args + env + config combo · transaction sequence

Isolation boundary: Docker container · full VM · remote sandbox · local testnet · none (static analysis)

Dedup signature: (crash_type, top_frame) · (vuln_type, endpoint, param) · (function, state_transition) · (component, precondition)

Report structure: primitive/heap/escalation (memory safety) · vector/auth/exposure (web) · invariant/path/impact (contracts) · or drop the report stage entirely if find+grade is the deliverable

Output format: result.json + poc.bin · SARIF · Nuclei template · prose report

Patch verification signal: ASAN-clean exit · uncaught-exception-free · sanitizer hook silent (Jazzer/Atheris) · canary file untouched · invariant assertion holds · differential output matches reference. This is what _t1_passes() in patch_grade.py encodes — "the bug is gone" for the new domain.

Background — architecture map (what changes vs. what stays)

STEP 3 — Present a plan and get confirmation

Before editing anything, summarize back to the user:

1. What you understood — restate their goal in one sentence
2. What will change — list each file you'll edit with a one-line rationale
3. What stays — reassure them the orchestration core is untouched
4. Open questions — anything you're still unsure about

Wait for explicit approval. If they adjust the plan, incorporate and re-confirm.

STEP 4 — Execute

Edit the files per the approved plan. Work through them in dependency order: prompts and parser first (they're standalone), then config/artifacts, then the target template, then README. Commit incrementally if the user wants checkpoints.

STEP 5 — Validate

1. Add a canary target under targets/<domain>-canary/ with 2–3 planted bugs of the new class
2. Run: bin/vp-sandboxed run <domain>-canary --model <model-id> --runs 3 --parallel --stream --max-turns 50 (use Claude Opus unless the user specifies a different model). Run ./scripts/setup_sandbox.sh once first if the sandbox isn't already set up.
3. Confirm all planted bugs are found and graded PASS
4. Confirm judge triage worked: cat results/<domain>-canary/<ts>/reports/judge_log.jsonl — expect one NEW per distinct bug, DUP_SKIP for repeats
5. Confirm reports landed: ls results/<domain>-canary/<ts>/reports/bug_*/report.json and spot-check section scores
6. Run vuln-pipeline dedup results/<domain>-canary/ and confirm signatures group correctly