Bugbash core workflow
Charter
- Goal: Reproduce misbehavior or characterize risk with evidence—not to own remediation.
- Default: No production patches, no “drive-by” refactors. If the user asks how to fix, then you may discuss fixes.
- Honesty: If something might be a bug but you cannot reproduce, log it under Suspected / flaky (see below)—do not upgrade to S2 without evidence.
0. Mission brief (before Phase A)
Keep this to two minutes unless the user already specified everything.
Confirm or state assumptions for:
| Field |
Question |
| Target |
What surface or feature is in scope? |
| Environment |
Local, staging URL, branch/commit? |
| Intensity |
quick | standard | deep (default: standard) |
| Constraints |
No prod? No billing? No deletes? Data rules? |
| Timebox |
Optional wall-clock or scenario cap |
If anything safety-critical is unclear, ask once; otherwise proceed with labeled assumptions.
Intensity modes (pick one)
quick (~ smoke + sanity)
- Primary journeys only; a small set of negative checks (empty input, obvious auth miss).
- No deliberate concurrency soaks or large payloads.
- Code: only if a failure already appeared.
standard (default)
- Full Phase A below at normal depth; representative edge cases across validation, authz, state, errors.
- Light concurrency (a few parallel actions), moderate payload sizes—stay well under abuse.
deep
- Everything in standard plus broader negative matrices on the riskiest area, longer soaks where safe, more parallel or rapid-fire sequences (still non-destructive unless approved).
- Explicit residual-risk section is mandatory.
Risk triage (where to spend minutes)
Before testing randomly, rank surfaces by:
- Data sensitivity (accounts, PII, permissions).
- Money or irreversible actions (payments, deletes, publishes).
- Cross-tenant or cross-user boundaries.
- New or complex code paths in scope.
Spend deep mode’s extra time on the top of this list.
Phase A — Running system (do this first)
A1. Recon
- Surfaces: CLI, HTTP, UI, workers, webhooks, migrations, flags.
- How to run; version/commit if known.
- Product invariants (who can do what, what must never happen).
A2. Happy path + smoke
- Core journeys end-to-end; note ordering or timing sensitivity.
A3. Edge and negative tests
Tune depth to intensity. Cover when relevant:
- Boundaries, types, unicode, whitespace, missing fields.
- AuthN/AuthZ: wrong user, role gaps, token edge cases.
- Idempotency: duplicates, retries, double submit.
- State machine: skip steps, invalid transitions, refresh/back.
- Time: timeouts, slow paths, rate limits.
- Dependencies: failures, partial success, error messaging.
A4. Stress / concurrency (standard+; deeper in deep)
- Non-production or approved sandboxes only.
- No DoS; cap parallelism and payload size conservatively.
A5. Observability
- Logs, console, network; redact secrets in notes.
Phase B — Code (after signals)
- Map symptoms to routes, handlers, validators, jobs, queries.
- Look for similar bugs and missing tests—still no fixes unless asked.
Evidence bar (minimum per severity)
| Severity |
Minimum evidence |
| S1 |
Repro twice OR one repro plus strong corroboration (e.g. clear data corruption, definitive 500 + stack); state blast radius. |
| S2 |
Reliable repro steps + concrete artifact (status/body/log/assertion). |
| S3 |
Clear repro; one run acceptable if stable. |
| S4 |
Observable issue; screenshot or short description OK. |
If you cannot meet the bar, downgrade severity or move to Suspected / flaky.
Suspected / flaky (separate from confirmed bugs)
Track these apart from the main findings table:
| ID |
Hypothesis |
Attempts |
Flake rate |
Notes |
Do not merge unconfirmed flakes into S1/S2 without meeting the evidence bar.
Output 1 — Findings tracker (human-readable)
| ID |
Severity |
Surface |
Summary |
Repro steps |
Expected |
Actual |
Evidence |
Confidence |
Severity: S1–S4 as before (critical → low). Confidence: High / Medium / Low.
Output 2 — Session coverage (always)
Short audit trail:
- Surfaces exercised (bullet list).
- Scenarios attempted (approximate count or buckets).
- Not tested (explicit gaps—time, access, environment).
Output 3 — Optional machine-readable block
If the user might file tickets or script follow-ups, end with a fenced JSON array using this shape (omit sensitive values):
[
{
"id": "BB-001",
"severity": "S2",
"surface": "api",
"title": "Short title",
"repro": ["step1", "step2"],
"expected": "...",
"actual": "...",
"evidence_type": "http|log|ui|cli",
"confidence": "high"
}
]
What Bugbash is not
- Not a substitute for formal security assessment unless explicitly scoped.
- Not permission to exfiltrate data or attack third parties.
- Not an excuse to rewrite unrelated code while “testing.”
Safety
- No real PII or production secrets in artifacts.
- Stop when the user revokes tools or scope.
- Destructive actions need explicit approval.
1---2name: bugbash-core3description: Core bug-bash workflow with mission brief, intensity modes, risk triage, evidence bars, and structured outputs—running system first, code second; find and reproduce, do not fix unless asked.4---56# Bugbash core workflow78## Charter910- **Goal:** **Reproduce** misbehavior or **characterize** risk with evidence—not to own remediation.11- **Default:** No production patches, no “drive-by” refactors. If the user asks how to fix, **then** you may discuss fixes.12- **Honesty:** If something **might** be a bug but you cannot reproduce, log it under **Suspected / flaky** (see below)—do not upgrade to S2 without evidence.1314---1516## 0. Mission brief (before Phase A)1718Keep this to **two minutes** unless the user already specified everything.1920Confirm or state assumptions for:2122| Field | Question |23|-------|----------|24| **Target** | What surface or feature is in scope? |25| **Environment** | Local, staging URL, branch/commit? |26| **Intensity** | `quick` \| `standard` \| `deep` (default: standard) |27| **Constraints** | No prod? No billing? No deletes? Data rules? |28| **Timebox** | Optional wall-clock or scenario cap |2930If anything safety-critical is unclear, **ask once**; otherwise proceed with labeled assumptions.3132---3334## Intensity modes (pick one)3536### `quick` (~ smoke + sanity)3738- Primary journeys only; a **small** set of negative checks (empty input, obvious auth miss).39- **No** deliberate concurrency soaks or large payloads.40- Code: only if a failure **already** appeared.4142### `standard` (default)4344- Full Phase A below at normal depth; representative edge cases across validation, authz, state, errors.45- **Light** concurrency (a few parallel actions), **moderate** payload sizes—stay well under abuse.4647### `deep`4849- Everything in standard plus **broader** negative matrices on the riskiest area, **longer** soaks where safe, **more** parallel or rapid-fire sequences (still non-destructive unless approved).50- Explicit **residual-risk** section is mandatory.5152---5354## Risk triage (where to spend minutes)5556Before testing randomly, rank **surfaces** by:57581. **Data sensitivity** (accounts, PII, permissions).592. **Money or irreversible actions** (payments, deletes, publishes).603. **Cross-tenant or cross-user** boundaries.614. **New or complex** code paths in scope.6263Spend **deep** mode’s extra time on the top of this list.6465---6667## Phase A — Running system (do this first)6869### A1. Recon7071- Surfaces: CLI, HTTP, UI, workers, webhooks, migrations, flags.72- How to run; version/commit if known.73- Product **invariants** (who can do what, what must never happen).7475### A2. Happy path + smoke7677- Core journeys end-to-end; note ordering or timing sensitivity.7879### A3. Edge and negative tests8081Tune depth to **intensity**. Cover when relevant:8283- Boundaries, types, unicode, whitespace, missing fields.84- AuthN/AuthZ: wrong user, role gaps, token edge cases.85- Idempotency: duplicates, retries, double submit.86- State machine: skip steps, invalid transitions, refresh/back.87- Time: timeouts, slow paths, rate limits.88- Dependencies: failures, partial success, error messaging.8990### A4. Stress / concurrency (standard+; deeper in `deep`)9192- Non-production or approved sandboxes only.93- No DoS; cap parallelism and payload size **conservatively**.9495### A5. Observability9697- Logs, console, network; redact secrets in notes.9899---100101## Phase B — Code (after signals)102103- Map symptoms to routes, handlers, validators, jobs, queries.104- Look for **similar bugs** and **missing tests**—still no fixes unless asked.105106---107108## Evidence bar (minimum per severity)109110| Severity | Minimum evidence |111|----------|------------------|112| **S1** | Repro **twice** OR one repro plus **strong** corroboration (e.g. clear data corruption, definitive 500 + stack); state blast radius. |113| **S2** | Reliable repro steps + concrete artifact (status/body/log/assertion). |114| **S3** | Clear repro; one run acceptable if stable. |115| **S4** | Observable issue; screenshot or short description OK. |116117If you cannot meet the bar, **downgrade** severity or move to **Suspected / flaky**.118119---120121## Suspected / flaky (separate from confirmed bugs)122123Track these apart from the main findings table:124125| ID | Hypothesis | Attempts | Flake rate | Notes |126|----|------------|----------|------------|-------|127128Do **not** merge unconfirmed flakes into S1/S2 without meeting the evidence bar.129130---131132## Output 1 — Findings tracker (human-readable)133134| ID | Severity | Surface | Summary | Repro steps | Expected | Actual | Evidence | Confidence |135|----|----------|---------|---------|-------------|----------|--------|----------|------------|136137**Severity:** S1–S4 as before (critical → low). **Confidence:** High / Medium / Low.138139---140141## Output 2 — Session coverage (always)142143Short audit trail:144145- **Surfaces exercised** (bullet list).146- **Scenarios attempted** (approximate count or buckets).147- **Not tested** (explicit gaps—time, access, environment).148149---150151## Output 3 — Optional machine-readable block152153If the user might file tickets or script follow-ups, end with a fenced JSON array **using this shape** (omit sensitive values):154155```json156[157 {158 "id": "BB-001",159 "severity": "S2",160 "surface": "api",161 "title": "Short title",162 "repro": ["step1", "step2"],163 "expected": "...",164 "actual": "...",165 "evidence_type": "http|log|ui|cli",166 "confidence": "high"167 }168]169```170171---172173## What Bugbash is **not**174175- **Not** a substitute for formal security assessment unless explicitly scoped.176- **Not** permission to exfiltrate data or attack third parties.177- **Not** an excuse to rewrite unrelated code while “testing.”178179---180181## Safety182183- No real PII or production secrets in artifacts.184- Stop when the user revokes tools or scope.185- Destructive actions need **explicit** approval.
riwaht