Repo Bug Finder

Repo Bug Finder

Purpose

Find more real Bugs by giving the LLM the densest useful code surface, then labeling evidence strength honestly. This skill is discovery-only: it builds profiles, probes, context packs, triage records, and readiness data. Final Bug records and HTML reports belong to the package subskill.

Core Principle

Scripts retrieve facts; the LLM judges semantics. Do not grow a large hard-coded vulnerability library. Use scripts to expose repository shape, entry points, risky primitives, siblings, and local context, then ask the LLM to plan repo-specific probes from that profile.

Discovery is mode-driven. The agent must not stay in the same discovery mode for more than two consecutive rounds unless that mode yields a new confirmed/probable finding family. Grep, optional local Semgrep-compatible scanning, context packs, and direct reads are tools inside modes, not modes by themselves.

Semgrep is an optional third-party local CLI accelerator, not a dependency. Use only local semgrep scan with skill-owned rules when it is already installed and allowed by the user's policy. Do not require Semgrep, do not log in, do not use Semgrep AppSec Platform, semgrep ci, Registry rules, Pro rules, managed policies, or result upload. If Semgrep is unavailable or inappropriate, continue with grep/Python AST fallback and Deep Flow analysis.

Methodology Contract

The required discovery method is now a three-layer state machine:

Profile
-> State machine selects mode
-> Harvest / Amplify / Deep Flow / Evidence Hardening
-> Discovery Readiness
-> Next mode recommendation
-> Package only when requested or required modes are complete

The layers are:

• Harvest: early Bug harvest, Fast Scout, optional local Semgrep Scout when available/policy-approved, direct-read anchors, low-hit precise packs, and config/CLI/deploy/frontend/scheduler quick passes. Output candidates freely; do not wait for perfect package evidence.
• Amplify: candidate upgrade, zero-record repo or zero-surface sweep, pattern amplification across sibling endpoints/repos, optional local Semgrep-compatible family scans, and R2 hardening. A confirmed/probable family should trigger amplification.
• Deep Flow: mandatory business-flow analysis for money, lifecycle, permissions, files, async jobs, retries, cleanup, and state machines. Grep or local scanner anchors may locate entry points, but the LLM must read and explain the flow.

Architecture signal, pattern seed, family, optional scanner anchors, and evidence debt synthesis are recall amplifiers. They preserve high-recall leads without replacing Bug discovery, skipping LLM planning, or creating final Bugs by themselves.

This is not a loose suggestion. Do not replace it with free-form browsing, a grep-first checklist, direct agent exploration, or final report writing. Universal patterns may seed retrieval only after profiles exist and the LLM has turned them into repo-specific probes. Fast Scout and optional local scanner scout are allowed early because they produce untriaged anchors, not Bug verdicts.

Workflow

1. Initialize a clean dated workspace with scripts/init_bug_workspace.py --project <name> --base-dir <parent-dir>. Use --workspace <workspace> only when the user explicitly wants a custom or existing workspace path.
2. Freeze the repository roster with scripts/discover_repositories.py /path/to/repos --workspace <workspace>. --base-dir /path/to/repos is accepted as a compatibility alias for the source root.
3. Build repository profiles with scripts/profile_repositories.py, including neutral structure_map observations plus compact work/repo-profiles/summary.json and index.md for large-repo planning. Immediately run scripts/update_progress_ledger.py --phase profiling --workspace <workspace> so later quality gates have the required phase history.
4. Read references/llm-risk-planning.md; perform Native Semantics Fanout from detected languages/frameworks/surfaces. The LLM should derive repo-specific invariants and falsifiers from profile facts, not from a hard-coded vulnerability library.
5. Run scripts/generate_fast_scout_plan.py --workspace <workspace> after profiles and before broad packs. Fast Scout outputs work/risk-plans/fast-scout.json anchors with verdict: untriaged-anchor; anchors are selected round-robin across repos so early attention is broad.
6. Run scripts/plan_invariant_matrix.py --workspace <workspace> to create profile-derived untested invariant rows. Matrix rows are planning prompts, not Bug verdicts.
7. Run scripts/run_semgrep_scout.py --workspace <workspace> only when Semgrep is installed and local policy allows third-party CLI use. If it is unavailable or not allowed, note the optional install path only as policy-dependent and continue with grep/Python AST fallback. Scanner hits are untriaged anchors only.
8. Run scripts/deep_flow_plan.py --workspace <workspace> and reserve early attention for required deep flows. For <=3 repos, deep flow starts in round 1 or 2; for larger groups, sample the highest business-priority repos.
9. For promising Fast Scout, local scanner, or direct-read anchors, create real ad-hoc packs with scripts/create_ad_hoc_context_pack.py; do not use direct-read::... pseudo refs for package-ready evidence.
10. Have the LLM mine repo-specific structure patterns from the profiles, Fast Scout anchors, invariant matrix, architecture signals, and prior sufficiency gaps, then write repo-specific probes with profile_coverage. Load references/probe-schema.md before writing probe JSON. For repo groups or bounded test rounds, profile_coverage must include per-repo repos_without_probe_reasons and, when applicable, repos_without_context_pack_reasons; a single generic coverage_notes sentence is not enough. Structure patterns are hypotheses for retrieval, not Bug verdicts.
11. Write round probes to a named file such as work/risk-plans/probes-<round>.json (for example, probes-round-1.json or probes-test-round-1.json), then run scripts/build_context_packs.py --probes <that-file> --round-id <round>. work/risk-plans/probes.json is allowed only as a temporary scratch alias. Use repo for one target repo or repos for a bounded multi-repo probe; omit both only when intentionally running across the whole roster. Probe queries are case-insensitive literal substrings by default; prefix a query with regex: when using regex syntax such as |, \s, .*, or escaped characters. If every pack has zero hits, or if the builder prints query/probe warnings, fix the probe file and rerun the same round with --replace-round before triage. Keep probe files compact: for bounded tests or early rounds, prefer 8-20 high-value probes and split additional work into the next named round instead of hand-writing a huge JSON file. work/context-packs/index.json is cumulative by default. If re-running the same round/probe file after an interrupted or bad run, use --replace-round instead of creating duplicate suffixed packs.
12. Run scripts/score_context_packs.py --workspace <workspace> before each triage pass. Use the scored queue to choose high-utility, diverse packs; do not discard low-scoring packs.
13. Triage context packs using references/triage-readiness.md and references/proof-bundle.md; cite packs with source_packs, or add pack_reviews for reviewed packs with no findings. Use the exact triage schema keys from the reference (id, priority, status, confidence, missing_gates, and the five core evidence fields). Do not substitute record_id, priority_hint, evidence_level, or other ad-hoc aliases in triage files. Write small triage batches, normally no more than 8-12 reviewed packs per file, run scripts/lint_triage_files.py --workspace <workspace> on the batch, then merge and validate before continuing.
14. Run adversarial R2 for P1/P2 confirmed, probable, and candidate records before promoting them into clean package-ready status. If R2 is written inside false_positive_review, run scripts/normalize_r2_evidence.py --workspace <workspace> before merge.
15. Run an almost-missed recall pass: recover candidates hidden in pack_reviews, split variants by endpoint/repo/resource/fix boundary, and sweep P3/P4 reliability, leakage, DoS, config, logging, frontend sink, and business-flow leads.
16. Run scripts/update_progress_ledger.py --phase <phase> --workspace <workspace> before the first quality gate if the current phase has not been recorded. The ledger needs at least profiling and triage/recall snapshots for validation to pass.
17. Run scripts/run_discovery_quality_gate.py --workspace <workspace> after triage and again after any post-gate edit to triage, packs, probes, or evidence fields. This sequentially merges readiness, consolidates families, runs zero-record/candidate/pattern/aggregate checks, synthesizes architecture signals, extracts pattern seeds, builds evidence debt, validates, reviews sufficiency, and recommends next mode. The last quality-gate JSON must reflect the final artifact state. Do not run these quality scripts as parallel tool calls; some agent runtimes cancel sibling calls after one warning/error.
18. If you run the quality scripts manually instead of the helper, run them sequentially in the same order as run_discovery_quality_gate.py, and continue through sufficiency and next-mode even if validation reports recall/evidence debt.
19. Validate discovery with scripts/validate_discovery.py --require-readiness; use --strict-evidence only before packaging and --require-complete-coverage only before claiming complete coverage.
20. The next-mode recommendation from scripts/recommend_next_discovery_mode.py is the default next action when the user says "continue".
21. Tell the user whether readiness is a partial snapshot or complete coverage, include scope, package-ready-by-status, clean package-ready, family/backlog counts, architecture-signal counts, pattern-seed counts, evidence-debt counts, deep-flow coverage, and the human-readable sufficiency recommendation. Use the machine decision code only as supporting detail when useful.
22. Use subskills/package only after discovery readiness exists and the user wants final Bug files, indexes, handoff materials, or an HTML report.

Run scripts/update_progress_ledger.py --phase <phase> after every major phase and relay the printed counts to the user.

Discovery Artifacts

The discovery workspace uses this shape:

<workspace>/
  work/
    repo-roster.json
    repo-profiles/
    risk-plans/
      fast-scout.json
      fast-scout.md
      invariant-matrix.json
      invariant-matrix.md
      pattern-seeds.json
      pattern-seeds.md
    context-packs/
      triage-queue.json
      triage-queue.md
    triage/
    discovery-readiness.json
    discovery-readiness.md
    finding-families.json
    finding-families.md
    architecture-signals.json
    architecture-signals.md
    evidence-debt.json
    evidence-debt.md
    discovery-next-mode.json
    discovery-next-mode.md
    zero-record-repos.json
    candidate-upgrade-queue.json
    pattern-amplification-plan.json
    quality/
      discovery-quality-gate.json
      discovery-quality-gate.md
      aggregate-finding-validation.json
    semgrep/
    progress-ledger.jsonl
  submit/
    quality/

Discovery must not write submit/findings/BUG-* files or submit/bug-audit-report.html.

Evidence Levels

• confirmed: code evidence, trigger path, failure mode, affected resource, and impact are concrete.
• probable: code evidence, failure mode, affected resource, and impact are concrete; only trigger path may be hypothesized.
• candidate: any non-trigger core field is missing, or more than one field is missing.
• refuted: the code path is unreachable, generated-only, sanitized, test-only, or otherwise contradicted by evidence.

Confirmed and probable records are package-ready by status. Candidates stay visible in readiness/backlog. Refuted records stay visible long enough to explain why they were dropped.

Readiness reports two package-ready counts:

• package_ready_by_status: all confirmed + probable records.
• package_ready_clean: confirmed/probable records whose source packs are indexed, whose schema/package-blocking debt is absent, and whose P1/P2 records have adversarial R2 evidence. Use this count for packaging decisions.

Completion Handoff

When validate_discovery.py --require-readiness passes, do not end with only "discovery complete." Give the user a short handoff:

• work/discovery-readiness.json is ready.
• work/discovery-sufficiency.json is ready.
• Context pack coverage is partial or complete.
• Report scope: repos in scope, probe repos, context-pack repos, triaged packs, saturated packs, and uncovered repos when present.
• Count confirmed, probable, candidate, refuted, package-ready records, and recall backlog.
• Distinguish package_ready_by_status from package_ready_clean.
• Count architecture risk signals and name the highest-value next probes when signals exist.
• Count pattern seeds and remind the user that seeds must be turned into repo-specific probes by LLM planning.
• Count evidence debt separately from package-ready findings. Evidence debt does not block continued discovery; it only names work needed before packaging affected records.
• State whether the almost-missed recall pass recovered candidates or variants.
• State required/completed deep-flow count. If required deep flows are incomplete, call the snapshot a pattern-scan snapshot, not sufficient discovery.
• State the next forced mode from work/discovery-next-mode.json when the user wants to continue.
• State the human-readable sufficiency recommendation from review_discovery_sufficiency.py first. Avoid leading with raw machine codes like continue-high; those belong in work/discovery-sufficiency.json.
• State validation as one of: passed without warnings, passed with recall/evidence debt, or failed. Do not collapse warnings into "success"; name the debt that blocks packaging.
• If the recommendation says more discovery is strongly recommended or likely worthwhile, lead with the next recall round before asking about packaging.
• Say that final Bug records and HTML report are generated by repo-bug-finder:package.
• Ask whether to continue recall or run the package subskill now.

Do not auto-package unless the user already requested final reports or packaging.

⛔ BLOCKING GATES

• Do not package during discovery.
• Do not skip the method contract: profile, LLM plan, retrieval/context packs, triage, R2, readiness.
• Do not run Fast Scout before profiles exist, and do not treat Fast Scout anchors as Bug verdicts.
• Do not start from a fixed vulnerability checklist before repo profiles exist.
• Do not embed Bug verdicts in profiles; scripts expose neutral structure and the LLM infers repo-specific patterns.
• Do not call a scanner hit a Bug without LLM triage.
• Do not use generic pattern volume as proof of risk.
• Do not claim thorough codebase understanding from profile reads alone; call it profile-level planning coverage until context packs and triage support stronger claims.
• Do not hide missing gates; name them.
• Do not claim complete coverage unless every context pack is covered by source_packs or pack_reviews.
• Do not shrink exploration just to make validation pass; untriaged packs are allowed as visible recall backlog.
• Do not treat "not package-ready" as "not a lead"; preserve code-anchored candidates.
• Do not treat a low triage score as a reason to discard a pack; scores only order attention.
• Do not drop direct-read or incomplete leads solely because they have evidence debt; record the debt and keep exploring.
• Do not keep chasing one family by inertia; use finding families, invariant matrix, and triage queue to reserve attention for new Bug families.
• Do not use direct-read::... pseudo refs as clean package-ready evidence; create an ad-hoc context pack or attach an existing indexed pack.
• Do not mark P1/P2 records clean package-ready without adversarial R2 evidence.
• Do not over-merge variants across repos, endpoints, affected resources, owners, or fix boundaries.
• Do not present architecture risk signals as standalone Bugs; every signal must cite readiness record IDs, code anchors, or source packs.
• Do not feed pattern seeds directly into build_context_packs.py; the LLM must convert seeds into repo-specific probes first.
• Do not claim complete process execution unless validate_discovery.py passes and work/progress-ledger.jsonl has phase snapshots.
• Do not run readiness, validation, sufficiency, and next-mode scripts as parallel tool calls. Use scripts/run_discovery_quality_gate.py or sequential commands so one validator warning does not cancel the rest.
• Do not rely on generic coverage_notes to explain unprobed repos. Large or bounded multi-repo runs need per-repo skip reasons in profile_coverage.repos_without_probe_reasons and, if context packs are intentionally omitted, profile_coverage.repos_without_context_pack_reasons.
• Do not write probes made only of broad signatures like def , class , import, ViewSet, APIView, or return. Pair source/sink/state/guard terms or use narrower globs before building packs.
• Do not read historical audit packages as comparison data unless the user explicitly asks. Current source is the authority.
• Do not stay in one discovery mode for more than two consecutive rounds unless it yields a new confirmed/probable family.
• Do not keep using grep/rg as the main action for three consecutive steps; switch to optional local scanner scout when allowed, candidate upgrade, pattern amplification, or deep-flow analysis.
• Do not call Semgrep or any scanner hits Bugs. Scanner output is only anchors that require LLM data-flow review and R2 before promotion.
• Do not require Semgrep installation, account login, cloud upload, Registry rules, Pro rules, or managed platform features.
• Do not mark aggregate/systematic findings confirmed/probable without representative trigger chains.
• Do not recommend packaging as the primary next action while required deep-flow analysis is incomplete, unless the user explicitly asks to package a partial snapshot.

Read Next

• references/discovery-workflow.md
• references/repo-profile.md
• references/llm-risk-planning.md
• references/probe-schema.md
• references/context-packs.md
• references/proof-bundle.md
• references/architecture-signals.md
• references/triage-readiness.md
• references/evidence-levels.md
• references/risk-taxonomy.md
• references/process-compliance.md
• scripts/summarize_profile_patterns.py when compact profile summaries need regeneration
• scripts/recommend_next_discovery_mode.py after each readiness merge to decide the next mode
• scripts/deep_flow_plan.py before claiming discovery sufficiency
• subskills/package/SKILL.md when packaging is requested