Review Fleet

Review Fleet

Dispatches 10 parallel finder agents against a PR diff, runs verifiers on candidates, deduplicates by cross-angle agreement, and posts a single structured comment.

Pre-flight

# Get PR diff file list and patch
PR_NUMBER=<number>
gh pr diff $PR_NUMBER --name-only > /tmp/fleet-files.txt
gh pr diff $PR_NUMBER > /tmp/fleet-diff.patch
cat /tmp/fleet-files.txt

If --dry-run is passed: print the agent prompts and file list, then stop. Do not dispatch agents.

Phase 1 — Parallel Finder Agents

Dispatch all 10 angles at once with Agent tool calls in a single message. Each agent:

Required in every agent prompt:

• The exact file list from cat /tmp/fleet-files.txt
• Instruction: "Use Read on these paths only. Do NOT Glob for other files."
• The angle-specific focus (see below)
• Output format: JSON array of findings

10 angles:

Finder agent output format (each agent returns this JSON):

[
  {
    "angle": "security",
    "file": "autobot-backend/api/analytics_quality.py",
    "line": 142,
    "severity": "BLOCKER|HIGH|MEDIUM",
    "title": "Short description",
    "evidence": "Exact quote from the diff or file showing the issue"
  }
]

If no findings: return []. Do not fabricate. Do not include speculative issues.

Phase 2 — Verifier Agents

For each finding from Phase 1, dispatch a verifier agent (batch to max 5 parallel):

Verifier prompt must include:

• The specific finding (file, line, evidence)
• The file list from /tmp/fleet-files.txt
• Instruction: "Read the actual file at the given line. Confirm this finding exists in HEAD code, not just the diff context. Return JSON with confirmed: true/false and confidence: 0.0-1.0."

Verifier output format:

{
  "confirmed": true,
  "confidence": 0.92,
  "note": "Optional clarification"
}

Drop any finding where confirmed: false OR confidence < 0.7.

Phase 3 — Deduplicate and Rank

1. Deduplicate by file:line — if multiple angles flagged the same location, merge into one finding, listing all angles: "angles": ["security", "correctness"]
2. Rank by cross-angle agreement — findings confirmed by 2+ angles rank highest
3. Within tier: rank BLOCKER > HIGH > MEDIUM

Phase 4 — Ownership Check and Post

# Confirm the target issue is ours before posting
ISSUE_ID=<issue-or-tracking-id>
gh issue view $ISSUE_ID --json assignees -q '.assignees[].login'

If not assigned to this agent: post to the PR comment directly instead of the issue.

Post a single PR comment:

gh pr comment $PR_NUMBER --body "$(cat <<'EOF'
## Review Fleet — PR #<number>

**Angles run:** 10 | **Candidates:** <N> | **Confirmed:** <M>

### BLOCKERS
<!-- Only if any -->
- **[security + correctness]** `file.py:142` — Title
  > Evidence quote

### HIGH
- **[security]** `file.py:88` — Title
  > Evidence quote

### MEDIUM
- **[observability]** `file.py:201` — Title
  > Evidence quote

---
*Review Fleet: 10 finder angles + verifier pass. Unconfirmed findings dropped.*
EOF
)"

If zero confirmed findings: post "Review Fleet found no confirmed issues across 10 angles."

Wiring into Paperclip Wake Payload

When a wake payload contains {"type": "pr_review", "effort": "high"} or {"type": "pr_review", "fleet": true}, trigger this skill automatically by including /review-fleet <PR_NUMBER> in the agent prompt.

Regression Test

To verify the fleet catches known bugs, run against a PR known to have issues:

# Pick a PR with known IDOR or privilege escalation finding from history
KNOWN_BUGGY_PR=<pr-number-from-history>
/review-fleet $KNOWN_BUGGY_PR --dry-run   # Verify prompts look right
/review-fleet $KNOWN_BUGGY_PR             # Run and confirm finding appears in output