Pr Audit

PR Review Auditor

Evaluate whether a PR review's claims are justified by evidence and stay within PR scope.

Invocation

/pr-audit                         # Prompts for review text
/pr-audit --with-diff             # Also provide PR diff for verification

Purpose

Find false positives, overstated claims, and non-evidence-based assertions in a PR review. This tool does NOT re-review the PR; it evaluates whether the review's claims are justified and whether the reviewer is asking for work that belongs outside the PR.

Input

Provide:

1. Review to audit (required): The PR review text to audit
2. PR diff (optional): If provided, enables verification of code-specific claims
3. Requirements pack (optional but recommended): If provided, enables scope checks against issue criteria, constraints, and non-goals

Audit Rules

1. Evidence Standard

• If review references specific code behavior, lines, or files, it must be supported by the PR diff
• If no diff provided, mark such claims "Not verifiable" rather than "false" (unless internally inconsistent)

2. False Positive Definition

A "false positive" is any of:

• Claim incorrect given provided evidence
• Claim presented as fact but not supported by evidence
• Overconfident phrasing without proof
• Speculation masquerading as certainty
• Logical error or mismatch between evidence and conclusion

3. Confidence-Language Violations

Flag as [Overstated] unless directly supported by cited code evidence:

• "correctly"
• "validated"
• "applied consistently"
• "high confidence"
• "fundamentally sound"
• "safe"
• "robust"
• "fully prevents"
• "guarantees"

Absence of evidence = classify as [Overstated] or [Unsubstantiated], not [Correct].

4. Fairness

• If claim might be true but isn't supported, label "Unsubstantiated" (not "Incorrect")
• Use precise language: "unverified", "overstated", "speculative", "missing evidence", "internally inconsistent"

5. Actionability

For every false-positive candidate, propose:

1. A tighter, defensible rewrite
2. The exact evidence needed to validate the original claim

6. Scope Discipline

• If a finding asks for work unrelated to the linked issue, PR purpose, touched behavior, or a credible regression from the diff, mark it as out of scope.
• Example: a PR adding a refresh button can justify findings about button behavior, regressions, or stated performance requirements; asking for a new diagnostics page to measure refresh speed is out of scope unless instrumentation was part of the issue/PR.

7. Verification Before Escalation

• Blockers and strong recommendations must be backed by concrete evidence, repro steps, a missing required behavior, or an explicit validation gap tied to scope.
• If the concern is plausible but not verified, downgrade it to a question or follow-up validation task rather than endorsing it as a blocker.

Output Format

<audit_report>
  <audit_summary>
    - [3-6 bullets on: overall reliability, degree of overconfidence, most serious false-positive risks]
  </audit_summary>

  <false_positive_candidates>
    <item>
      <claim>[Exact quote from review]</claim>
      <classification>[Incorrect | Unsubstantiated | Overstated | Speculative | Internally inconsistent]</classification>
      <why>
        - [1-3 bullets explaining the issue]
      </why>
      <evidence_check>[Cite diff snippet or "No diff provided"]</evidence_check>
      <better_phrasing>[Conservative rewrite]</better_phrasing>
      <what_to_ask_for>[Exact file/line/snippet needed to verify]</what_to_ask_for>
    </item>
  </false_positive_candidates>

  <blocker_sanity_check>
    <blocker>
      <claim>[Blocker from review]</claim>
      <truly_blocking>[Yes | No | Unclear]</truly_blocking>
      <scope_status>[In scope | Out of scope | Unclear]</scope_status>
      <verification_status>[Verified | Not verifiable | Contradicted]</verification_status>
      <notes>[Tight, factual notes]</notes>
    </blocker>
  </blocker_sanity_check>

  <scope_violations>
    - [Review finding]: asks for work outside the issue/PR scope
  </scope_violations>

  <verification_gaps>
    - [Review finding]: escalated without concrete evidence or a real repro
  </verification_gaps>

  <missing_context_requests>
    - [Minimal list of PR snippets required to verify disputed claims]
    - [Prioritize blocker claims, scope disputes, and exploit-related claims]
  </missing_context_requests>

  <cleaned_review_snippet>
    [Rewrite of the review's summary section with:
     - No unverifiable certainty
     - No global correctness claims
     - Explicit uncertainty where appropriate]
  </cleaned_review_snippet>
</audit_report>

Process

1. List strongest confidence statements from the review
2. Check each for evidence support
3. Audit each blocker:
   • Is it actually blocking given stated requirements and changed behavior?
   • Is it in scope for this PR, or is it reviewer preference / unrelated follow-up work?
   • Does repro logically demonstrate the issue?
   • Does proposed fix address the stated problem?
4. Check major suggestions and follow-ups:
   • Are they in scope?
   • Are they verified strongly enough for their stated severity?
5. Check for internal contradictions (e.g., "epsilon applied consistently" vs later complaints about epsilon behavior)

Classification Guide

Examples

Example 1: Overstated Claim

Input claim: "Guard conditions: Input/output amounts validated as positive, ratios validated as finite, epsilon threshold (1e-8) applied consistently"

Audit:

• Classification: Overstated
• Why:
  • Multiple factual assertions bundled into one statement
  • "Applied consistently" is a global claim requiring proof across all code paths
• Better phrasing: "I see positive/finiteness checks in the cycle-analysis path shown, but I haven't verified all ratio comparisons use EPS consistently."
• Ask for: "Paste all ratio comparison sites (profit detection + cycle validation) to verify EPS usage."

Example 2: Unsubstantiated Claim

Input claim: "Correctly uses N iterations of Bellman-Ford to detect negative-weight cycles"

Audit:

• Classification: Unsubstantiated
• Why:
  • "Correctly" implies validation of iteration count, initialization, and termination
  • No supporting code cited
• Better phrasing: "Implements a Bellman-Ford-style relaxation loop; iteration count and cycle reconstruction need confirmation."
• Ask for: "findNetPositiveCycle implementation including loop bounds and predecessor handling."

Example 3: Out-of-Scope Review Ask

Input claim: "This refresh-button PR should add a diagnostics page so we can measure refresh speed."

Audit:

• Classification: Unsubstantiated
• Why:
  • The requested diagnostics page is a net-new surface, not a defect in the changed refresh-button behavior
  • Unless the issue/PR explicitly includes instrumentation or perf tooling, this is scope drift
• Better phrasing: "If refresh performance is a stated requirement, request a concrete validation step for the button path; otherwise do not block on unrelated diagnostics UI."
• Ask for: "Linked issue text or PR description showing diagnostics/performance instrumentation is in scope."