Report Accuracy Auditor

Report Accuracy Auditor

Verify a long AI-generated report claim-by-claim before the user relies on it. The user commissions research from agents (legal/tax/compliance, OSINT, business) and audits the output before acting — accuracy is the whole job, not summary.

Standing user preferences

• Order: issues and errors FIRST, strengths/good points LAST. Never open with praise.
• Neutral tone. No cheerleading, no alarmism. State what checks out and what does not.
• Follow-ups usually come in this sequence: audit → "discard what doesn't help / errors" → cleaned-up corrected report → "what does this mean for me / action items / deadlines". Offer the next step; do not start it unasked.

Workflow

1. Read the entire report before judging anything. For long files: wc -l, grep -n "^## " to map chapters, then read in offset/limit chunks. Do not audit from a partial read.
2. Extract the checkable claims. Hard facts only: numbers, thresholds, dollar amounts, dates, statute/case identifiers, "in force / repealed / pending" statuses, penalty figures, named entities. Skip opinions, recommendations, and framing.
3. Verify in batches. Fire up to ~4 focused web_search queries per call, one per claim cluster. Prefer primary sources: statute/regulation text (eCFR, Cornell LII, official codes), agency pages (IRS, FTB, FinCEN...), court dockets/opinions. If an agency's own marketing page contradicts the current regulation text, trust the regulation text and note the discrepancy.
4. Classify every claim into exactly three buckets:
   • Confirmed — with the confirming source named.
   • Error — quote the report's claim, give the correct value, cite the source, and note how many times the error repeats in the document.
   • Unverifiable — say so plainly. Zero search results means unverifiable, never "wrong" and never laundered into "confirmed".
5. Check the document machinery, not just facts:
   • Citation architecture: in-text footnote markers must map to the reference list. Per-chapter marker numbering ([^1^] restarting each chapter) against one global reference list = broken traceability — flag it.
   • Internal consistency: the same fact stated differently across chapters.
   • Magnitude language: check words like "eight-figure" against the actual number (observed 2026-07-24: an $8.5M verdict repeatedly called "eight-figure" — it is seven-figure).
   • Reference captions: statute captions in the reference list must match what the in-text usage cites them for.
6. Deliver the audit: errors/issues first (each with quote → correction → source), then unverifiable items, then what is confirmed and genuinely strong, last. Close with a short concrete fix list.
7. Cleaned-up version (only when asked): regenerate the report with errors corrected, unverifiable claims removed or flagged, and citation machinery repaired — preserve the original structure and disclaimers. Then, when asked, extract the user's personal deadlines, obligations, and action items.

Pitfalls

• Do not fact-check from memory. Thresholds, penalty amounts, and case status shift yearly — every hard claim gets a fresh search, even ones that "look right".
• Do not accept the report's own citations as proof. Reports cite secondary sources (law-firm blogs, vendors); confirm the underlying fact via primary or reputable coverage before marking Confirmed.
• Do not expand scope. Audit the report as delivered; new research directions are a separate task the user will ask for explicitly.
• Do not bury errors inside a balanced narrative. A material error repeated three times in the source is one finding, stated once, with its locations.
• Respect read-only framing: "analyze / accuracy check / break it down" means zero file writes until the user asks for the cleaned-up version.