trac3r00 Claim Audit
Make an answer show its work — or flag exactly where it didn't.
Overview
A model is far likelier to be wrong on a hard factual claim it asserted with no
evidence and no hedge than on one it cited or qualified. claim_audit.py reads
an agent's answer and sorts every sentence into:
- ✓ grounded — carries a citation, URL, quote, or source marker
- ~ hedged — explicitly uncertain ("likely", "roughly", "I'm not sure")
- ⚠ bare — a hard assertion (is/was/founded/numbers/dates) with neither → verify this
- · opinion/meta — recommendation or instruction, not a world-fact
It's a linter, not a fact-checker: it can't tell you a grounded claim is true, but it reliably surfaces the bare assertions worth checking — the ones that hallucinate. It reports a BARE risk ratio and can fail CI when that ratio is too high.
Proven on a mixed answer: it flagged "The capital of Australia is Sydney" (false)
and "founded in 1788" as ⚠ bare, passed a [1]-cited census figure as ✓ grounded,
and let a hedged population estimate through as ~ — exactly the triage you want.
When to use
- A pre-ship self-check: an agent audits its own draft, then goes and grounds or hedges the ⚠ claims before sending.
- CI over a transcript corpus: fail when >X% of checkable claims are bare.
- Reviewing an LLM feature's outputs for where it asserts without support.
Not for: verifying truth (it finds claims to verify), or auditing opinions/plans.
The method
- Capture the answer to a file or pipe it in.
- Audit it.
echo "$ANSWER" | python scripts/claim_audit.py - # or: python scripts/claim_audit.py draft.txt --json - Read the ⚠ bare list. These are hard assertions with no evidence. For each, do one of three things: cite a source, hedge it honestly, or drop it.
- Gate it. In an agent's reply pipeline or CI:
Now an answer that's mostly unsupported assertions gets caught before it ships.python scripts/claim_audit.py draft.txt --fail-over 0.4 # exit!=0 if >40% bare - Re-audit after grounding. Watch the bare count drop to zero (or intentional, justified hedges). The ratio is the scoreboard.
Anti-patterns
- Treating grounded as true. A cited claim can still be wrong; this tool only proves support exists, not support is correct. Verify the citations too.
- Gaming the linter. Sprinkling "likely" on real assertions to dodge ⚠ is lying, not hedging. The point is honesty, not a green check.
- Auditing plans/opinions as facts. Recommendations aren't world-claims; the
tool already buckets them as
·— don't force them to ground. - Zero-bare as the goal. Some bare claims are fine (well-known, low-risk). The goal is awareness and triage, not a mechanical count of zero.
Example
$ echo "Python was released in 1991. The GIL was removed in 2020." \
| python scripts/claim_audit.py - --fail-over 0.4
⚠ [bare ] Python was released in 1991.
⚠ [bare ] The GIL was removed in 2020.
BARE risk = 100% of checkable claims
Verify these before shipping:
⚠ Python was released in 1991.
⚠ The GIL was removed in 2020. # (false — worth catching)
$ echo $?
1