majiayu000 FactCheck Guard — Runtime Claim Verification
Prevents false claims about the codebase. Grep results are hypotheses, not proof. Every assertion about "what the code does" must be verified by reading the actual code.
Why This Exists
An 80% false claim rate was observed when agents trusted grep results without reading files. The pattern repeats: grep finds a string, agent asserts behavior, behavior is wrong. This skill converts the claim-verification guidelines into an actionable protocol.
Confidence Markers (MANDATORY)
Every factual claim about the codebase MUST carry one of these markers:
| Marker | Meaning | Requirements |
|---|---|---|
| [VERIFIED] | Read the file, traced the logic | Must cite file:line |
| [INFERRED] | Based on grep or file name | Must verify before asserting to others |
| [UNCERTAIN] | Not yet checked | Must investigate before making the claim |
Never promote an [INFERRED] claim to a statement of fact. Never let [UNCERTAIN] appear in a response to the user — investigate first.
Verification Protocol
Level 1: Existence Claims
When asserting "X exists in the codebase":
Step 1 — Search (produces [INFERRED])
Grep/Glob for the pattern.
Result: "X appears at path/to/file.ts:42" [INFERRED]
Step 2 — Read the file (required to reach [VERIFIED])
Read path/to/file.ts, locate line 42.
Confirm the match is actual code, not a comment or string literal.
Step 3 — Check context
Is the code in a live code path, or inside a dead branch / disabled block?
Is it imported and actually used?
Does the function name match what you expected, or is it a different concept?
Step 4 — State the claim with citation
"Error handling exists at src/api/handler.ts:67 — try/catch wraps the database call" [VERIFIED]
Level 2: Absence Claims
When asserting "X does NOT exist" — absence claims are the most dangerous. A single missed search term invalidates the entire claim.
Step 1 — Primary search
Grep for the primary term. No results found.
Status: [INFERRED ABSENT] — not yet VERIFIED
Step 2 — Alternative terms (minimum 3)
Different naming conventions, abbreviations, synonyms.
Example: Searching for "rate limiting"
Try: "rateLimit", "rate_limit", "throttle", "throttling", "requestLimit", "limiter"
If any match: investigate that file before concluding
Step 3 — Check likely locations manually
Where would X logically live if it existed?
Read those files directly — grep may miss dynamically constructed patterns.
Example: Checking for auth middleware
Read: src/middleware/, app/middleware.ts, server.ts, app.ts
Even if grep found nothing, scan these files for auth-related logic.
Step 4 — Only then claim VERIFIED ABSENT
"Rate limiting does not exist. Searched: rateLimit, rate_limit, throttle, throttling,
requestLimit, limiter. Checked: src/middleware/index.ts, server.ts, app.ts. None found." [VERIFIED ABSENT]
Level 3: Behavior Claims
When asserting "the system does X" — the most subtle failure mode. Finding a function name does not tell you what the function actually does.
Step 1 — Locate the function
Grep/Glob finds the function name.
Status: [INFERRED] — you know it exists, not what it does
Step 2 — Read the full function body
Read the file. Read the entire function, not just the signature.
Look for: branches, early returns, side effects, what it actually returns.
Step 3 — Trace the logic path
Follow the execution path relevant to your claim.
If the function calls other functions, read those too (one level deep minimum).
Check: what happens in the error path? What are the edge cases?
Step 4 — State behavior with evidence
"processPayment() validates card number via Luhn check (payment.ts:34),
then calls Stripe API (payment.ts:41), and on failure rolls back the DB
transaction (payment.ts:58) — it does NOT charge the card on validation failure." [VERIFIED]
Common False Claim Patterns
| Pattern | Why It Fails | Correct Approach |
|---|---|---|
| grep found "try.*catch" | Could be in comment, string, or test mock | Read the file, confirm it wraps real code |
| grep found nothing | Wrong term, wrong directory, different naming | Try 3+ alternative terms |
| File named auth.ts exists | Doesn't mean auth is implemented | Read the file |
| Type definition has a field | Doesn't mean the field is set or used | Grep for field assignment and usage |
| Function is called somewhere | Doesn't mean it does what the name implies | Read the function body |
| Found in tests directory | Test mocks are not real implementations | Confirm the actual source file |
| grep -L returned files | Those files lack the pattern — but pattern might exist under a different name | Try alternative terms before concluding |
| Found in commented-out block | Not active code | Check if block is enabled |
Two-Pass Audit Protocol
For system audits, security reviews, and architecture reviews:
Pass 1: Generate Hypotheses
Go through each area systematically. Produce hypothesis statements only:
"Input validation might be missing from the /api/upload route" [INFERRED]
"The auth token might not be refreshed on expiry" [INFERRED]
"Database queries might not use parameterized statements" [INFERRED]
Do not state these as facts. Do not include them in the user-facing summary.
Pass 2: Verify Each Hypothesis
For every [INFERRED] hypothesis:
Hypothesis: "Input validation missing from /api/upload" [INFERRED]
Action: Read src/app/api/upload/route.ts
Found: Zod schema validation at line 18, applied before file processing.
Result: "Input validation exists at api/upload/route.ts:18" [VERIFIED]
Hypothesis was wrong.
Hypothesis: "Database queries might not be parameterized" [INFERRED]
Action: Search for raw SQL patterns (string interpolation in queries)
Grep: `db.query(\`.*\${`, `db.execute(\`.*\${`
Read: src/lib/db.ts, src/repositories/*.ts
Found: All queries use prepared statements. One raw query at analytics.ts:89.
Result: "Raw SQL injection risk at src/analytics.ts:89" [VERIFIED]
Only after Pass 2 are claims safe to report.
Quick Checklist
Before making any codebase claim, confirm:
- Have I read the actual file, not just grep results?
- Is the code in a live path (not commented out, not test-only)?
- For absence claims: did I try 3+ alternative search terms?
- For absence claims: did I manually check the likely locations?
- For behavior claims: did I read the full function, not just the name?
- Does my claim include a file:line citation?
- Is my marker accurate: [VERIFIED] / [INFERRED] / [UNCERTAIN]?
Integration Points
claim-verification.md — the rule file provides guidelines; this skill provides the action protocol.
Agents that must use this skill:
- code-reviewer: before asserting "the code lacks X"
- security-reviewer: before asserting "vulnerability exists/doesn't exist"
- architect: before asserting "pattern is/isn't used"
- sleuth: before asserting "root cause is X"
- coroner: before asserting "same bug exists elsewhere"
Hook integration (optional):
After every Grep tool result, the epistemic-reminder hook injects a warning.
That warning is this skill's Level 1 trigger — search results are [INFERRED], not [VERIFIED].
Examples: Wrong vs Right
Existence claim — WRONG
Grep found "authMiddleware" in 3 files.
Claim: "Auth middleware is applied to all routes."
Existence claim — RIGHT
Grep found "authMiddleware" in 3 files.
Read server.ts: middleware applied only to /api/* routes, not /public/*.
Read admin/router.ts: authMiddleware missing from /admin/health endpoint.
Claim: "Auth middleware applied to /api/* (server.ts:23) but missing from
/admin/health (admin/router.ts:8)." [VERIFIED]
Absence claim — WRONG
Grep "rate limit" returned no results.
Claim: "No rate limiting implemented."
Absence claim — RIGHT
Grep "rate limit" → no results.
Grep "rateLimit", "throttle", "limiter", "requestLimit" → no results.
Read: src/middleware/index.ts, server.ts, next.config.js.
Confirmed none contain request throttling logic.
Claim: "No rate limiting found after searching 6 term variants and
reading 3 likely files." [VERIFIED ABSENT]