Fact Check

Fact Check: Verify Before You Ship

Verify that every claim, reference, and technical detail is backed by current evidence. AI models have training data cutoffs and frequently hallucinate version numbers, API signatures, field names, CLI flags, and dates. This skill provides a systematic process to catch these errors before they reach users.

Why This Exists

AI models commonly produce these types of false information:

The Verification Process

For every piece of output that contains technical claims, run this process:

Step 1: Identify Claims

Scan the output and tag every verifiable claim:

• Version numbers (library, SDK, model, API)
• Field names (API response, config keys, database columns)
• CLI commands and flags
• URLs and endpoints
• Dates and timelines
• Default values and limits
• File paths and function names
• "It supports X" or "X is available" statements

Step 2: Classify Each Claim

Step 3: Verify Using Tools

For each unverified claim, use the appropriate verification method:

Step 4: Label the Output

After verification, every claim in the output should be one of:

• Verified (with evidence: "confirmed by npm info output")
• Unverified (explicitly marked: "not yet confirmed")
• Corrected (original was wrong, replaced with verified data)

Do / Don't Checklist

Do

• Verify every version number mentioned in output with a tool or API call
• Verify URLs you generate with curl -s -o /dev/null -w "%{http_code}" URL before recommending
• Never guess URLs — if you can't see it, ask the user to provide it
• For user-provided URLs: only check HTTP status code (-o /dev/null), don't fetch or render content from untrusted sources (phishing/malware risk)
• Check today's date with date if you reference it
• Run actual API calls to confirm field names (don't guess from memory)
• Use --help or docs to confirm CLI flags exist
• Check if URLs return 200 before recommending them
• Mark claims as "unverified" if you cannot check them
• Date your verification ("verified on 2026-04-11")
• Re-verify if the source information is older than 6 months
• Verify PR/issue status with gh commands, not from memory

Don't

• Don't trust your memory for version numbers, API shapes, or dates
• Don't assume an API field name — always verify from real output
• Don't report "I fixed it" without verifying CI passes (gh pr checks)
• Don't report "PR has reviews" without checking (gh pr view)
• Don't cite documentation without confirming the URL exists
• Don't use old StackOverflow answers as current truth — platforms change
• Don't claim "push succeeded" means "task complete"
• Don't state defaults or limits without checking the actual source code or schema

Common Hallucination Patterns to Watch For

Pattern 1: Outdated Model/Version References

AI models are frozen at their training cutoff. Always verify:

# Check latest version of a package
npm info @anthropic-ai/sdk version
pip show openai | grep Version
gh release list --repo owner/repo --limit 3

Pattern 2: Invented API Fields

Never guess API response field names. Always run the API and read the actual response:

# Real API call to verify field names
curl -s "https://api.example.com/endpoint" | jq '.[0] | keys'

Pattern 3: False Success Reporting

The most dangerous hallucination. "I did it" when the work is incomplete:

Pattern 4: Phantom References

Referencing things that don't exist:

# Verify a file exists before referencing it
ls /path/to/claimed/file

# Verify a function exists in a codebase
grep -r "functionName" src/

# Verify an npm package exists
npm info claimed-package-name

Pattern 5: Date Confusion

AI models often get dates wrong, including today's date:

# Always verify current date
date "+%Y-%m-%d"

# Check when a release was published
gh release view v1.0.0 --repo owner/repo --json publishedAt

Pattern 6: URL Guessing

AI models will fabricate plausible-looking URLs when they don't know the real one:

Prevention:

• Never guess URLs. If you can't see the full output, ask the user to expand it.
• If recommending a URL, verify it with curl -s -o /dev/null -w "%{http_code}" URL first.
• Say "I don't have the URL — can you check the terminal output?" instead of guessing.
• URLs are not inferrable from patterns. auth.openai.com/activate vs auth.openai.com/codex/device — one character path difference, completely different endpoints.

Verification Report Format

When fact-checking, produce a report like this:

## Fact Check Report

| # | Claim | Status | Evidence |
|---|-------|--------|----------|
| 1 | "API returns `like_count`" | Verified | Real API run, dataset abc123 |
| 2 | "Latest version is v2.3" | Corrected → v2.5 | `npm info package version` |
| 3 | "Supports `--recursive` flag" | Unverified | Cannot test without install |
| 4 | "Default timeout is 30s" | Verified | Source code line 42 |

Tips

• The more confident you feel about a claim, the more important it is to verify. Confidence ≠ correctness.
• "I'm pretty sure" is not evidence. Run the command.
• Verification has a cost (time, API calls), but false information has a higher cost (broken code, lost trust, wasted user time).
• When in doubt, say "I'm not sure — let me verify" instead of guessing.
• If you can't verify something, say so explicitly. An honest "unverified" is better than a confident hallucination.