Overview
LLMs hallucinate — they generate plausible-sounding but factually incorrect content with high confidence. In production systems, hallucinations cause incorrect medical advice, broken code, wrong legal citations, fabricated API calls, and erroneous financial data.
This skill builds hallucination detection and mitigation into your AI pipeline architecture — before output reaches users or downstream systems.
When to Use
- Any AI pipeline where factual accuracy matters
- When agents generate code, SQL, or API calls that will be executed
- When agents retrieve information that will influence decisions
- RAG (Retrieval-Augmented Generation) systems
- Agentic pipelines with tool use and multi-step reasoning
Process
Step 1: Classify Your Hallucination Risk
- Map all AI outputs in your system. For each, classify:
- Critical: Incorrect output causes harm (medical, legal, financial, code execution)
- High: Incorrect output wastes significant user/system resources
- Medium: Incorrect output is annoying but recoverable
- Low: Incorrect output is cosmetic
- Apply this skill's full rigor to Critical and High outputs. Lighter checks for Medium and Low.
Verify: Every AI output in your pipeline has a hallucination risk classification.
Step 2: Ground LLM Outputs
- For factual queries: always provide source documents in the prompt (RAG pattern). Instruct the model to cite sources and refuse to answer if the information isn't in the provided documents.
- For code generation: run the code in a sandbox and verify it produces the expected output before returning to the user.
- For structured outputs (JSON, SQL, etc.): validate against a schema before using.
- For numerical claims: require the model to show its work step-by-step and verify key calculations independently.
Verify: Every Critical/High output is grounded in provided sources or verified by execution.
Step 3: Build Verification Layers
- Self-consistency checking: Ask the model the same question 2–3 ways and compare answers. Divergent answers signal uncertain ground.
- Chain-of-thought with verification: Prompt the model to reason step-by-step, then verify the reasoning chain, not just the conclusion.
- Separate generation from verification: Use one prompt to generate, a different prompt to critically evaluate. Never use the same model call for both.
- Human-in-the-loop for Critical outputs: Require human review before Critical AI-generated content reaches end users.
Verify: Critical outputs have at least two independent verification mechanisms.
Step 4: Design Fail-Safe Defaults
- When the model is uncertain, it should say so — never present uncertain information as confident fact.
- Prompt pattern: "If you're not sure, say 'I don't know' rather than guessing."
- Confidence scores: Where available, expose confidence thresholds. Route low-confidence outputs to human review.
- Graceful degradation: If the AI cannot answer accurately, show the user relevant source documents instead.
Verify: The system has a defined behavior for when AI confidence is low.
Step 5: Monitor for Hallucinations in Production
- Log all AI inputs and outputs.
- Implement feedback mechanisms (thumbs up/down, explicit corrections).
- Regularly audit a sample of outputs for accuracy.
- Track and alert on: sudden changes in output patterns, high user correction rates, model responses contradicting grounded sources.
Verify: A feedback mechanism exists and hallucination incidents are tracked.
Common Rationalizations (and Rebuttals)
| Excuse |
Rebuttal |
| "This model is highly accurate" |
All LLMs hallucinate. Accuracy ≠ 100%. Design for the failure case. |
| "Users will catch errors" |
Users often can't distinguish hallucinated content from real content. Don't rely on them. |
| "It's a demo/MVP" |
Hallucinations that reach demos become the baseline expectation. Fix them early. |
| "We'll add verification in v2" |
Hallucinations that cause harm in v1 may not give you a v2. |
Red Flags
- AI-generated code is executed without testing or sandbox verification
- AI-generated facts are displayed without source citations
- No human review for medical, legal, or financial AI outputs
- The same model is used to generate and verify its own output
- No logging or monitoring of AI outputs in production
Verification
References
1---2name: hallucination-prevention3description: Detects and mitigates LLM hallucinations in production pipelines. Validates AI-generated facts, code, and decisions before they reach end users or downstream systems.4---5
6## Overview
7
8LLMs hallucinate — they generate plausible-sounding but factually incorrect content with high confidence. In production systems, hallucinations cause incorrect medical advice, broken code, wrong legal citations, fabricated API calls, and erroneous financial data.
9
10This skill builds hallucination detection and mitigation into your AI pipeline architecture — before output reaches users or downstream systems.
11
12## When to Use
13
14- Any AI pipeline where factual accuracy matters
15- When agents generate code, SQL, or API calls that will be executed
16- When agents retrieve information that will influence decisions
17- RAG (Retrieval-Augmented Generation) systems
18- Agentic pipelines with tool use and multi-step reasoning
19
20## Process
21
22### Step 1: Classify Your Hallucination Risk
23
241. Map all AI outputs in your system. For each, classify:
25 - **Critical**: Incorrect output causes harm (medical, legal, financial, code execution)
26 - **High**: Incorrect output wastes significant user/system resources
27 - **Medium**: Incorrect output is annoying but recoverable
28 - **Low**: Incorrect output is cosmetic
292. Apply this skill's full rigor to Critical and High outputs. Lighter checks for Medium and Low.
30
31**Verify:** Every AI output in your pipeline has a hallucination risk classification.
32
33### Step 2: Ground LLM Outputs
34
353. For factual queries: **always provide source documents** in the prompt (RAG pattern). Instruct the model to cite sources and refuse to answer if the information isn't in the provided documents.
364. For code generation: run the code in a sandbox and verify it produces the expected output before returning to the user.
375. For structured outputs (JSON, SQL, etc.): validate against a schema before using.
386. For numerical claims: require the model to show its work step-by-step and verify key calculations independently.
39
40**Verify:** Every Critical/High output is grounded in provided sources or verified by execution.
41
42### Step 3: Build Verification Layers
43
447. **Self-consistency checking**: Ask the model the same question 2–3 ways and compare answers. Divergent answers signal uncertain ground.
458. **Chain-of-thought with verification**: Prompt the model to reason step-by-step, then verify the reasoning chain, not just the conclusion.
469. **Separate generation from verification**: Use one prompt to generate, a different prompt to critically evaluate. Never use the same model call for both.
4710. **Human-in-the-loop for Critical outputs**: Require human review before Critical AI-generated content reaches end users.
48
49**Verify:** Critical outputs have at least two independent verification mechanisms.
50
51### Step 4: Design Fail-Safe Defaults
52
5311. When the model is uncertain, it should say so — never present uncertain information as confident fact.
54 - Prompt pattern: *"If you're not sure, say 'I don't know' rather than guessing."*
5512. Confidence scores: Where available, expose confidence thresholds. Route low-confidence outputs to human review.
5613. Graceful degradation: If the AI cannot answer accurately, show the user relevant source documents instead.
57
58**Verify:** The system has a defined behavior for when AI confidence is low.
59
60### Step 5: Monitor for Hallucinations in Production
61
6214. Log all AI inputs and outputs.
6315. Implement feedback mechanisms (thumbs up/down, explicit corrections).
6416. Regularly audit a sample of outputs for accuracy.
6517. Track and alert on: sudden changes in output patterns, high user correction rates, model responses contradicting grounded sources.
66
67**Verify:** A feedback mechanism exists and hallucination incidents are tracked.
68
69## Common Rationalizations (and Rebuttals)
70
71| Excuse | Rebuttal |
72|--------|----------|
73| "This model is highly accurate" | All LLMs hallucinate. Accuracy ≠ 100%. Design for the failure case. |
74| "Users will catch errors" | Users often can't distinguish hallucinated content from real content. Don't rely on them. |
75| "It's a demo/MVP" | Hallucinations that reach demos become the baseline expectation. Fix them early. |
76| "We'll add verification in v2" | Hallucinations that cause harm in v1 may not give you a v2. |
77
78## Red Flags
79
80- AI-generated code is executed without testing or sandbox verification
81- AI-generated facts are displayed without source citations
82- No human review for medical, legal, or financial AI outputs
83- The same model is used to generate and verify its own output
84- No logging or monitoring of AI outputs in production
85
86## Verification
87
88- [ ] All AI outputs classified by hallucination risk
89- [ ] Critical/High outputs are grounded in source documents or execution
90- [ ] At least two independent verification mechanisms for Critical outputs
91- [ ] System has a defined low-confidence fallback behavior
92- [ ] Human-in-the-loop for Critical outputs
93- [ ] Production monitoring and feedback mechanism in place
94
95## References
96
97- [prompt-injection-defense skill](../prompt-injection-defense/SKILL.md)
98- [ai-output-validation skill](../ai-output-validation/SKILL.md)
99- [rag-and-memory skill](../rag-and-memory/SKILL.md)
100- [OWASP LLM09: Overreliance](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
gabrielmoreira