Hallucination Rate Tracker — Weekly Report
Internal quality-assurance report tracking the rate at which the AI assistant produces legally false content. Hallucination in a legal AI product is not a UX inconvenience — it is a professional-liability risk and the primary reason users stop trusting the product.
Why this matters
A lawyer who acts on a fabricated statute loses in court, or worse, misleads a client. Even a single confirmed hallucination — a case citation that doesn't exist, an article number with the wrong content, a regulatory threshold that is off by an order of magnitude — can end a firm's relationship with the product. Tracking hallucination rate weekly enables:
- Early detection of regression when models are updated
- Identification of which skill categories carry the highest fabrication risk
- Validation that grounding / RAG improvements are working
- Evidence for compliance and due-diligence conversations with enterprise clients
Sampling methodology
Sample size
100 conversations per week, drawn from:
- Production traffic (random sample, anonymized, PII-stripped)
- Synthetic benchmark prompts (the fixed eval set)
Stratification
Ensure the sample is stratified by:
- Skill category: at least 10% from each major category (research, review, draft, kb)
- Jurisdiction: at minimum one slice each for LB, KSA, UAE-onshore, DIFC, ADGM, and one non-MENA
- Model version: if multiple models are in rotation, sample proportionally
Selecting conversations with citation-risk
Not every conversation can produce a hallucination — a purely procedural task has low risk. Pre-filter for conversations that:
- Contain a specific statute or regulation name
- Reference a case or ruling
- State a numerical threshold (penalty amount, limitation period, ownership percentage)
- Name a specific regulatory body and an attributed position
Verification protocol
For each flagged assertion
- Identify the claim: Extract the precise factual assertion (e.g., "Article 14 of UAE Federal Decree-Law No. 33 of 2021 limits non-compete duration to two years").
- Primary-source check: Verify against the canonical authoritative source (official gazette, DIFC Laws portal, BOE, etc.). Do not verify against secondary commentary — only primary sources count.
- Classify the finding:
- Verified correct: assertion matches primary source verbatim or in substance
- Plausible, unverified: claim sounds reasonable but could not be confirmed against primary source (counts as half-hallucination for rate purposes — signals a RAG gap)
- Confirmed hallucination: assertion is demonstrably wrong — wrong article number, nonexistent case, wrong threshold, wrong jurisdiction rule
Double-check method
Every "confirmed hallucination" classification requires:
- Manual expert check (a lawyer familiar with the jurisdiction), AND
- LLM-judge second opinion using the verification prompt:
You are a legal fact-checker. Here is an assertion: [assertion].
Here is the relevant primary source text: [retrieved text].
Does the assertion accurately reflect the source? Answer: ACCURATE / INACCURATE / CANNOT VERIFY.
Explain your reasoning in one sentence.
The LLM-judge result is logged but a human expert has final say on classification.
Output format
Weekly summary header
Week: [ISO week + dates]
Sample: [N conversations, X containing citation-risk assertions]
Model version(s): [list]
Overall hallucination rate: X% (N confirmed / Y assertions checked)
Change vs last week: ▲/▼ X pp
Change vs 4-week avg: ▲/▼ X pp
Rate by skill category
| Skill category |
Assertions checked |
Confirmed hallucinations |
Rate |
| research — statute lookup |
N |
N |
X% |
| research — case law |
N |
N |
X% |
| research — regulation |
N |
N |
X% |
| draft |
N |
N |
X% |
| review |
N |
N |
X% |
| kb / ref |
N |
N |
X% |
| Total |
N |
N |
X% |
Rate by jurisdiction
| Jurisdiction |
Assertions checked |
Confirmed hallucinations |
Rate |
Notable pattern |
| LB |
N |
N |
X% |
|
| KSA |
N |
N |
X% |
|
| UAE-onshore |
N |
N |
X% |
|
| DIFC |
N |
N |
X% |
|
| ADGM |
N |
N |
X% |
|
| EU/FR |
N |
N |
X% |
|
| Other |
N |
N |
X% |
|
Top-5 confirmed hallucinations this week
For each:
- Skill that produced it
- The fabricated assertion (verbatim)
- What the primary source actually says
- Root-cause hypothesis (training cutoff gap? RAG miss? prompt induced fabrication?)
- Remediation action (skill update, KB update, RAG index update, model guardrail)
Alert thresholds
| Metric |
Yellow alert |
Red alert — escalate immediately |
| Overall weekly rate |
> 3% |
> 7% |
| Any single skill rate |
> 5% |
> 12% |
| Any single jurisdiction |
> 8% |
> 15% |
| Week-on-week delta |
> +2 pp |
> +5 pp |
Red-alert triggers a same-day incident review. If the affected skill is in production, consider disabling it until root cause is addressed.
Remediation workflow
- Identify root cause: RAG miss (document not indexed), training cutoff (law changed after training), prompt design flaw (model is encouraged to speculate), or model-specific tendency.
- Fix: Update KB / RAG index → re-test on eval set → deploy if rate improves.
- Verify: Next week's tracker must show rate improvement in the affected category.
- Post-mortem: For any confirmed hallucination that reached a paying user, write a one-paragraph post-mortem and log it in the incident register.
Related skills
- [[report-weekly-ai-quality-trend]]
- [[report-competitor-output-comparison-weekly]]
- [[eval-output-quality]]
- [[ref-vocabulary]]
- [[router-confidence-scorer]]
1---2name: report-hallucination-rate-tracker3description: Use when the product or quality team needs to measure, track, and trend the rate at which the AI assistant generates fabricated legal content — false citations, invented statute numbers, nonexistent cases, or wrong regulatory thresholds — broken down by skill, model, and jurisdiction. This is an internal quality-assurance report. Trigger on the weekly quality-review schedule or whenever a concerning spike in user-reported errors is observed.4license: MIT5metadata:6 id: report.hallucination-rate-tracker7 category: report8 jurisdictions: [__multi__]9 priority: P310 intent: [__internal__, hallucination, quality-assurance, eval, llm-judge]11 related: [report-weekly-ai-quality-trend, report-competitor-output-comparison-weekly, eval-output-quality, ref-vocabulary]12 source: https://skillmd.com/skills/sboghossian-mini-claude-for-legal/report-hallucination-rate-tracker13 upstream: Louis — HAQQ Legal AI (github.com/sboghossian/mini-claude-for-legal)14 version: "1.0"15---1617# Hallucination Rate Tracker — Weekly Report1819Internal quality-assurance report tracking the rate at which the AI assistant produces legally false content. Hallucination in a legal AI product is not a UX inconvenience — it is a professional-liability risk and the primary reason users stop trusting the product.2021## Why this matters2223A lawyer who acts on a fabricated statute loses in court, or worse, misleads a client. Even a single confirmed hallucination — a case citation that doesn't exist, an article number with the wrong content, a regulatory threshold that is off by an order of magnitude — can end a firm's relationship with the product. Tracking hallucination rate weekly enables:2425- Early detection of regression when models are updated26- Identification of which skill categories carry the highest fabrication risk27- Validation that grounding / RAG improvements are working28- Evidence for compliance and due-diligence conversations with enterprise clients2930## Sampling methodology3132### Sample size33100 conversations per week, drawn from:34- Production traffic (random sample, anonymized, PII-stripped)35- Synthetic benchmark prompts (the fixed eval set)3637### Stratification38Ensure the sample is stratified by:39- **Skill category**: at least 10% from each major category (research, review, draft, kb)40- **Jurisdiction**: at minimum one slice each for LB, KSA, UAE-onshore, DIFC, ADGM, and one non-MENA41- **Model version**: if multiple models are in rotation, sample proportionally4243### Selecting conversations with citation-risk44Not every conversation can produce a hallucination — a purely procedural task has low risk. Pre-filter for conversations that:45- Contain a specific statute or regulation name46- Reference a case or ruling47- State a numerical threshold (penalty amount, limitation period, ownership percentage)48- Name a specific regulatory body and an attributed position4950## Verification protocol5152### For each flagged assertion53541. **Identify the claim**: Extract the precise factual assertion (e.g., "Article 14 of UAE Federal Decree-Law No. 33 of 2021 limits non-compete duration to two years").552. **Primary-source check**: Verify against the canonical authoritative source (official gazette, DIFC Laws portal, BOE, etc.). Do not verify against secondary commentary — only primary sources count.563. **Classify the finding**:57 - **Verified correct**: assertion matches primary source verbatim or in substance58 - **Plausible, unverified**: claim sounds reasonable but could not be confirmed against primary source (counts as half-hallucination for rate purposes — signals a RAG gap)59 - **Confirmed hallucination**: assertion is demonstrably wrong — wrong article number, nonexistent case, wrong threshold, wrong jurisdiction rule6061### Double-check method62Every "confirmed hallucination" classification requires:63- Manual expert check (a lawyer familiar with the jurisdiction), AND64- LLM-judge second opinion using the verification prompt:6566```67You are a legal fact-checker. Here is an assertion: [assertion]. 68Here is the relevant primary source text: [retrieved text].69Does the assertion accurately reflect the source? Answer: ACCURATE / INACCURATE / CANNOT VERIFY. 70Explain your reasoning in one sentence.71```7273The LLM-judge result is logged but a human expert has final say on classification.7475## Output format7677### Weekly summary header78```79Week: [ISO week + dates]80Sample: [N conversations, X containing citation-risk assertions]81Model version(s): [list]8283Overall hallucination rate: X% (N confirmed / Y assertions checked)84Change vs last week: ▲/▼ X pp85Change vs 4-week avg: ▲/▼ X pp86```8788### Rate by skill category8990| Skill category | Assertions checked | Confirmed hallucinations | Rate |91|----------------|-------------------|--------------------------|------|92| research — statute lookup | N | N | X% |93| research — case law | N | N | X% |94| research — regulation | N | N | X% |95| draft | N | N | X% |96| review | N | N | X% |97| kb / ref | N | N | X% |98| **Total** | **N** | **N** | **X%** |99100### Rate by jurisdiction101102| Jurisdiction | Assertions checked | Confirmed hallucinations | Rate | Notable pattern |103|---|---|---|---|---|104| LB | N | N | X% | |105| KSA | N | N | X% | |106| UAE-onshore | N | N | X% | |107| DIFC | N | N | X% | |108| ADGM | N | N | X% | |109| EU/FR | N | N | X% | |110| Other | N | N | X% | |111112### Top-5 confirmed hallucinations this week113For each:114- Skill that produced it115- The fabricated assertion (verbatim)116- What the primary source actually says117- Root-cause hypothesis (training cutoff gap? RAG miss? prompt induced fabrication?)118- Remediation action (skill update, KB update, RAG index update, model guardrail)119120## Alert thresholds121122| Metric | Yellow alert | Red alert — escalate immediately |123|--------|-------------|----------------------------------|124| Overall weekly rate | > 3% | > 7% |125| Any single skill rate | > 5% | > 12% |126| Any single jurisdiction | > 8% | > 15% |127| Week-on-week delta | > +2 pp | > +5 pp |128129Red-alert triggers a same-day incident review. If the affected skill is in production, consider disabling it until root cause is addressed.130131## Remediation workflow1321331. **Identify root cause**: RAG miss (document not indexed), training cutoff (law changed after training), prompt design flaw (model is encouraged to speculate), or model-specific tendency.1342. **Fix**: Update KB / RAG index → re-test on eval set → deploy if rate improves.1353. **Verify**: Next week's tracker must show rate improvement in the affected category.1364. **Post-mortem**: For any confirmed hallucination that reached a paying user, write a one-paragraph post-mortem and log it in the incident register.137138## Related skills139140- [[report-weekly-ai-quality-trend]]141- [[report-competitor-output-comparison-weekly]]142- [[eval-output-quality]]143- [[ref-vocabulary]]144- [[router-confidence-scorer]]