Hallucination Rate Tracker — Weekly Report
Internal quality-assurance report tracking the rate at which the AI assistant produces legally false content. Hallucination in a legal AI product is not a UX inconvenience — it is a professional-liability risk and the primary reason users stop trusting the product.
Why this matters
A lawyer who acts on a fabricated statute loses in court, or worse, misleads a client. Even a single confirmed hallucination — a case citation that doesn't exist, an article number with the wrong content, a regulatory threshold that is off by an order of magnitude — can end a firm's relationship with the product. Tracking hallucination rate weekly enables:
- Early detection of regression when models are updated
- Identification of which skill categories carry the highest fabrication risk
- Validation that grounding / RAG improvements are working
- Evidence for compliance and due-diligence conversations with enterprise clients
Sampling methodology
Sample size
100 conversations per week, drawn from:
- Production traffic (random sample, anonymized, PII-stripped)
- Synthetic benchmark prompts (the fixed eval set)
Stratification
Ensure the sample is stratified by:
- Skill category: at least 10% from each major category (research, review, draft, kb)
- Jurisdiction: at minimum one slice each for LB, KSA, UAE-onshore, DIFC, ADGM, and one non-MENA
- Model version: if multiple models are in rotation, sample proportionally
Selecting conversations with citation-risk
Not every conversation can produce a hallucination — a purely procedural task has low risk. Pre-filter for conversations that:
- Contain a specific statute or regulation name
- Reference a case or ruling
- State a numerical threshold (penalty amount, limitation period, ownership percentage)
- Name a specific regulatory body and an attributed position
Verification protocol
For each flagged assertion
- Identify the claim: Extract the precise factual assertion (e.g., "Article 14 of UAE Federal Decree-Law No. 33 of 2021 limits non-compete duration to two years").
- Primary-source check: Verify against the canonical authoritative source (official gazette, DIFC Laws portal, BOE, etc.). Do not verify against secondary commentary — only primary sources count.
- Classify the finding:
- Verified correct: assertion matches primary source verbatim or in substance
- Plausible, unverified: claim sounds reasonable but could not be confirmed against primary source (counts as half-hallucination for rate purposes — signals a RAG gap)
- Confirmed hallucination: assertion is demonstrably wrong — wrong article number, nonexistent case, wrong threshold, wrong jurisdiction rule
Double-check method
Every "confirmed hallucination" classification requires:
- Manual expert check (a lawyer familiar with the jurisdiction), AND
- LLM-judge second opinion using the verification prompt:
You are a legal fact-checker. Here is an assertion: [assertion].
Here is the relevant primary source text: [retrieved text].
Does the assertion accurately reflect the source? Answer: ACCURATE / INACCURATE / CANNOT VERIFY.
Explain your reasoning in one sentence.
The LLM-judge result is logged but a human expert has final say on classification.
Output format
Weekly summary header
Week: [ISO week + dates]
Sample: [N conversations, X containing citation-risk assertions]
Model version(s): [list]
Overall hallucination rate: X% (N confirmed / Y assertions checked)
Change vs last week: ▲/▼ X pp
Change vs 4-week avg: ▲/▼ X pp
Rate by skill category
| Skill category |
Assertions checked |
Confirmed hallucinations |
Rate |
| research — statute lookup |
N |
N |
X% |
| research — case law |
N |
N |
X% |
| research — regulation |
N |
N |
X% |
| draft |
N |
N |
X% |
| review |
N |
N |
X% |
| kb / ref |
N |
N |
X% |
| Total |
N |
N |
X% |
Rate by jurisdiction
| Jurisdiction |
Assertions checked |
Confirmed hallucinations |
Rate |
Notable pattern |
| LB |
N |
N |
X% |
|
| KSA |
N |
N |
X% |
|
| UAE-onshore |
N |
N |
X% |
|
| DIFC |
N |
N |
X% |
|
| ADGM |
N |
N |
X% |
|
| EU/FR |
N |
N |
X% |
|
| Other |
N |
N |
X% |
|
Top-5 confirmed hallucinations this week
For each:
- Skill that produced it
- The fabricated assertion (verbatim)
- What the primary source actually says
- Root-cause hypothesis (training cutoff gap? RAG miss? prompt induced fabrication?)
- Remediation action (skill update, KB update, RAG index update, model guardrail)
Alert thresholds
| Metric |
Yellow alert |
Red alert — escalate immediately |
| Overall weekly rate |
> 3% |
> 7% |
| Any single skill rate |
> 5% |
> 12% |
| Any single jurisdiction |
> 8% |
> 15% |
| Week-on-week delta |
> +2 pp |
> +5 pp |
Red-alert triggers a same-day incident review. If the affected skill is in production, consider disabling it until root cause is addressed.
Remediation workflow
- Identify root cause: RAG miss (document not indexed), training cutoff (law changed after training), prompt design flaw (model is encouraged to speculate), or model-specific tendency.
- Fix: Update KB / RAG index → re-test on eval set → deploy if rate improves.
- Verify: Next week's tracker must show rate improvement in the affected category.
- Post-mortem: For any confirmed hallucination that reached a paying user, write a one-paragraph post-mortem and log it in the incident register.
Related skills
- [[report-weekly-ai-quality-trend]]
- [[report-competitor-output-comparison-weekly]]
- [[eval-output-quality]]
- [[ref-vocabulary]]
- [[router-confidence-scorer]]
1---2name: report-hallucination-rate-tracker3description: Use when the product or quality team needs to measure, track, and trend the rate at which the AI assistant generates fabricated legal content — false citations, invented statute numbers, nonexistent cases, or wrong regulatory thresholds — broken down by skill, model, and jurisdiction. This is an internal quality-assurance report. Trigger on the weekly quality-review schedule or whenever a concerning spike in user-reported errors is observed.4license: MIT5---67# Hallucination Rate Tracker — Weekly Report89Internal quality-assurance report tracking the rate at which the AI assistant produces legally false content. Hallucination in a legal AI product is not a UX inconvenience — it is a professional-liability risk and the primary reason users stop trusting the product.1011## Why this matters1213A lawyer who acts on a fabricated statute loses in court, or worse, misleads a client. Even a single confirmed hallucination — a case citation that doesn't exist, an article number with the wrong content, a regulatory threshold that is off by an order of magnitude — can end a firm's relationship with the product. Tracking hallucination rate weekly enables:1415- Early detection of regression when models are updated16- Identification of which skill categories carry the highest fabrication risk17- Validation that grounding / RAG improvements are working18- Evidence for compliance and due-diligence conversations with enterprise clients1920## Sampling methodology2122### Sample size23100 conversations per week, drawn from:24- Production traffic (random sample, anonymized, PII-stripped)25- Synthetic benchmark prompts (the fixed eval set)2627### Stratification28Ensure the sample is stratified by:29- **Skill category**: at least 10% from each major category (research, review, draft, kb)30- **Jurisdiction**: at minimum one slice each for LB, KSA, UAE-onshore, DIFC, ADGM, and one non-MENA31- **Model version**: if multiple models are in rotation, sample proportionally3233### Selecting conversations with citation-risk34Not every conversation can produce a hallucination — a purely procedural task has low risk. Pre-filter for conversations that:35- Contain a specific statute or regulation name36- Reference a case or ruling37- State a numerical threshold (penalty amount, limitation period, ownership percentage)38- Name a specific regulatory body and an attributed position3940## Verification protocol4142### For each flagged assertion43441. **Identify the claim**: Extract the precise factual assertion (e.g., "Article 14 of UAE Federal Decree-Law No. 33 of 2021 limits non-compete duration to two years").452. **Primary-source check**: Verify against the canonical authoritative source (official gazette, DIFC Laws portal, BOE, etc.). Do not verify against secondary commentary — only primary sources count.463. **Classify the finding**:47 - **Verified correct**: assertion matches primary source verbatim or in substance48 - **Plausible, unverified**: claim sounds reasonable but could not be confirmed against primary source (counts as half-hallucination for rate purposes — signals a RAG gap)49 - **Confirmed hallucination**: assertion is demonstrably wrong — wrong article number, nonexistent case, wrong threshold, wrong jurisdiction rule5051### Double-check method52Every "confirmed hallucination" classification requires:53- Manual expert check (a lawyer familiar with the jurisdiction), AND54- LLM-judge second opinion using the verification prompt:5556```57You are a legal fact-checker. Here is an assertion: [assertion]. 58Here is the relevant primary source text: [retrieved text].59Does the assertion accurately reflect the source? Answer: ACCURATE / INACCURATE / CANNOT VERIFY. 60Explain your reasoning in one sentence.61```6263The LLM-judge result is logged but a human expert has final say on classification.6465## Output format6667### Weekly summary header68```69Week: [ISO week + dates]70Sample: [N conversations, X containing citation-risk assertions]71Model version(s): [list]7273Overall hallucination rate: X% (N confirmed / Y assertions checked)74Change vs last week: ▲/▼ X pp75Change vs 4-week avg: ▲/▼ X pp76```7778### Rate by skill category7980| Skill category | Assertions checked | Confirmed hallucinations | Rate |81|----------------|-------------------|--------------------------|------|82| research — statute lookup | N | N | X% |83| research — case law | N | N | X% |84| research — regulation | N | N | X% |85| draft | N | N | X% |86| review | N | N | X% |87| kb / ref | N | N | X% |88| **Total** | **N** | **N** | **X%** |8990### Rate by jurisdiction9192| Jurisdiction | Assertions checked | Confirmed hallucinations | Rate | Notable pattern |93|---|---|---|---|---|94| LB | N | N | X% | |95| KSA | N | N | X% | |96| UAE-onshore | N | N | X% | |97| DIFC | N | N | X% | |98| ADGM | N | N | X% | |99| EU/FR | N | N | X% | |100| Other | N | N | X% | |101102### Top-5 confirmed hallucinations this week103For each:104- Skill that produced it105- The fabricated assertion (verbatim)106- What the primary source actually says107- Root-cause hypothesis (training cutoff gap? RAG miss? prompt induced fabrication?)108- Remediation action (skill update, KB update, RAG index update, model guardrail)109110## Alert thresholds111112| Metric | Yellow alert | Red alert — escalate immediately |113|--------|-------------|----------------------------------|114| Overall weekly rate | > 3% | > 7% |115| Any single skill rate | > 5% | > 12% |116| Any single jurisdiction | > 8% | > 15% |117| Week-on-week delta | > +2 pp | > +5 pp |118119Red-alert triggers a same-day incident review. If the affected skill is in production, consider disabling it until root cause is addressed.120121## Remediation workflow1221231. **Identify root cause**: RAG miss (document not indexed), training cutoff (law changed after training), prompt design flaw (model is encouraged to speculate), or model-specific tendency.1242. **Fix**: Update KB / RAG index → re-test on eval set → deploy if rate improves.1253. **Verify**: Next week's tracker must show rate improvement in the affected category.1264. **Post-mortem**: For any confirmed hallucination that reached a paying user, write a one-paragraph post-mortem and log it in the incident register.127128## Related skills129130- [[report-weekly-ai-quality-trend]]131- [[report-competitor-output-comparison-weekly]]132- [[eval-output-quality]]133- [[ref-vocabulary]]134- [[router-confidence-scorer]]
sboghossian