Legal AI Index — HAQQ Quarterly Benchmark Whitepaper
What the Legal AI Index is
The HAQQ Legal AI Index is a quarterly publication benchmarking the performance of legal AI systems — including but not limited to Louis — across a standardized test suite of legal tasks. Its goals are:
- Transparency: give legal professionals an objective basis for evaluating AI tools, rather than relying on vendor marketing.
- Accountability: commit HAQQ to public, reproducible measurement of Louis's own performance.
- Research contribution: provide a shared evaluation framework the academic legal-tech community can build on.
- MENA coverage: be one of the first benchmarks to include Arabic-language legal tasks and MENA-jurisdiction legal questions as first-class evaluation dimensions.
Publication cadence
| Edition |
Coverage period |
Target publication |
| Q1 |
January – March |
April |
| Q2 |
April – June |
July |
| Q3 |
July – September |
October |
| Q4 |
October – December |
January (following year) |
Each edition includes: new benchmark results, a methodology note on any changes to the eval suite, jurisdiction coverage updates, and a commentary section on observed trends.
Methodology
Task categories evaluated
| Category |
Description |
Example task |
| Contract analysis |
Identify risk clauses, missing provisions, non-standard terms |
"Find all liability cap deviations in this SPA" |
| Legal drafting |
Generate a compliant first-draft document |
"Draft an NDA under DIFC law" |
| Regulatory QA |
Answer specific regulatory questions accurately |
"What is the CDD threshold under UAE AML law?" |
| Case/statute research |
Retrieve and summarize applicable law |
"Summarize key provisions of KSA Labour Law on overtime" |
| Bilingual tasks |
Arabic legal tasks and AR/EN bilingual document generation |
"Draft an employment contract in Arabic and English" |
| Translation quality |
Evaluate accuracy of Arabic ↔ English legal translation |
Score against a reference translation by a qualified translator |
Jurisdictions in scope
Current edition covers: UAE (onshore + DIFC + ADGM), KSA, Lebanon, Egypt, Qatar (QFC), Bahrain, and secondary coverage of UK, EU, and US for comparative benchmarking.
Scoring dimensions
Each task is scored on:
- Accuracy (0–5): factual and legal correctness.
- Completeness (0–5): does the response cover all required elements?
- Jurisdiction specificity (0–5): does the response reflect the correct jurisdiction's law, not a generic or US-default answer?
- Language quality (0–5): for Arabic tasks, is the output in legal-register Arabic free of colloquialism and translation artifacts?
- Hallucination rate: percentage of responses containing fabricated statute numbers, case citations, or article numbers.
Scoring panel
Results are evaluated by a panel including:
- Qualified lawyers admitted in the relevant jurisdiction.
- Legal linguists for Arabic-language tasks.
- A scoring rubric published alongside the results for reproducibility.
Reproducible eval suite
The test prompts, scoring rubrics, and reference answers (where a ground truth exists) are published in a companion repository. Researchers and practitioners can:
- Run the eval suite against any model.
- Submit results to HAQQ for inclusion in future editions (with methodology disclosure).
- Fork and extend the suite for their own benchmarking.
How to interpret the results
- Scores are task-specific: a model that performs well on US contract analysis may perform poorly on KSA regulatory QA.
- Hallucination rate is the red-line metric: for legal use, a model that fabricates statute numbers is dangerous regardless of its overall accuracy score.
- Jurisdiction coverage gaps: if a jurisdiction is not yet covered, do not infer performance from adjacent jurisdictions.
- The Index benchmarks the AI output layer, not the full product: a well-designed product (good prompting, good retrieval, human-in-the-loop review) may substantially outperform raw model benchmarks. Conversely, a high-scoring model used carelessly may underperform.
How to obtain the whitepaper
Published at https://haqq.ai/research/legal-ai-index. Each edition is available as:
- PDF download (no registration required).
- Interactive results dashboard (registered users).
- Academic pre-print on request.
Related skills
- [[docs-whitepaper-general]]
- [[docs-security-overview]]
- [[docs-roi-calculator]]
1---2name: docs-whitepaper-legal-ai-index3description: Use when a reader asks about HAQQ's Legal AI Index — the quarterly benchmark measuring legal AI performance across jurisdictions, languages, and task types. Covers the methodology, evaluation dimensions, publication cadence (Q1–Q4), and the reproducible eval suite that lets practitioners and researchers validate the results. Useful for technical audiences, press, academic collaborators, and enterprise evaluators comparing AI tools on objective criteria.4license: MIT5---67# Legal AI Index — HAQQ Quarterly Benchmark Whitepaper89## What the Legal AI Index is1011The HAQQ Legal AI Index is a quarterly publication benchmarking the performance of legal AI systems — including but not limited to Louis — across a standardized test suite of legal tasks. Its goals are:12131. **Transparency**: give legal professionals an objective basis for evaluating AI tools, rather than relying on vendor marketing.142. **Accountability**: commit HAQQ to public, reproducible measurement of Louis's own performance.153. **Research contribution**: provide a shared evaluation framework the academic legal-tech community can build on.164. **MENA coverage**: be one of the first benchmarks to include Arabic-language legal tasks and MENA-jurisdiction legal questions as first-class evaluation dimensions.1718## Publication cadence1920| Edition | Coverage period | Target publication |21|---|---|---|22| Q1 | January – March | April |23| Q2 | April – June | July |24| Q3 | July – September | October |25| Q4 | October – December | January (following year) |2627Each edition includes: new benchmark results, a methodology note on any changes to the eval suite, jurisdiction coverage updates, and a commentary section on observed trends.2829## Methodology3031### Task categories evaluated3233| Category | Description | Example task |34|---|---|---|35| **Contract analysis** | Identify risk clauses, missing provisions, non-standard terms | "Find all liability cap deviations in this SPA" |36| **Legal drafting** | Generate a compliant first-draft document | "Draft an NDA under DIFC law" |37| **Regulatory QA** | Answer specific regulatory questions accurately | "What is the CDD threshold under UAE AML law?" |38| **Case/statute research** | Retrieve and summarize applicable law | "Summarize key provisions of KSA Labour Law on overtime" |39| **Bilingual tasks** | Arabic legal tasks and AR/EN bilingual document generation | "Draft an employment contract in Arabic and English" |40| **Translation quality** | Evaluate accuracy of Arabic ↔ English legal translation | Score against a reference translation by a qualified translator |4142### Jurisdictions in scope4344Current edition covers: UAE (onshore + DIFC + ADGM), KSA, Lebanon, Egypt, Qatar (QFC), Bahrain, and secondary coverage of UK, EU, and US for comparative benchmarking.4546### Scoring dimensions4748Each task is scored on:49- **Accuracy** (0–5): factual and legal correctness.50- **Completeness** (0–5): does the response cover all required elements?51- **Jurisdiction specificity** (0–5): does the response reflect the correct jurisdiction's law, not a generic or US-default answer?52- **Language quality** (0–5): for Arabic tasks, is the output in legal-register Arabic free of colloquialism and translation artifacts?53- **Hallucination rate**: percentage of responses containing fabricated statute numbers, case citations, or article numbers.5455### Scoring panel5657Results are evaluated by a panel including:58- Qualified lawyers admitted in the relevant jurisdiction.59- Legal linguists for Arabic-language tasks.60- A scoring rubric published alongside the results for reproducibility.6162### Reproducible eval suite6364The test prompts, scoring rubrics, and reference answers (where a ground truth exists) are published in a companion repository. Researchers and practitioners can:65- Run the eval suite against any model.66- Submit results to HAQQ for inclusion in future editions (with methodology disclosure).67- Fork and extend the suite for their own benchmarking.6869## How to interpret the results7071- **Scores are task-specific**: a model that performs well on US contract analysis may perform poorly on KSA regulatory QA.72- **Hallucination rate is the red-line metric**: for legal use, a model that fabricates statute numbers is dangerous regardless of its overall accuracy score.73- **Jurisdiction coverage gaps**: if a jurisdiction is not yet covered, do not infer performance from adjacent jurisdictions.74- **The Index benchmarks the AI output layer, not the full product**: a well-designed product (good prompting, good retrieval, human-in-the-loop review) may substantially outperform raw model benchmarks. Conversely, a high-scoring model used carelessly may underperform.7576## How to obtain the whitepaper7778Published at `https://haqq.ai/research/legal-ai-index`. Each edition is available as:79- PDF download (no registration required).80- Interactive results dashboard (registered users).81- Academic pre-print on request.8283## Related skills8485- [[docs-whitepaper-general]]86- [[docs-security-overview]]87- [[docs-roi-calculator]]
sboghossian