# Eval Rubric Legal Soundness

> Use when scoring AI legal output on whether it correctly states the law and applies it correctly to the facts presented. A 0–5 rubric covering rule statement accuracy, application reasoning, citation reliability, jurisdiction fit, and currency of the law. The primary quality rubric and a deployment blocking gate if it drops significantly.

- Skill: `sboghossian-mini-claude-for-legal/eval-rubric-legal-soundness` (Agent Skill)
- Install (CLI): `npx skillmds@latest add sboghossian-mini-claude-for-legal/eval-rubric-legal-soundness`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sboghossian-mini-claude-for-legal/eval-rubric-legal-soundness/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- License: MIT
- Author: sboghossian (https://skillmd.com/u/sboghossian-mini-claude-for-legal)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/sboghossian-mini-claude-for-legal/eval-rubric-legal-soundness

---


# Eval Rubric — Legal Soundness (0–5)

## When to use this

Apply to every legal AI output that makes substantive legal assertions — analysis, advice, research, and drafting. Legal soundness is the primary quality dimension: a well-written response stating wrong law is dangerous; a poorly formatted response stating correct law is at least safe to act on.

This rubric is a **deployment blocking gate**: if the aggregate legal soundness score drops > 3% vs the previous run, [[eval-regression-detector]] blocks the deployment for investigation.

Run via [[eval-llm-as-judge-system-prompt]] using an ensemble of judge models.

## Scoring (0–5)

| Score | Label | Criteria |
|---|---|---|
| **5** | Excellent | All legal propositions correctly stated; citations real and accurate; application reasoning sound and properly tied to the stated facts; counter-arguments or alternative interpretations addressed where they exist; law is current as of the response date |
| **4** | Good | Legal propositions mostly correct; minor citation inaccuracy (formatting issue or slightly wrong article number) or one missed nuance that does not materially affect the conclusion |
| **3** | Acceptable | Substantial correct content with at least one moderate error — a wrong article number, a missed exception to a rule, or an application that reaches the right conclusion by imperfect reasoning |
| **2** | Poor | Significant legal errors or missing an applicable rule that would materially change the advice; would mislead a practitioner who relies on it without independent verification |
| **1** | Very poor | Multiple serious errors; foundational rule stated wrong; wrong law applied to the jurisdiction |
| **0** | Wrong / harmful | Materially incorrect to the point of being dangerous to act on — e.g., stating that a type of clause is enforceable when it is void in the stated jurisdiction; advising that no formality is required when notarization is mandatory |

## Sub-criteria

### Rule statement
Is the cited rule the right one, and is it correctly articulated?
- Is the legal rule identified at the right level of specificity? (Not just "UAE law governs" but the specific statute and principle)
- Is the rule accurately paraphrased? (Not overstated, not understated)
- Are exceptions to the rule mentioned where they are material?
- Is the rule the one that actually applies to the stated facts? (Not a general principle when a specific rule exists)

### Application
Does the analysis correctly apply the rule to the facts?
- Is the application logical? Does it follow from the stated rule?
- Are key counter-considerations addressed? (e.g., "The contract says X, but UAE courts have the power to modify penalty clauses under Article 390 of the Civil Transactions Law")
- Is the conclusion justified by the stated rule and facts?

### Citations
Are the cited authorities real, correctly attributed, and not fabricated?
(Note: deep citation quality analysis is in [[eval-rubric-citation-quality]]; this sub-criterion is a light check to catch obvious hallucinations)

### Jurisdiction fit
Does the answer cover the right jurisdiction?
(Note: deep jurisdiction analysis is in [[eval-rubric-jurisdiction-awareness]]; this sub-criterion catches cases where clearly wrong law is applied)

### Currency
Is the law as stated current as of the response date?
- Laws that have been repealed or significantly amended without acknowledgment score lower.
- A response that cites the old UAE Labour Law (Federal Law No. 8 of 1980) without noting it was replaced by Federal Decree-Law No. 33 of 2021 scores ≤ 3.
- For post-training-cutoff changes: the model is not expected to know, but must say "as of my knowledge cutoff" when there is reasonable risk of change.

## MENA-specific legal soundness checkpoints

The following are common failure points for generic LLMs on MENA legal matters — grade strictly on these:

| Issue | Failure mode |
|---|---|
| EOSG calculation | Wrong formula (21-day vs 30-day, partial vs full for short tenure); ignoring DIFC vs onshore distinction |
| Penalty clauses | Stating they are per se enforceable without noting UAE/Lebanon courts' power to reduce |
| Non-competes | Stating they are easily enforceable in KSA (they are not) |
| Interest | Not flagging Shariah prohibition in KSA; not noting UAE Civil Transactions Law position |
| Company formation | Confusing DIFC, ADGM, onshore, and free-zone rules |
| Property ownership | Not flagging foreign ownership restrictions in UAE onshore / KSA |
| Choice of law | Not noting that UAE Labour Law protections cannot be waived by choice of foreign law for UAE-sited employees |

Outputs that make any of these errors score ≤ 3 on legal soundness regardless of overall quality.

## Use in automated scoring

Inject this rubric definition into [[eval-llm-as-judge-system-prompt]]. Weight it 0.35 (highest weight) in the composite score:

```
composite_score = 0.35 × legal_soundness
               + 0.20 × citation_quality
               + 0.20 × jurisdiction_awareness
               + 0.15 × completeness
               + 0.10 × (binary hallucination gate)
```

## Limits & escalation

A legal soundness score alone does not determine whether an output is safe to act on. Always pair with [[eval-rubric-hallucination-detection]] (existence gate) and [[eval-rubric-jurisdiction-awareness]] (applicability gate). A score of 4/5 on legal soundness from a model that regularly fabricates citations is meaningless without the hallucination gate.

## Related skills

- [[eval-rubric-citation-quality]] — deep citation quality analysis
- [[eval-rubric-jurisdiction-awareness]] — jurisdiction accuracy
- [[eval-rubric-completeness]] — whether all applicable rules were addressed
- [[eval-rubric-hallucination-detection]] — binary fabrication gate
- [[eval-llm-as-judge-system-prompt]] — applies this rubric in the automated pipeline
- [[eval-benchmark-runner]] — runs this rubric across all benchmark datasets