Legal Red Team

⚠️ EXPERIMENTAL — This skill is provided for educational and informational purposes only. It does NOT constitute legal advice. All responsibility for usage rests with the user. Consult qualified legal professionals before acting on any output.

Legal Red Team

Production-ready adversarial verification framework for AI-generated legal content. Covers factual accuracy, citation validation, arithmetic checking, speculation detection, and distribution readiness scoring.

Table of Contents

• Verification Categories
• Tools
• Six-Step Methodology
• Severity Taxonomy
• Quality Score
• Known Hallucination Patterns
• Reference Guides
• Workflows
• Troubleshooting
• Success Criteria
• Scope and Limitations
• Anti-Patterns
• Tool Reference

Verification Categories

Every AI-generated legal document must be checked across 6 categories.

Clarify First

Before the verification, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• Claimed jurisdiction(s) and legal domain — sets which official source each citation is checked against (EUR-Lex vs congress.gov vs legislation.gov.uk); a document mixing jurisdictions is itself a HIGH finding
• Distribution context / audience — internal note vs client-facing vs regulator-facing sets the quality-score gate (4/5) and how strict the disclaimer review must be
• Whether live source verification is available — Step 2 requires checking citations against official sources; if unavailable, every citation is flagged "unverifiable" rather than confirmed

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the report.

Tools

Legal Fact Checker

Scans legal text for verifiable claims and flags potential hallucination patterns.

# Check a legal document
python scripts/legal_fact_checker.py --input document.txt

# Check with JSON output
python scripts/legal_fact_checker.py --input memo.txt --json

# Check inline text
python scripts/legal_fact_checker.py --text "Under GDPR Article 83(5), fines can reach EUR 20 million..."

# Save verification report
python scripts/legal_fact_checker.py --input document.txt --output report.json

Legal Quality Scorer

Scores legal document quality across all 6 verification categories.

# Score a document
python scripts/legal_quality_scorer.py --input document.txt

# Score with JSON output
python scripts/legal_quality_scorer.py --input document.txt --json

# Score with detailed breakdown
python scripts/legal_quality_scorer.py --input document.txt --verbose

# Save quality assessment
python scripts/legal_quality_scorer.py --input document.txt --output assessment.json

Six-Step Methodology

Step 1: Initial Review

Read the entire document with an adversarial mindset. For each claim, ask:

• Is this verifiable?
• Does this sound too specific to be generated without a source?
• Does this sound too confident for an uncertain area?

Mark every factual assertion, citation, date, number, and predictive statement.

Step 2: Source Verification (ALWAYS Web Search)

For every verifiable claim, attempt to verify against official sources.

Rule: If a claim cannot be verified from an official source, flag it. Do not assume accuracy.

Step 3: Arithmetic Verification

Check every calculation, date computation, and numerical claim.

Step 4: Citation Validation

For every legal citation, verify:

Step 5: Speculation Identification

Distinguish fact from opinion, certainty from prediction.

Step 6: Disclaimer Review

Every AI-generated legal document must include:

Severity Taxonomy

Quality Score

Known Hallucination Patterns

AI models exhibit 5 recurring patterns when generating legal content.

See references/hallucination_patterns.md for detailed examples and prevention strategies.

Reference Guides

Workflows

Workflow 1: Full Adversarial Review

1. Run scripts/legal_fact_checker.py on the document.
2. Review flagged items and verify each against official sources.
3. Run scripts/legal_quality_scorer.py for category scores.
4. For each CRITICAL/HIGH finding, document: the error, the correct information, and the source.
5. Produce a verification report with findings by severity.
6. Assign quality score and distribution readiness assessment.
7. Validation: Every verifiable claim checked, score assigned, recommendations provided.

Workflow 2: Quick Citation Check

1. Run scripts/legal_fact_checker.py on the document.
2. Focus on citation extraction results.
3. Verify each extracted citation against official source.
4. Flag any citation that cannot be verified.
5. Validation: All citations verified or flagged.

Workflow 3: Pre-Distribution Gate

1. Run scripts/legal_quality_scorer.py on the final document.
2. Review composite score.
3. If score < 4/5, document must not be distributed.
4. If score >= 4/5, verify CRITICAL count is zero.
5. Confirm all disclaimers are present and adequate.
6. Validation: Quality score >= 4/5, zero CRITICAL issues, disclaimers complete.

Troubleshooting

Success Criteria

Scope & Limitations

In scope: Verifying factual claims in legal text, validating citations, detecting hallucination patterns, scoring document quality, assessing distribution readiness.

Out of scope: Verifying legal conclusions or interpretations, assessing litigation strategy, replacing professional legal review, accessing paid legal databases (Westlaw, LexisNexis).

Disclaimer: This skill provides a structured adversarial verification methodology. It catches common AI errors but cannot guarantee complete accuracy. Professional legal review remains essential for high-stakes documents.

Anti-Patterns

Tool Reference