Global AI Regulation Eval

global-ai-regulation-eval

Navigating Global AI Regulation: A Multi-Jurisdictional Retrieval-Augmented Generation System — Ford et al. (2026) (arXiv:2604.25448, 2026)

What this evaluates

Evaluates a Retrieval-Augmented Generation system's ability to ground responses in retrieved legal documents and directly address jurisdiction-specific AI regulation queries. It tests the system's performance on both single-entity and multi-jurisdictional comparison tasks using automatic LLM-based scoring.

Datasets

• Global AI Regulation Test Queries — total 50; splits: test (50)

Metrics

• faithfulness (primary) — range: [0, 1]
  • Proportion of claims in the generated answer that are supported by the retrieved source chunks. Scores range from 0 (fully unsupported) to 1 (all claims grounded in sources).
• answer relevancy — range: [0, 1]
  • Score indicating how well the generated answer directly addresses the original question. Scores range from 0 (completely irrelevant) to 1 (fully addresses the question).

Input / output format

Input: Query question (e.g., jurisdiction-specific AI regulation question) and top-5 retrieved document chunks provided as context.

Output: Generated natural language answer text.

Scoring recipe

# Faithfulness
claims = gpt4_decompose(prediction)
grounded = sum(1 for c in claims if gpt4_verify(c, gold_context))
faithfulness = grounded / len(claims) if claims else 0.0

# Answer Relevancy
relevancy = gpt4_score(prediction, question) # 0-1 scale

Common pitfalls

• One query was excluded from evaluation (n=49 reported) because entity detection failed at retrieval, yielding no context.
• Multi-jurisdictional queries often score lower on relevancy when retrieval is incomplete, even if faithfulness remains high due to explicit limitation reporting.
• Fragmented source chunks (e.g., sentence fragments) can artificially lower faithfulness scores despite factually correct generated answers.

Evidence (verbatim from paper)

We assess performance on two metrics: faithfulness, measuring whether generated answers are grounded in retrieved sources rather than hallucinated, and answer relevancy, measuring whether generated answers directly address the question asked. We evaluate on 50 test queries divided into two categories: single-entity queries (n=25) and multi-jurisdictional comparison queries (n=25).

Citation

@misc{ford2026navigating,
  title={Navigating Global AI Regulation: A Multi-Jurisdictional Retrieval-Augmented Generation System},
  author={Ford et al. (2026)},
  year={2026},
  note={arXiv:2604.25448}
}

• arXiv: 2604.25448