German Legal QA Eval

german-legal-qa-eval

Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law — Bashir et al. (2026) (arXiv:2601.14160, 2026)

What this evaluates

Evaluates large language models' ability to answer German legal questions accurately in both open-ended and multiple-choice formats. It probes factual grounding in noisy, real-world legal documents (LegalMC4) versus clean statutory text (BGB), and tests robustness to distractor information typical of retrieval-augmented generation (RAG) pipelines.

Datasets

• LegalMC4 QA — total 732; splits: test (732)
• BGB QA — total 715; splits: test (715)
• LegalMC4 MCQ — total 1315; splits: test (1315)
• BGB MCQ — total 1845; splits: test (1845)
• ARC (Easy/Challenge) — total ?; splits: test (-1)
• ARC-DE — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• LLM-judged factual correctness (%) (primary) — range: percent
  • Percentage of open-ended questions where an LLM judge (GPT-4.1) rates the model's answer as factually correct relative to the gold reference, following standard LLM-as-a-judge protocols.
• exact accuracy (%) — range: percent
  • Percentage of multiple-choice questions where the model's predicted option exactly matches the gold label.

Input / output format

Input: Open-ended questions grounded in provided legal passages (BGB sections or LegalMC4 documents with noisy context), or multiple-choice questions with a prompt and candidate options.

Output: For open-ended: a free-text answer grounded in the provided context. For multiple-choice: the selected option letter/text.

Scoring recipe

def score(predictions, golds, task_type):
    scores = []
    for pred, gold in zip(predictions, golds):
        if task_type == 'open_qa':
            # LLM judge (GPT-4.1) evaluates factual correctness
            is_correct = llm_judge(prompt=gold['question'], context=gold['context'], answer=pred, reference=gold['answer'])
            scores.append(1 if is_correct else 0)
        elif task_type == 'mcq':
            scores.append(1 if pred == gold['option'] else 0)
    return sum(scores) / len(scores) * 100

Common pitfalls

• LegalMC4 test sets intentionally contain noisy, context-irrelevant passages mimicking RAG retrieval outputs; models must extract relevant facts rather than relying on clean statute text.
• Open-ended QA scoring relies on an LLM judge (GPT-4.1) rather than exact string matching, which can introduce judge bias or inconsistency if not carefully prompted.
• Performance on general benchmarks (ARC, MMLU) is reported as a secondary check for catastrophic forgetting, not as the primary evaluation target.

Evidence (verbatim from paper)

Two sets contain open-ended questions and are scored for factual correctness using an LLM judge (GPT-4.1), following standard LLM-based evaluation protocols (Zheng et al., [2023]; Liu et al., [2023]) which yielded in high correlations with human experts in similar scenarios (Hillebrand et al., [2024]). Two sets contain multiple-choice questions and are evaluated by exact accuracy.

Citation

@misc{bashir2026domainadaptation,
  title={Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law},
  author={Bashir et al. (2026)},
  year={2026},
  note={arXiv:2601.14160}
}

• arXiv: 2601.14160