Coliee Task4 Legal QA Eval

coliee-task4-legal-qa-eval

GPTs and Language Barrier: A Cross-Lingual Legal QA Examination — Nguyen et al. (2024) (arXiv:2403.18098, 2024)

What this evaluates

Evaluates large language models' ability to perform legal textual entailment and question answering in monolingual and cross-lingual settings. It probes how well models handle linguistic and structural disparities between English and Japanese legal contexts and questions.

Datasets

• COLIEE Task 4 — total 429; splits: full (429)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted binary answers (Y or N) out of the total number of instances.

Input / output format

Input: Legal context (articles) followed by a question, formatted with language-specific headers.

Output: Binary answer: 'Y' or 'N'.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() in ['Y', 'N'] and pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = correct / len(predictions)

Common pitfalls

• Cross-lingual settings (EN-JA, JA-EN) introduce linguistic and structural mismatches that degrade performance compared to monolingual baselines.
• Models must strictly output 'Y' or 'N' without explanation; verbose outputs will fail exact-match scoring.
• Context and question lengths vary significantly across years, potentially affecting tokenization and model attention.

Evidence (verbatim from paper)

We formatted the input prompt as follows for monolingual prompting: Prompt in English: {context} Question: {question} Answer (Y or N), no explain. In our experiments, we explore different combinations of context and question languages, yielding four distinct settings: English context and English question (EN-EN), Japanese context and Japanese question (JA-JA), and two cross-lingual settings: English context with Japanese question (EN-JA) and Japanese context with English question (JA-EN).

Citation

@misc{nguyen2024gpts,
  title={GPTs and Language Barrier: A Cross-Lingual Legal QA Examination},
  author={Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2403.18098}
}

• arXiv: 2403.18098