Coliee Task4 Eval

coliee-task4-eval

Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task — Nguyen et al. (2023) (arXiv:2309.05501, 2023)

What this evaluates

Evaluates the ability of large language models to perform legal textual entailment, specifically measuring how model accuracy changes over time based on the year of the Japanese statute law data used.

Datasets

• COLIEE Task 4 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted entailment labels out of the total number of instances. Calculated as correct predictions divided by total predictions.

Input / output format

Input: Pairs of legal texts (premise and hypothesis) in either English or Japanese.

Output: Classification label indicating the entailment relationship (e.g., entailment, contradiction, neutral).

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Performance fluctuates significantly across years due to shifts in legal language and dataset distribution, so single-year results may not generalize.
• Models show divergent performance between English and Japanese versions, heavily influenced by their respective training data distributions.
• Historical Japanese calendar years (Heisei/Reiwa) must be correctly mapped to Gregorian years for temporal analysis.

Evidence (verbatim from paper)

Table I presents a summary of the experimental results for GPT-3.5 and GPT-4 in both English and Japanese across the years. The table highlights each model's accuracy score for the English and Japanese data in the corresponding years.

Citation

@misc{nguyen2023blackbox,
  title={Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task},
  author={Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2309.05501}
}

• arXiv: 2309.05501