Lar Echr Eval

lar-echr-eval

LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rights — Chlapanis et al. (2024) (arXiv:2410.13352, 2024)

What this evaluates

Evaluates an LLM's ability to perform legal argument reasoning by predicting the correct continuation of a court's argument chain. Given case facts and preceding arguments, the model must select the most plausible next argument from multiple options, testing its understanding of legal logic and precedent application.

Datasets

• LAR-ECHR — total 403; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted continuation options (A, B, C, or D) out of the total test instances. Computed as the average over three random seeds.

Input / output format

Input: Facts (case background), Preceding arguments (excerpt of arguments from the case), Continuation options (four possible next arguments labeled A, B, C, D).

Output: Three structured sections: 'Analysis:' (evaluation of each option), 'Explanation:' (reasoning for the choice), 'Answer:' (single letter A, B, C, or D).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        pred_letter = pred.split('Answer:')[1].strip().split()[0]
        if pred_letter == gold:
            correct += 1
    return correct / len(gold_labels)

Common pitfalls

• Models may output the full text of the option instead of just the letter, requiring robust parsing of the 'Answer:' field.
• Using summarized facts instead of complete proceedings significantly impacts performance and may introduce bias, especially for models with smaller context windows.
• The zero-shot CoT prompt requires strict adherence to the three-step format; deviations can break automated answer extraction.

Evidence (verbatim from paper)

We report the average classification accuracy (over the three random seeds) and the standard deviation for each LLM on the test subset of LAR-ECHR. All experiments are performed in a zero-shot setting with a Chain-of-Thought (CoT) prompt we designed for LAR-ECHR (Table 4). The prompt explains the provided input (facts, preceding arguments, continuation options) and the task, asking the LLM to generate an output in three steps... In the last step (‘Answer’), the LLM outputs only the letter (A, B, C, D) of its choice (to facilitate answer collection).

Citation

@misc{chlapanis2024lar_echr,
  title={LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rights},
  author={Chlapanis et al. (2024)},
  year={2024},
  note={arXiv:2410.13352}
}

• arXiv: 2410.13352