Auslaw Citation Eval

auslaw-citation-eval

Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study — Han et al. (2024) (arXiv:2412.06272, 2024)

What this evaluates

Evaluates an LLM's ability to predict the correct legal citation for a given query text. It probes the model's capacity to retrieve or generate accurate references from a large Australian legal corpus, testing both retrieval and generation capabilities in a domain-specific setting.

Datasets

• AusLaw Citation Benchmark — total 55000; splits: train (-1), test (-1)

Metrics

• ACC@1 (primary) — range: percent
  • The percentage of instances where the model's top-1 predicted citation exactly matches the ground-truth citation.
• ACC@5 — range: percent
  • The percentage of instances where the ground-truth citation appears within the model's top-5 predicted citations.

Input / output format

Input: Query text, optionally augmented with RoC (Rules of Court) or RoC Aggregations.

Output: A ranked list of candidate citations (Top-5 or Top-1).

Scoring recipe

def compute_acc_at_k(predictions, gold, k):
    hits = sum(1 for pred in predictions[:k] if pred == gold)
    return (hits / len(gold)) * 100

Common pitfalls

• Performance is heavily skewed by citation frequency; cases cited >100 times achieve ~100% accuracy, while those cited <20 times drop below 40%.
• Index granularity significantly impacts retrieval performance; RoC Aggregations outperform both Full Cases and Catchwords, contrary to initial expectations.
• Domain-specific pre-training alone is insufficient for accurate citation prediction; targeted instruction tuning is required to achieve competitive results.

Evidence (verbatim from paper)

As evaluation metrics, Accuracy@1 and Accuracy@5 are used.

Citation

@misc{han2024auslawcitation,
  title={Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study},
  author={Han et al. (2024)},
  year={2024},
  note={arXiv:2412.06272}
}

• arXiv: 2412.06272