Lecavrdv2 Eval

lecavrdv2-eval

LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset — Li et al. (2023) (arXiv:2310.17609, 2023)

What this evaluates

Probes a model's ability to retrieve relevant Chinese criminal case documents from a large corpus based on legal queries. It specifically tests alignment with multi-dimensional legal relevance criteria, including case characterization, penalty matching, and procedural similarity.

Datasets

• LeCaRDv2 — total 800; splits: train (640), test (160); repo https://github.com/THUIR/LeCaRDv2

Metrics

• Recall@K (primary) — range: [0, 1]
  • Recall@K measures the proportion of queries for which the ground-truth relevant case appears in the top K retrieved results. Formula: Recall@K = (1/|Q|) * Σ_{q∈Q} 1[relevant_doc ∈ top_K(q)].

Input / output format

Input: A legal query (case description) and a fixed candidate corpus of 55,192 criminal case documents.

Output: A ranked list of candidate cases for each query.

Scoring recipe

def compute_recall_at_k(predictions, gold, k):
    hits = 0
    for pred_list, gold_id in zip(predictions, gold):
        if gold_id in pred_list[:k]:
            hits += 1
    return hits / len(gold)

Common pitfalls

• Zero-shot evaluation uses all 800 queries without any training data, whereas fine-tuning strictly uses an 80/20 split per charge (640 train, 160 test).
• Relevance is defined by three legal dimensions (characterization, penalty, procedure), so standard semantic similarity often fails to capture true legal relevance.
• The candidate pool is fixed at 55,192 cases, but metrics are reported only for top-100/200/500/1000, not full-corpus recall.

Evidence (verbatim from paper)

Since we focus on retrieval performance in large corpus, we adopt recall as the evaluation metric.

Citation

@misc{li2023lecavrdv2,
  title={LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2310.17609}
}

• arXiv: 2310.17609