Vlegal Bench Eval

vlegal-bench-eval

VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models — Nguyen Tien Dong et al. (arXiv:2512.14554, 2025)

What this evaluates

Evaluates large language models on Vietnamese legal reasoning within a civil law framework. It probes capabilities ranging from statutory recall and hierarchical navigation to multi-step conflict detection, penalty estimation, and ethical bias analysis.

Datasets

• VLegal-Bench — total 10450; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Proportion of correctly predicted answers out of total instances. Standard exact-match for multiple-choice and classification tasks.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. Reported separately for positive (Y-F1) and negative (N-F1) classes in conflict detection tasks, and overall for extraction-style questions.
• ROUGE-L — range: [0, 1]
  • Longest Common Subsequence score measuring overlap between generated text and reference answers. Used for generation-level tasks like legal opinion drafting and summarization.

Input / output format

Input: Task instructions and input query. For few-shot evaluation, a single task-specific demonstration example is prepended.

Output: Direct final answer, or intermediate chain-of-thought reasoning followed by the final response.

Scoring recipe

def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    f1 = f1_score(golds, predictions, average='macro')
    rouge_l = rouge_l_score(golds, predictions)
    return {'accuracy': acc, 'f1': f1, 'rouge_l': rouge_l}

Common pitfalls

• Middle truncation is applied when inputs exceed context windows, potentially discarding critical information at the beginning or end of legal texts.
• Conflict detection tasks exhibit severe negative-class bias, causing many models to achieve 0.00 Y-F1 by predicting 'no conflict' for all cases.
• Few-shot demonstrations are explicitly drawn from a separate development set to prevent test-set leakage.
• Decoding temperature is fixed to 0 to eliminate sampling variance, which may penalize models that benefit from stochastic generation.

Evidence (verbatim from paper)

We adopt standardized evaluation metrics across tasks: Accuracy, F1 for multiple-choice and extraction-style questions, and ROUGE-L for generation-level tasks.

Citation

@misc{nguyen2025vlegalbench,
  title={VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models},
  author={Nguyen Tien Dong et al.},
  year={2025},
  note={arXiv:2512.14554}
}

• arXiv: 2512.14554