Milpac Eval

milpac-eval

MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages — Mahapatra et al. (2023) (arXiv:2310.09765, 2023)

What this evaluates

Evaluates the quality of machine translation systems translating English legal text into nine Indian languages. It benchmarks commercial and open-source models against human-translated references to assess domain-specific translation accuracy and metric correlation.

Datasets

• MILPaC — total ?; splits: MILPaC-IP (-1), MILPaC-Acts (-1), MILPaC-CCI-FAQ (1460)

Metrics

• BLEU (primary) — range: [0, 100]
  • BP * exp(sum_{n=1}^4 w_n log p_n), where p_n is clipped n-gram precision (n=1..4) and BP is brevity penalty. Scaled to [0,100].
• GLEU — range: [0, 100]
  • Minimum of n-gram precision and recall for n=1..4. Scaled to [0,100].
• chrF++ — range: [0, 100]
  • Arithmetic mean of character n-gram (order 6) and word n-gram (order 2) F-scores. Scaled to [0,100].

Input / output format

Input: English legal text unit ($x_i$) to be translated to a target Indian language. For LLMs, a one-shot prompt containing an example English-to-target translation pair is prepended.

Output: Translated text in the target Indian language.

Scoring recipe

def compute_metrics(hypothesis, reference, lang):
    hyp_tok = indicnlp_tokenize(hypothesis, lang)
    ref_tok = indicnlp_tokenize(reference, lang)
    p_n = [clipped_precision(hyp_tok, ref_tok, n) for n in range(1, 5)]
    bp = 1.0 if len(hyp_tok) > len(ref_tok) else math.exp(1 - len(ref_tok)/len(hyp_tok))
    bleu = bp * math.exp(sum(0.25 * math.log(p) for p in p_n))
    prec = ngram_precision(hyp_tok, ref_tok, max_n=4)
    rec = ngram_recall(hyp_tok, ref_tok, max_n=4)
    gleu = min(prec, rec)
    chrf = char_word_fscore(hyp_tok, ref_tok, nc=6, nw=2)
    return bleu * 100, gleu * 100, chrf * 100

Common pitfalls

• LLMs (Davinci-003, GPT-3.5T-Inst) have a 4096 token limit, causing the 77 longest text units in MILPaC-CCI-FAQ to be excluded from their evaluation.
• mBART-50 lacks support for Oriya and Punjabi, so it is not evaluated for those languages.
• All metric scores are scaled to [0, 100] for reporting, not the standard [0, 1] range.
• Tokenization must use IndicNLP for Indian languages before computing BLEU/GLEU/chrF++ to ensure reproducibility.

Evidence (verbatim from paper)

We use the following standard metrics to evaluate the performance of MT systems: BLEU: BLEU (Bi-Lingual Evaluation Understudy) is an automatic MT evaluation metric (Papineni et al., 2002). It measures the overlap between the translation generated by an MT system (t_h) and the reference translation (t_r), by considering n-grams based precision (n=1,2,3,4).

Citation

@misc{mahapatra2023milpac,
  title={MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages},
  author={Mahapatra et al. (2023)},
  year={2023},
  note={arXiv:2310.09765}
}

• arXiv: 2310.09765