Pralekha Eval

pralekha-eval

Pralekha: Cross-Lingual Document Alignment for Indic Languages — Sanjay Suryanarayanan et al. (2024) (arXiv:2411.19096, 2024)

What this evaluates

Evaluates cross-lingual document alignment (CLDA) techniques by measuring chunk/sentence-level alignment accuracy (intrinsic) and the resulting document-level machine translation quality (extrinsic) across English and 11 Indic languages.

Datasets

• Pralekha — total ?; splits: train (-1), val (1000), test (1000); repo https://github.com/AI4Bharat/Pralekha

Metrics

• Precision — range: [0, 1]
  • Ratio of correctly predicted alignment links to all predicted links.
• Recall — range: [0, 1]
  • Ratio of correctly predicted alignment links to all gold alignment links.
• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (P * R) / (P + R).
• DocCOMET (primary) — range: [0, 1]
  • Reference-based neural MT quality metric using the Unbabel/wmt22-comet-da model, extended to incorporate document-level context.
• ChrF — range: percent
  • Character-level F-score based on n-gram overlaps between reference and hypothesis.

Input / output format

Input: Parallel document pairs in English and an Indic language, with 50% unalignable noise documents injected for intrinsic evaluation. For extrinsic evaluation, aligned document pairs are used as training data for fine-tuning LLMs.

Output: Intrinsic: predicted alignment links between text chunks/sentences/documents. Extrinsic: translated document text generated via greedy decoding (max 4,096 tokens).

Scoring recipe

def compute_alignment_metrics(pred_links, gold_links):
    tp = len(pred_links & gold_links)
    fp = len(pred_links - gold_links)
    fn = len(gold_links - pred_links)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

def score_mt(hypothesis, reference):
    comet_score = run_unbabel_wmt22_comet_da(hypothesis, reference)
    chrf_score = compute_chrf_ngram_overlap(hypothesis, reference)
    return comet_score, chrf_score

Common pitfalls

• Omitting the 50% injection of unalignable noise documents from Sangraha Unverified, which artificially inflates precision and misrepresents real-world corpus conditions.
• Using a DAC threshold other than 0.1, which the paper explicitly notes is calibrated to balance precision and recall based on appendix analysis.
• Evaluating extrinsic performance without fine-tuning bilingual models for each specific English↔Indic language pair, as cross-lingual transfer without language-specific tuning degrades results.

Evidence (verbatim from paper)

To assess the intrinsic performance of various document alignment methods, we evaluate three key metrics: precision, recall, and F1 score. ... To evaluate document-level MT performance, we use DocCOMET, an extension of COMET that incorporates document-level context for improved quality assessment. Alongside COMET, we also report ChrF, a character-level F-score metric based on n-gram overlaps.

Citation

@misc{suryanarayanan2024pralekha,
  title={Pralekha: Cross-Lingual Document Alignment for Indic Languages},
  author={Sanjay Suryanarayanan et al. (2024)},
  year={2024},
  note={arXiv:2411.19096}
}

• arXiv: 2411.19096