# Pralekha Eval

> Evaluates cross-lingual document alignment (CLDA) techniques by measuring chunk/sentence-level alignment accuracy (intrinsic) and the resulting document-level machine translation quality (extrinsic) across English and 11 Indic languages. Use when the user wants to benchmark on Pralekha, or asks about evaluating this task. Reports F1 Score, DocCOMET.

- Skill: `qhjqhj00/pralekha-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/pralekha-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/pralekha-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/pralekha-eval

---


# pralekha-eval

> Pralekha: Cross-Lingual Document Alignment for Indic Languages — Sanjay Suryanarayanan et al. (2024) (arXiv:2411.19096, 2024)

## What this evaluates

Evaluates cross-lingual document alignment (CLDA) techniques by measuring chunk/sentence-level alignment accuracy (intrinsic) and the resulting document-level machine translation quality (extrinsic) across English and 11 Indic languages.

## Datasets

- **Pralekha** — total ?; splits: train (-1), val (1000), test (1000); repo https://github.com/AI4Bharat/Pralekha

## Metrics

- `Precision` — range: [0, 1]
  - Ratio of correctly predicted alignment links to all predicted links.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted alignment links to all gold alignment links.
- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (P * R) / (P + R).
- `DocCOMET` **(primary)** — range: [0, 1]
  - Reference-based neural MT quality metric using the Unbabel/wmt22-comet-da model, extended to incorporate document-level context.
- `ChrF` — range: percent
  - Character-level F-score based on n-gram overlaps between reference and hypothesis.

## Input / output format

**Input**: Parallel document pairs in English and an Indic language, with 50% unalignable noise documents injected for intrinsic evaluation. For extrinsic evaluation, aligned document pairs are used as training data for fine-tuning LLMs.

**Output**: Intrinsic: predicted alignment links between text chunks/sentences/documents. Extrinsic: translated document text generated via greedy decoding (max 4,096 tokens).

## Scoring recipe

```python
def compute_alignment_metrics(pred_links, gold_links):
    tp = len(pred_links & gold_links)
    fp = len(pred_links - gold_links)
    fn = len(gold_links - pred_links)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

def score_mt(hypothesis, reference):
    comet_score = run_unbabel_wmt22_comet_da(hypothesis, reference)
    chrf_score = compute_chrf_ngram_overlap(hypothesis, reference)
    return comet_score, chrf_score
```

## Common pitfalls

- Omitting the 50% injection of unalignable noise documents from Sangraha Unverified, which artificially inflates precision and misrepresents real-world corpus conditions.
- Using a DAC threshold other than 0.1, which the paper explicitly notes is calibrated to balance precision and recall based on appendix analysis.
- Evaluating extrinsic performance without fine-tuning bilingual models for each specific English↔Indic language pair, as cross-lingual transfer without language-specific tuning degrades results.

## Evidence (verbatim from paper)

> To assess the intrinsic performance of various document alignment methods, we evaluate three key metrics: precision, recall, and F1 score. ... To evaluate document-level MT performance, we use DocCOMET, an extension of COMET that incorporates document-level context for improved quality assessment. Alongside COMET, we also report ChrF, a character-level F-score metric based on n-gram overlaps.

## Citation

```bibtex
@misc{suryanarayanan2024pralekha,
  title={Pralekha: Cross-Lingual Document Alignment for Indic Languages},
  author={Sanjay Suryanarayanan et al. (2024)},
  year={2024},
  note={arXiv:2411.19096}
}
```

- arXiv: 2411.19096