Direct Evidence Score

direct-evidence-score

Quality Estimation of Machine Translated Texts based on Direct Evidence from Training Data — Kumari et al. (2023) (arXiv:2306.15399, 2023)

What this evaluates

Assesses machine translation quality by measuring the proportion of source words that have strong lexical co-occurrence evidence in the training corpus. It evaluates whether data-driven lexical transfer fidelity correlates with standard translation quality metrics like BLEU.

Datasets

• English-Kannada Parallel Corpus — total 4014931; splits: train (4004894), dev (5000), test (5037)
• English-Hindi WMT-2014 — total 2507; splits: test (2507)

Metrics

• DE Score (primary) — range: [0, 100]
  • Percentage of source words (excluding very high-frequency words) that co-occur at least 20 times with any target word in the training corpus. DE Score = (count of words with strong evidence / total source words) * 100.
• BLEU — range: [0, 100]
  • Standard sentence-level BLEU score computed between the machine-translated output and the human reference translation.
• Pearson Correlation Coefficient — range: [-1, 1]
  • Pearson correlation coefficient measuring the linear relationship between DE Scores and BLEU scores across the test set.

Input / output format

Input: Source sentence (SL), machine-translated target sentence (TL), and human reference translation (for BLEU computation). Training corpus used to build the Word Co-Occurrence Matrix (WCM).

Output: DE Score (0–100), BLEU score, and Pearson correlation coefficient.

Scoring recipe

# 1. Build WCM from training corpus
WCM = defaultdict(int)
for src, tgt in training_corpus:
    for w in src.split():
        if 20 <= count(w) <= 10000:
            for t in tgt.split():
                WCM[(w, t)] += 1
strong_pairs = {k for k, v in WCM.items() if v >= 20}

# 2. Compute DE Score per segment
de_scores = []
for src, mt_out in test_data:
    count_strong = sum(1 for w in src.split() if any((w, t) in strong_pairs for t in mt_out.split()))
    de_scores.append((count_strong / len(src.split())) * 100)

# 3. Compute BLEU and Correlation
bleu_scores = [compute_sentence_bleu(mt, ref) for mt, ref in test_data]
pearson_r = pearsonr(de_scores, bleu_scores)

Common pitfalls

• Very high-frequency words (e.g., determiners, prepositions) must be excluded from WCM construction, as they map to many target words and blur lexical evidence.
• Low-frequency words lack statistical co-occurrence data, making DE scores inherently noisy for rare vocabulary.
• DE Score measures lexical transfer fidelity, not fluency or syntax, so correlation with BLEU is moderate (~0.21) and does not imply perfect quality prediction.

Evidence (verbatim from paper)

We take the percentage of words with strong evidence as a score for ranking the translations. We call these scores Direct Evidence (DE) Scores. DE Scores range from 0 to 100. Then for each segment in the test set, we check the number of words (excluding very high frequency words) for which there is strong evidence in the training data. This we do by checking if the SL word co-occurs at least 20 times with any of the TL words in the translated text.

Citation

@misc{kumari2023qualityestimation,
  title={Quality Estimation of Machine Translated Texts based on Direct Evidence from Training Data},
  author={Kumari et al. (2023)},
  year={2023},
  note={arXiv:2306.15399}
}

• arXiv: 2306.15399