Langid Web Crawl Eval

langid-web-crawl-eval

Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus — Caswell et al. (2020) (arXiv:2010.14571, 2020)

What this evaluates

Evaluates language identification models on noisy web crawl data to measure their ability to accurately filter in-language sentences for low-resource languages. It probes domain mismatch and class imbalance effects on real-world LangID deployment.

Datasets

• Web Crawl & Held-out Eval Set — total ?; splits: test (-1)

Metrics

• precision (primary) — range: percent
  • Percentage of in-language sentences in the filtered web crawl output, judged by human raters over a sample of 100 sentences per filtering method.
• recall — range: percent
  • Percentage of correctly identified in-language sentences on a held-out evaluation set with ground truth labels.

Input / output format

Input: Raw text snippets (sentences) extracted from a noisy multilingual web crawl.

Output: Binary decision: keep (in-language) or discard (out-of-language).

Scoring recipe

# Precision (human-judged on crawl sample)
human_judgments = [1 if sentence is in-language else 0 for sentence in sample_100_sentences]
precision = sum(human_judgments) / len(human_judgments) * 100

# Recall (on held-out eval set)
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
recall = correct / len(gold_labels) * 100

Common pitfalls

• Precision is not model-computed but human-judged on a small sample (100 sentences) from the crawl.
• Recall is measured on a separate held-out eval set, not the full crawl, because the crawl lacks ground truth labels.

Evidence (verbatim from paper)

For each example language, we report 1. the precision of the crawl (percent of in-language sentences), as judged by human raters over a sample of 100 sentences per filtering method, 2. the recall of this method on our held-out eval sets, and 3. the percentage of the crawl removed by this filtering method. (Keep in mind that, while the precision and % filtered rows are measured on the noisy web crawl, the recall is measured on the held-out eval set.)

Citation

@misc{caswell2020languageid,
  title={Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus},
  author={Caswell et al. (2020)},
  year={2020},
  note={arXiv:2010.14571}
}

• arXiv: 2010.14571