qhjqhj00 ccnet-dataset-eval
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data — Wenzek et al. (2019) (arXiv:1911.00359, 2019)
What this evaluates
Evaluates the quality of a large-scale monolingual web corpus by measuring downstream performance on standard linguistic analogy tasks and a cross-lingual natural language inference benchmark.
Datasets
- CCNet — total ?; splits: head (-1), mid (-1), tail (-1)
- XNLI — total ?; splits: train (-1)
Metrics
XNLI(primary) — range: percent- Percentage of correctly classified sentence pairs on the XNLI training set.
fastText analogy accuracy— range: percent- Percentage of correct answers on standard semantic and syntactic analogy datasets.
Input / output format
Input: For fastText: word triples for analogy tasks. For BERT/XNLI: sentence pairs with natural language inference labels.
Output: For fastText: predicted analogy completion. For BERT/XNLI: predicted entailment/contradiction/neutral label.
Scoring recipe
def compute_xnli_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels) * 100
def compute_analogy_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels) * 100
Common pitfalls
- Perplexity thresholds for splitting the corpus into head/mid/tail are language-specific and set to yield equal-sized partitions, not a fixed global score.
- BERT models are early-stopped after a fixed training time (2 days on 16 GPUs) and step count, not trained to convergence or on equal data volumes across languages.
- XNLI evaluation uses the training split of XNLI rather than the standard development/test split.
Evidence (verbatim from paper)
We assess the quality of the resulting dataset by learning unsupervised word and sentence representations through fastText and BERT models. For fastText, we train 300-dimensional word embeddings on the head, middle and tail subsets of the English and Polish CommonCrawl corpora, sorted by document perplexity. We evaluate these on standard semantic and syntactic analogy datasets (Mikolov et al., 2013). We observe in Table 1 a steady increase in performance as we go from the tail to the head of the dataset, confirming the positive impact of our filtering method based on document perplexity. We also train BERT models on the English (en), Russian (ru), Chinese (zh) and Urdu (ur) languages... We evaluate each model on the XNLI (Conneau et al., 2018) corpus by using the training data in each language. Results presented in Table 2 indicate that BERT-BASE models trained on CommonCrawl outperform identical models trained on Wikipedia by 3.3% on average.
Citation
@misc{wenzek2019ccnet,
title={CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data},
author={Wenzek et al. (2019)},
year={2019},
note={arXiv:1911.00359}
}
- arXiv: 1911.00359