Deduplicating Training Data Makes Language Models Better Arxiv 2107 06499v2

Deduplicating Training Data Makes Language Models Better

One-line decision

Use this skill when you want to apply exact and near-duplicate removal to LLM training data using suffix arrays and MinHash. Avoid it when your dataset is already deduplicated or too small for dedup to matter.

Skill metadata

• Skill type: exact-and-near-deduplication
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Demonstrate that both exact and near-duplicate removal from training data improves LLM quality, using suffix arrays for exact dedup and MinHash for near dedup.

Problem signature

• Modality: text data for LLM training.
• Data state: training corpora with significant exact and near duplication.
• Scale regime: C4 (800GB) and other web-scale corpora.
• Model requirement: Any LLM; dedup is a preprocessing step.

Use when

• You have web-scale text data with duplication.
• You want to improve LLM training efficiency.
• You can afford the compute for deduplication.

Do not use when

• Your data is already deduplicated.
• Your dataset is small and curated.
• You need all data including duplicates.

Required inputs

• training_corpus: Text corpus to deduplicate.
• suffix_array: Suffix array implementation for exact dedup.
• minhash: MinHash implementation for near dedup.

Optional inputs

• dedup_threshold: Jaccard similarity threshold for near-dedup.

Outputs

• deduped_corpus: Corpus with duplicates removed.
• dedup_statistics: Statistics on duplicates found and removed.

Assumptions and prerequisites

• Web-scale corpora contain significant duplication.
• Removing duplicates improves training efficiency and model quality.
• Both exact and near dedup are important.

Procedure

1. Apply exact deduplication Action: Use suffix arrays to find and remove exact duplicate substrings. Why: Exact duplicates waste training compute. Note: See paper for details.
2. Apply near deduplication Action: Use MinHash to find and remove near-duplicate documents. Why: Near-duplicates are also wasteful. Note: See paper for details.
3. Measure dedup impact Action: Train models on deduped vs. non-deduped data and compare. Why: Validates the benefit of deduplication. Note: See paper for details.
4. Analyze memorization Action: Measure whether dedup reduces memorization of training data. Why: Dedup should reduce verbatim memorization. Note: See paper for details.

Parameters to set

• exact_dedup_method — Role: Algorithm for exact dedup. How to set: Suffix arrays for efficiency. Default/range: Suffix array. Effect: Finds all exact duplicate substrings.
• near_dedup_threshold — Role: Jaccard threshold for near-dedup. How to set: 0.8 for moderate dedup. Default/range: 0.8. Effect: Lower threshold removes more documents.
• minhash_params — Role: MinHash signature parameters. How to set: 128 permutations typical. Default/range: 128. Effect: More permutations improve accuracy.

Validation checks

• Dedup should remove a significant fraction of data.
• Model quality should improve or maintain with dedup.
• Memorization should decrease after dedup.

Failure modes

• Aggressive dedup may remove valid repeated content.
• Suffix array computation requires significant memory.
• Near-dedup thresholds need tuning per dataset.

Adaptation notes for VLM training

• Apply both dedup methods to VLM text data.
• Extend to image-text pair dedup using image hashes.
• Combine with SemDeDup for multi-level deduplication.

Implementation notes

• Use the paper's suffix array implementation for efficiency.
• Apply dedup before other filtering steps.
• Monitor dedup rate to calibrate threshold.

Evidence from the paper

• Deduplication removes 1-19% of C4 data depending on method.
• LLMs trained on deduped data achieve better perplexity.
• Dedup significantly reduces verbatim memorization of training data.
• Both exact and near dedup contribute independently to improvement.

Source paper

• Title: Deduplicating Training Data Makes Language Models Better
• Year: 2022
• Venue: ACL
• Paper ID: arxiv-2107.06499v2
• URL: http://arxiv.org/abs/2107.06499v2
• arXiv ID: 2107.06499v2