adu2021 Overview
Challenges conventional wisdom that training data quality depends primarily on correctness.
Core Technique
Distribution Proximity Hypothesis:
# Human traces (H): correct but distribution-mismatched
# Model traces correct (G): correct and distribution-matched
# Model traces incorrect (W): incorrect but distribution-matched
# W outperforms H despite incorrectness
# because distribution proximity enables faster learning
When to Use
Use when: Curating reasoning datasets, SFT training, synthetic data selection.
References
- Distribution alignment vs correctness
- Partial correctness in synthetic data
- Dataset curation guidance