qhjqhj00 clothing1mpp-eval
Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond — Liu et al. (2024) (arXiv:2408.11338, 2024)
What this evaluates
Evaluates the robustness of image classification models when trained on datasets with inherent label noise and class imbalance. It probes how effectively learning algorithms can filter mislabeled samples and adapt to skewed class distributions without manual curation.
Datasets
- Clothing1mPP — total 1000000; splits: train (-1), val (-1), test (-1)
Metrics
Noise Rate(primary) — range: percent- Percentage of samples in a subset where human annotators disagree with the original label or express uncertainty, aggregated via majority vote or unanimous agreement.
Top-1 Accuracy— range: [0, 1]- Standard image classification metric: the proportion of test samples where the model's predicted class matches the ground truth label.
Input / output format
Input: RGB images resized to 256x256 pixels with corresponding class labels.
Output: Predicted class label or class probability distribution.
Scoring recipe
def compute_noise_rate(votes):
# votes: list of 3 annotator responses per sample
clean = sum(1 for v in votes if v == ['Yes', 'Yes', 'Yes'])
return (1 - clean / len(votes)) * 100
def compute_top1_accuracy(preds, labels):
correct = (preds == labels).sum().item()
return correct / len(labels)
Common pitfalls
- Confounding label noise with class imbalance: The authors explicitly filter data using Docta and CE to disentangle these two issues before imbalance learning experiments.
- Noise evaluation aggregation: Using majority vote yields 22.15% noise rate, but requiring unanimous agreement retains only 61.25% of samples, significantly affecting downstream learning results.
- Tiny dataset size: The 'tiny' subset used for repeated experiments contains only 50 samples, which may not generalize to full-scale performance.
Evidence (verbatim from paper)
For the label noise evaluation task, we utilized a subset of 20,000 samples from the Clothing-ADC dataset, collecting three votes from unique workers for each sample... Using a simple majority vote aggregation, we found that the noise rate in our dataset is 22.15%. However, if a higher level of certainty is required for clean labels, we can apply a more stringent aggregation method, considering more samples as mislabeled.
Citation
@misc{liu2024automaticdatasetconstruction,
title={Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond},
author={Liu et al. (2024)},
year={2024},
note={arXiv:2408.11338}
}
- arXiv: 2408.11338