tiered-data-management-eval
Data Science and Technology Towards AGI Part I: Tiered Data Management — Wang et al. (2026) (arXiv:2602.09003, 2026)
What this evaluates
Evaluates the impact of tiered data management (L1–L3) on model performance across general knowledge, reasoning, math, and code domains. It compares models trained on different data quality tiers and different training strategies (mix vs. tiered) to validate data curation and scheduling efficacy.
Datasets
Metrics
Average benchmark scores (primary) — range: percent
- Arithmetic mean of per-benchmark accuracy scores across a specified domain (English, Chinese, Math, Code) or overall.
Per-benchmark accuracy — range: percent
- Standard accuracy: fraction of correctly answered multiple-choice or generation tasks. For PPL benchmarks, accuracy is derived from log-likelihood over answer choices.
Input / output format
Input: Few-shot prompts (0 to 8 shots) containing questions, code problems, or math reasoning tasks.
Output: Model-generated text, code, or selected answer choices.
Scoring recipe
def compute_accuracy(predictions, golds):
correct = 0
for pred, gold in zip(predictions, golds):
if extract_final_answer(pred) == gold:
correct += 1
return (correct / len(golds)) * 100
Common pitfalls
- PPL-based benchmarks (MMLU, ARC, etc.) evaluate log-likelihood over predefined answer choices, not generation probability.
- Generation benchmarks (BBH, MATH, GSM8K, MBPP, HumanEval) require robust answer extraction or code execution, which can fail due to minor formatting differences.
- The paper reports domain-specific averages (English, Chinese, Math, Code) and an overall average; mixing these up leads to incorrect comparisons.
Evidence (verbatim from paper)
Across all four domains, downstream performance improves steadily from L1 to L3, demonstrating that data quality increases with each data tier. Specifically, average benchmark scores rise from 52.26 percentage points (pp) to 53.96pp in English (+1.70 pp), 49.44pp to 51.48pp in Chinese (+2.04 pp), 23.78pp to 30.84pp in Math (+7.06 pp), and 34.49pp to 36.28pp in Code (+1.79 pp).
Citation
@misc{wang2026tiereddatamanagement,
title={Data Science and Technology Towards AGI Part I: Tiered Data Management},
author={Wang et al. (2026)},
year={2026},
note={arXiv:2602.09003}
}
1---2name: tiered-data-management-eval3description: Evaluates the impact of tiered data management (L1–L3) on model performance across general knowledge, reasoning, math, and code domains. It compares models trained on different data quality tiers and different training strategies (mix vs. tiered) to validate data curation and scheduling efficacy. Use when the user wants to benchmark on OpenCompass Benchmarks, or asks about evaluating this task. Reports Average benchmark scores.4---56# tiered-data-management-eval78> Data Science and Technology Towards AGI Part I: Tiered Data Management — Wang et al. (2026) (arXiv:2602.09003, 2026)910## What this evaluates1112Evaluates the impact of tiered data management (L1–L3) on model performance across general knowledge, reasoning, math, and code domains. It compares models trained on different data quality tiers and different training strategies (mix vs. tiered) to validate data curation and scheduling efficacy.1314## Datasets1516- **OpenCompass Benchmarks** — total ?; splits: test (-1); repo https://github.com/open-compass/opencompass1718## Metrics1920- `Average benchmark scores` **(primary)** — range: percent21 - Arithmetic mean of per-benchmark accuracy scores across a specified domain (English, Chinese, Math, Code) or overall.22- `Per-benchmark accuracy` — range: percent23 - Standard accuracy: fraction of correctly answered multiple-choice or generation tasks. For PPL benchmarks, accuracy is derived from log-likelihood over answer choices.2425## Input / output format2627**Input**: Few-shot prompts (0 to 8 shots) containing questions, code problems, or math reasoning tasks.2829**Output**: Model-generated text, code, or selected answer choices.3031## Scoring recipe3233```python34def compute_accuracy(predictions, golds):35 correct = 036 for pred, gold in zip(predictions, golds):37 if extract_final_answer(pred) == gold:38 correct += 139 return (correct / len(golds)) * 10040```4142## Common pitfalls4344- PPL-based benchmarks (MMLU, ARC, etc.) evaluate log-likelihood over predefined answer choices, not generation probability.45- Generation benchmarks (BBH, MATH, GSM8K, MBPP, HumanEval) require robust answer extraction or code execution, which can fail due to minor formatting differences.46- The paper reports domain-specific averages (English, Chinese, Math, Code) and an overall average; mixing these up leads to incorrect comparisons.4748## Evidence (verbatim from paper)4950> Across all four domains, downstream performance improves steadily from L1 to L3, demonstrating that data quality increases with each data tier. Specifically, average benchmark scores rise from 52.26 percentage points (pp) to 53.96pp in English (+1.70 pp), 49.44pp to 51.48pp in Chinese (+2.04 pp), 23.78pp to 30.84pp in Math (+7.06 pp), and 34.49pp to 36.28pp in Code (+1.79 pp).5152## Citation5354```bibtex55@misc{wang2026tiereddatamanagement,56 title={Data Science and Technology Towards AGI Part I: Tiered Data Management},57 author={Wang et al. (2026)},58 year={2026},59 note={arXiv:2602.09003}60}61```6263- arXiv: 2602.09003
qhjqhj00