Craw4llm Eval

craw4llm-eval

Craw4LLM: Efficient Web Crawling for LLM Pretraining — Shi Yu et al. (2025) (arXiv:2502.13347, 2025)

What this evaluates

Evaluates the efficiency and data quality of web crawling strategies for LLM pretraining by measuring downstream model performance after training on crawled or selected documents. It compares graph-connectivity-based, random, and pretraining-influence-based URL scoring methods against an oracle baseline.

Datasets

• ClueWeb22-A (English subset) — total 900000000; splits: simulation_corpus (900000000)

Metrics

• Average performance on 22 core tasks (DCLM evaluation recipe) (primary) — range: other
  • Computes task-specific metrics (e.g., accuracy, F1) for each of the 22 unique core tasks using the DCLM evaluation recipe, then averages the scores across tasks. Exact formulas depend on the individual task benchmarks.

Input / output format

Input: Standard LLM benchmark prompts and instructions for 22 unique core evaluation tasks.

Output: Model-generated text completions for each benchmark task.

Scoring recipe

scores = []
for task in dclm_core_tasks:
    preds = model.generate(prompts[task])
    gold = labels[task]
    scores.append(compute_task_metric(preds, gold))
return mean(scores)

Common pitfalls

• Simulation uses a static historical web graph (ClueWeb22-A) rather than live crawling, ignoring real-time link dynamics and crawl frontier updates.
• Downstream evaluation uses a small 411M-parameter model trained on 32.9B tokens, so performance gains may not generalize to larger-scale pretraining runs.
• The oracle baseline samples 20M documents from the top 10% by score, while crawl baselines are hard-capped at 20M total, making volume-controlled comparisons necessary.

Evidence (verbatim from paper)

To evaluate the pretrained LLMs, we follow the DCLM evaluation recipe, assessing performance on 23 (22 unique) core tasks.

Citation

@misc{shi2025craw4llm,
  title={Craw4LLM: Efficient Web Crawling for LLM Pretraining},
  author={Shi Yu et al. (2025)},
  year={2025},
  note={arXiv:2502.13347}
}

• arXiv: 2502.13347