adu2021 Essential-Web v1.0: 24T tokens of organized web data
Core Concept
Essential-Web v1.0 is a large-scale, taxonomy-annotated web corpus of 24 trillion tokens where each document receives categorical labels across twelve dimensions (topic, format, complexity, quality). A lightweight 0.5B parameter annotation model (EAI-Distill-0.5b) achieves performance within 3% of 32B models. Simple SQL filtering on taxonomy labels yields competitive domain-specific datasets, demonstrating practical utility for efficient data curation at scale.
Architecture Overview
- 12-Category Taxonomy: Covers topic (STEM, code, news, etc.), format (article, code, list, etc.), complexity (beginner, advanced), quality (high, medium, low)
- Efficient Annotation Model: EAI-Distill-0.5b trained for classification, matching large-model performance at 0.5B scale
- Document-Level Labels: Every document in 24T-token corpus receives categorical assignments
- SQL-Based Curation: Query taxonomy labels to construct specialized datasets for specific domains
- Quality Metrics: Annotator agreement validated against large reference models
Implementation
Step 1: Define Taxonomy Schema
Design the 12-category taxonomy for document classification:
import json
class WebTaxonomy:
"""
Defines taxonomy for document annotation across multiple dimensions.
"""
TAXONOMY = {
'topic': {
'STEM': 'Science, technology, engineering, mathematics content',
'CODE': 'Programming and software development',
'NEWS': 'News and current events',
'BUSINESS': 'Business and economics',
'CREATIVE': 'Creative writing and entertainment',
'SOCIAL': 'Social and humanities content',
'OTHER': 'Miscellaneous topics'
},
'format': {
'ARTICLE': 'News or blog article format',
'CODE': 'Source code or code snippet',
'LIST': 'Lists, tables, or structured data',
'DIALOGUE': 'Conversational or question-answer format',
'REFERENCE': 'Reference material or documentation',
'CREATIVE': 'Story, poem, or creative text',
'OTHER': 'Other formats'
},
'complexity': {
'BEGINNER': 'Introductory or basic content',
'INTERMEDIATE': 'Moderate difficulty level',
'ADVANCED': 'Expert or specialized content'
},
'quality': {
'HIGH': 'High-quality, well-written content',
'MEDIUM': 'Reasonable quality content',
'LOW': 'Poor quality or spam-like content'
}
}
@classmethod
def to_prompt(cls):
"""Generate prompt for annotation model"""
prompt = "Classify the document across these dimensions:\n\n"
for dimension, categories in cls.TAXONOMY.items():
prompt += f"{dimension.upper()}:\n"
for cat, desc in categories.items():
prompt += f" - {cat}: {desc}\n"
return prompt
@classmethod
def schema_sql(cls):
"""Generate SQL schema for storing labels"""
return """
CREATE TABLE documents (
doc_id VARCHAR PRIMARY KEY,
content TEXT,
topic VARCHAR,
format VARCHAR,
complexity VARCHAR,
quality VARCHAR,
timestamp DATETIME
);
"""
Step 2: Implement Efficient Annotation Model
Build lightweight classifier matching large-model performance:
import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel
class AnnotationModel(nn.Module):
"""
Efficient 0.5B parameter annotation model.
Smaller than 32B reference but matches performance within 3%.
"""
def __init__(self, base_model='bert-base', hidden_dim=512):
super().__init__()
# Use lightweight base model
self.tokenizer = AutoTokenizer.from_pretrained(base_model)
self.encoder = AutoModel.from_pretrained(base_model)
# Freeze encoder weights for efficiency
for param in self.encoder.parameters():
param.requires_grad = False
# Classification heads
self.topic_head = nn.Sequential(
nn.Linear(self.encoder.config.hidden_size, hidden_dim),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(hidden_dim, 7) # 7 topic categories
)
self.format_head = nn.Sequential(
nn.Linear(self.encoder.config.hidden_size, hidden_dim),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(hidden_dim, 7) # 7 format categories
)
self.complexity_head = nn.Sequential(
nn.Linear(self.encoder.config.hidden_size, hidden_dim),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(hidden_dim, 3) # 3 complexity levels
)
self.quality_head = nn.Sequential(
nn.Linear(self.encoder.config.hidden_size, hidden_dim),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(hidden_dim, 3) # 3 quality levels
)
def forward(self, documents):
"""
Args:
documents: list of document strings
Returns:
predictions: dict of category predictions
"""
# Tokenize
inputs = self.tokenizer(
documents,
max_length=512,
truncation=True,
padding=True,
return_tensors='pt'
)
# Encode
with torch.no_grad():
outputs = self.encoder(**inputs)
pooled = outputs.last_hidden_state[:, 0] # [CLS] token
# Classify across dimensions
topic_logits = self.topic_head(pooled)
format_logits = self.format_head(pooled)
complexity_logits = self.complexity_head(pooled)
quality_logits = self.quality_head(pooled)
return {
'topic': torch.argmax(topic_logits, dim=1),
'format': torch.argmax(format_logits, dim=1),
'complexity': torch.argmax(complexity_logits, dim=1),
'quality': torch.argmax(quality_logits, dim=1)
}
Step 3: Annotate Documents at Scale
Process corpus and assign taxonomy labels:
def annotate_corpus(documents, annotation_model, batch_size=32):
"""
Annotate large document corpus with taxonomy labels.
Args:
documents: list of document strings
annotation_model: AnnotationModel instance
batch_size: batch size for inference
Returns:
annotations: list of label dictionaries
"""
annotations = []
num_batches = (len(documents) + batch_size - 1) // batch_size
annotation_model.eval()
for i in range(num_batches):
start_idx = i * batch_size
end_idx = min((i + 1) * batch_size, len(documents))
batch_docs = documents[start_idx:end_idx]
# Get predictions
predictions = annotation_model(batch_docs)
# Convert to category names
taxonomy = WebTaxonomy.TAXONOMY
for j, doc_idx in enumerate(range(start_idx, end_idx)):
topic_idx = predictions['topic'][j].item()
format_idx = predictions['format'][j].item()
complexity_idx = predictions['complexity'][j].item()
quality_idx = predictions['quality'][j].item()
topic_cats = list(taxonomy['topic'].keys())
format_cats = list(taxonomy['format'].keys())
complexity_cats = list(taxonomy['complexity'].keys())
quality_cats = list(taxonomy['quality'].keys())
annotation = {
'doc_id': doc_idx,
'topic': topic_cats[topic_idx],
'format': format_cats[format_idx],
'complexity': complexity_cats[complexity_idx],
'quality': quality_cats[quality_idx]
}
annotations.append(annotation)
return annotations
Step 4: SQL-Based Dataset Curation
Query taxonomy labels to curate specialized datasets:
import sqlite3
class DatasetCurator:
"""
Curates domain-specific datasets using SQL queries on taxonomy.
"""
def __init__(self, db_path='essential_web.db'):
self.conn = sqlite3.connect(db_path)
self.cursor = self.conn.cursor()
def curate_code_dataset(self, quality_threshold='HIGH'):
"""
Create code-focused dataset.
"""
query = f"""
SELECT doc_id, content FROM documents
WHERE format = 'CODE' AND quality = '{quality_threshold}'
ORDER BY doc_id;
"""
self.cursor.execute(query)
return self.cursor.fetchall()
def curate_stem_dataset(self, min_complexity='INTERMEDIATE',
quality_threshold='HIGH'):
"""
Create STEM-focused dataset with complexity filter.
"""
complexity_order = {'BEGINNER': 0, 'INTERMEDIATE': 1, 'ADVANCED': 2}
min_level = complexity_order[min_complexity]
query = f"""
SELECT doc_id, content FROM documents
WHERE topic = 'STEM'
AND complexity IN ('INTERMEDIATE', 'ADVANCED')
AND quality = '{quality_threshold}'
ORDER BY doc_id;
"""
self.cursor.execute(query)
return self.cursor.fetchall()
def get_stats(self):
"""Get dataset statistics by category"""
stats = {}
for dimension in ['topic', 'format', 'complexity', 'quality']:
query = f"""
SELECT {dimension}, COUNT(*) as count
FROM documents GROUP BY {dimension};
"""
self.cursor.execute(query)
stats[dimension] = dict(self.cursor.fetchall())
return stats
def close(self):
self.conn.close()
Step 5: Validate Annotation Quality
Compare against reference models for agreement:
def validate_annotations(documents, annotations, reference_model):
"""
Validate annotation quality against larger reference model.
Args:
documents: list of document strings
annotations: list of annotation dicts from EAI-Distill-0.5b
reference_model: larger model for validation
Returns:
agreement_scores: dict of agreement rates per dimension
"""
reference_predictions = reference_model(documents)
taxonomy = WebTaxonomy.TAXONOMY
dimensions = ['topic', 'format', 'complexity', 'quality']
agreement_scores = {}
for dim in dimensions:
matches = 0
total = len(documents)
cats = list(taxonomy[dim].keys())
for i in range(total):
annotation_cat = annotations[i][dim]
reference_idx = reference_predictions[dim][i].item()
reference_cat = cats[reference_idx]
if annotation_cat == reference_cat:
matches += 1
agreement = matches / total
agreement_scores[dim] = agreement
# Report overall agreement
avg_agreement = sum(agreement_scores.values()) / len(agreement_scores)
print(f"Average agreement with reference: {avg_agreement:.1%}")
return agreement_scores
Practical Guidance
- Taxonomy Design: Iteratively refine categories based on corpus analysis; ensure coverage of major content types
- Model Selection: Start with BERT-base or DistilBERT as encoder base; experiment with quantization for further efficiency
- Batch Processing: Process in batches of 32-64 documents for GPU efficiency; stream results to database
- Quality Validation: Sample 5-10% of corpus for human agreement validation; target >97% agreement with reference
- Storage: Use SQLite for smaller corpora, PostgreSQL/BigQuery for 24T-token scale
- Filtering Strategies: Start with high-quality documents (quality='HIGH'); gradually expand as needed
- Downstream Use: Share curated datasets on HuggingFace; enable reproducible dataset construction
Reference
Paper: arXiv:2506.14111 Key metrics: EAI-Distill-0.5b achieves 3% difference from Qwen2.5-32B; web code +14.3%, STEM +24.5% improvements Dataset: 24T tokens with 12-category taxonomy labels Related work: Data curation, web scraping, dataset annotation, knowledge distillation