Data Labeling Strategies

Data Labeling Strategies

Implementing data labeling workflows — from tool selection and annotation guidelines through quality control, inter-annotator agreement, and labeling cost optimization.

When to Use

• Setting up data labeling for ML projects
• Choosing labeling tools (Label Studio, Supervisely, Scale)
• Defining annotation guidelines for consistency
• Measuring and improving label quality
• Optimizing labeling budget with active learning

Labeling Process

LABELING_APPROACHES = {
    'in_house': 'Domain experts label data — highest quality, highest cost',
    'outsourced': 'Labeling services (Scale, Labelbox) — moderate quality/cost',
    'crowd': 'Mechanical Turk, Appen — variable quality, lowest cost',
    'synthetic': 'Programmatically generated labels — no human cost, quality depends on generator',
    'weak_supervision': 'Snorkel, labeling functions — noisy labels, heuristic rules',
}

def inter_annotator_agreement(labels1: List[str], labels2: List[str]) -> float:
    """Cohen's Kappa for two annotators."""
    from sklearn.metrics import cohen_kappa_score
    return round(cohen_kappa_score(labels1, labels2), 3)

Verification Checklist

• Annotation guidelines documented with examples
• Labeling tool chosen and configured
• Pilot labeling round (50-100 items) before full scale
• Inter-annotator agreement measured (target: 0.8+ Cohen's Kappa)
• Quality assurance process (10% re-label, consensus review)
• Active learning reduces labeling volume
• Labeled data versioned and stored with provenance
• Labeling cost per item tracked