# Data Labeling Strategies

> Use when implementing data labeling workflows and tools.

- Skill: `loopyluci/data-labeling-strategies` (Agent Skill)
- Install (CLI): `npx skillmds@latest add loopyluci/data-labeling-strategies`
- Raw SKILL.md: https://api.skillmd.com/api/skills/loopyluci/data-labeling-strategies/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: LoopyLuci (https://skillmd.com/u/loopyluci)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/loopyluci/data-labeling-strategies

---


# Data Labeling Strategies

Implementing data labeling workflows — from tool selection and annotation guidelines through quality control, inter-annotator agreement, and labeling cost optimization.

## When to Use

- Setting up data labeling for ML projects
- Choosing labeling tools (Label Studio, Supervisely, Scale)
- Defining annotation guidelines for consistency
- Measuring and improving label quality
- Optimizing labeling budget with active learning

## Labeling Process

```python
LABELING_APPROACHES = {
    'in_house': 'Domain experts label data — highest quality, highest cost',
    'outsourced': 'Labeling services (Scale, Labelbox) — moderate quality/cost',
    'crowd': 'Mechanical Turk, Appen — variable quality, lowest cost',
    'synthetic': 'Programmatically generated labels — no human cost, quality depends on generator',
    'weak_supervision': 'Snorkel, labeling functions — noisy labels, heuristic rules',
}

def inter_annotator_agreement(labels1: List[str], labels2: List[str]) -> float:
    """Cohen's Kappa for two annotators."""
    from sklearn.metrics import cohen_kappa_score
    return round(cohen_kappa_score(labels1, labels2), 3)
```

## Verification Checklist

- [ ] Annotation guidelines documented with examples
- [ ] Labeling tool chosen and configured
- [ ] Pilot labeling round (50-100 items) before full scale
- [ ] Inter-annotator agreement measured (target: 0.8+ Cohen's Kappa)
- [ ] Quality assurance process (10% re-label, consensus review)
- [ ] Active learning reduces labeling volume
- [ ] Labeled data versioned and stored with provenance
- [ ] Labeling cost per item tracked