Training Data Annotation
Treat annotation as a measurement system and a human workflow. The objective is
useful, auditable labels for a defined downstream decision, not the largest
possible label count.
Workflow
- Define the downstream task, population, label unit, label set, edge cases,
abstention label, and what constitutes an unusable item. Separate training,
validation, held-out evaluation, and challenge data before sampling.
- Write an annotation guide with positive and negative examples, boundary
rules, source requirements, escalation, and a change log. Have domain owners
approve consequential definitions.
- Establish provenance and rights: source, collection time, license/consent,
transformations, sensitive fields, access, retention/deletion, and dataset
version. Do not move data into a labeling system until this record is clear.
- Acquire items using a declared mix of representative sampling and targeted
acquisition. Uncertainty finds confusing items; diversity and novelty find
gaps; stratification protects rare or high-risk slices. Keep acquisition
batches identifiable and retain a representative held-out audit sample.
- Qualify annotators for the task, train them on calibration items, and measure
quality and workload. Record role and relevant expertise in restricted
metadata. Do not use agreement as a universal truth score.
- Run blinded calibration and periodic blind rechecks. Compare labels with
adjudicated references where available; preserve meaningful disagreement,
ambiguity, and abstention instead of forcing consensus.
- Review disagreement by item, label, slice, annotator, and guide version.
Adjudicate only with a stated rule and qualified authority. Feed confirmed
guide defects back into training and re-review affected labels.
- Stop or continue based on marginal downstream value, coverage, quality,
cost, and time. An item quota may stop spending but cannot by itself establish sufficient quality. Freeze the
manifest and hand off to data-engineering for storage/orchestration and
ml-engineering for training use.
Acquisition guardrails
Never let model confidence define the whole dataset. Sample a representative
held-out audit independently of active-learning selection, and compare active
batch outcomes against it. Track uncertainty method, model/version, seed,
selection scores, slice quotas, and excluded items. If confidence is uncalibrated
or the model is out of distribution, treat its ranking as a search heuristic and
increase diversity, novelty, and human review.
Human and quality guardrails
Design pay, volume, time limits, breaks, and escalation so workers can perform
the task reliably; route employment, compensation, and legal questions to the
appropriate human owner. Blind annotators to model predictions when measuring
independent labels. If preannotations are shown, measure anchoring by a blinded
control or separate review pass. Report agreement by label type and slice with
the uncertainty and prevalence needed to interpret it. A minority label can be
correct when the item is ambiguous; adjudication must explain the rule used.
Routing and exit
Load references/task-and-label-design.md
for schemas and guide design, references/acquisition-and-stopping.md
for sampling and stopping, references/annotator-quality.md
for people and disagreement, and references/provenance-and-rights.md
for dataset authority. Use the templates when a durable record is needed.
For primary methodology sources and transfer limits, read
references/source-index.md.
Route statistical estimators and power to data-scientist,
storage and pipelines to data-engineering, model
training and leakage checks to ml-engineering, and
labeling-interface interaction design to product-design-and-ux.
When not to use
Do not use this skill as the primary owner for statistical inference or
experimental design, data platforms, model training, or UI implementation.
Those neighboring skills consume this skill's annotation manifest and quality
evidence.
Templates
- annotation guide — use when recording this stage of the annotation workflow.
- annotation plan — use when recording this stage of the annotation workflow.
- quality report — use when recording this stage of the annotation workflow.
1---2name: training-data-annotation3description: >-4 Design and improve human annotation programs for machine-learning training,5 validation, and evaluation data, including task and label schemas, active6 acquisition, annotator guidance, calibration, disagreement, adjudication,7 provenance, and cost-quality stopping. Do not use for general statistical8 inference, data storage pipelines, model training, or annotation-interface9 implementation; route those concerns to data-scientist, data-engineering,10 ml-engineering, or product-design-and-ux.11license: MIT12compatibility: Reference-only methodology; no runtime dependency.13metadata:14 source: https://skillmd.com/skills/magnus919/training-data-annotation15 spec-version: '1.0'16 tags: annotation, active-learning, labeling, human-review, training-data17---1819# Training Data Annotation2021Treat annotation as a measurement system and a human workflow. The objective is22useful, auditable labels for a defined downstream decision, not the largest23possible label count.2425## Workflow26271. Define the downstream task, population, label unit, label set, edge cases,28 abstention label, and what constitutes an unusable item. Separate training,29 validation, held-out evaluation, and challenge data before sampling.302. Write an annotation guide with positive and negative examples, boundary31 rules, source requirements, escalation, and a change log. Have domain owners32 approve consequential definitions.333. Establish provenance and rights: source, collection time, license/consent,34 transformations, sensitive fields, access, retention/deletion, and dataset35 version. Do not move data into a labeling system until this record is clear.364. Acquire items using a declared mix of representative sampling and targeted37 acquisition. Uncertainty finds confusing items; diversity and novelty find38 gaps; stratification protects rare or high-risk slices. Keep acquisition39 batches identifiable and retain a representative held-out audit sample.405. Qualify annotators for the task, train them on calibration items, and measure41 quality and workload. Record role and relevant expertise in restricted42 metadata. Do not use agreement as a universal truth score.436. Run blinded calibration and periodic blind rechecks. Compare labels with44 adjudicated references where available; preserve meaningful disagreement,45 ambiguity, and abstention instead of forcing consensus.467. Review disagreement by item, label, slice, annotator, and guide version.47 Adjudicate only with a stated rule and qualified authority. Feed confirmed48 guide defects back into training and re-review affected labels.498. Stop or continue based on marginal downstream value, coverage, quality,50 cost, and time. An item quota may stop spending but cannot by itself establish sufficient quality. Freeze the51 manifest and hand off to data-engineering for storage/orchestration and52 ml-engineering for training use.5354## Acquisition guardrails5556Never let model confidence define the whole dataset. Sample a representative57held-out audit independently of active-learning selection, and compare active58batch outcomes against it. Track uncertainty method, model/version, seed,59selection scores, slice quotas, and excluded items. If confidence is uncalibrated60or the model is out of distribution, treat its ranking as a search heuristic and61increase diversity, novelty, and human review.6263## Human and quality guardrails6465Design pay, volume, time limits, breaks, and escalation so workers can perform66the task reliably; route employment, compensation, and legal questions to the67appropriate human owner. Blind annotators to model predictions when measuring68independent labels. If preannotations are shown, measure anchoring by a blinded69control or separate review pass. Report agreement by label type and slice with70the uncertainty and prevalence needed to interpret it. A minority label can be71correct when the item is ambiguous; adjudication must explain the rule used.7273## Routing and exit7475Load [references/task-and-label-design.md](references/task-and-label-design.md)76for schemas and guide design, [references/acquisition-and-stopping.md](references/acquisition-and-stopping.md)77for sampling and stopping, [references/annotator-quality.md](references/annotator-quality.md)78for people and disagreement, and [references/provenance-and-rights.md](references/provenance-and-rights.md)79for dataset authority. Use the templates when a durable record is needed.80For primary methodology sources and transfer limits, read81[references/source-index.md](references/source-index.md).8283Route statistical estimators and power to [data-scientist](../data-scientist/SKILL.md),84storage and pipelines to [data-engineering](../data-engineering/SKILL.md), model85training and leakage checks to [ml-engineering](../ml-engineering/SKILL.md), and86labeling-interface interaction design to [product-design-and-ux](../product-design-and-ux/SKILL.md).8788## When not to use8990Do not use this skill as the primary owner for statistical inference or91experimental design, data platforms, model training, or UI implementation.92Those neighboring skills consume this skill's annotation manifest and quality93evidence.9495## Templates9697- [annotation guide](templates/annotation-guide.md) — use when recording this stage of the annotation workflow.98- [annotation plan](templates/annotation-plan.md) — use when recording this stage of the annotation workflow.99- [quality report](templates/quality-report.md) — use when recording this stage of the annotation workflow.