CDF Auto-Labeling
Use this skill when a user wants to kick off CDF Auto-Labeling on their
own data, domain, or use case, or when the request matches a shipped cookbook,
stage, authoring, or migration task. This is a router: sequence the specialized
references instead of duplicating their detail.
Routing (Read First)
| Request looks like |
Read |
| New user, clean checkout, first validated run, "how do I get started" |
This file, then the matching reference below |
| Choose annotation targets / stage subset for a domain |
references/scenario-planning.md |
| Create, review, or adapt a cookbook |
references/cookbook-authoring.md |
| Write or adapt VLM/LLM prompts or question banks |
references/prompt-authoring.md |
| Migrate an existing annotation repo into this one |
references/pipeline-migration.md |
| Run the video data augmentation cookbook |
references/video-data-augmentation.md |
| Run or choose an EPAS / PAS cookbook |
references/event-and-person-attribute-search.md |
| Run event-verification reasoning |
references/event-verification-reasoning.md |
| Debug an already-integrated workflow |
references/workflow-runner-debugging.md |
Pull prebuilt stage images from NGC, log in to nvcr.io, or fix a missing cdf-* image |
references/ngc-containers.md |
| Implement or review a new stage or Dockerized service |
references/workflow-stage-integration.md |
| Configure or debug one production stage |
The matching file under references/stages/ |
Stage references: super-resolution,
detection-and-tracking,
captioning,
visual-qa,
reasoning,
person-attribute-search,
grounding-2d,
referring-expressions,
training-export.
Instructions
- Confirm the critical run inputs with the user before doing anything else, and
ask a concise question whenever one is missing or ambiguous - never guess or
silently invent a default. At minimum confirm: input data path, output path,
VLM/LLM endpoint URLs and model names, model cache path, GPU ids, and (for
reasoning-capable models) the
max_tokens cap. Restate the confirmed values
back to the user before the first execution.
- Verify the environment: repository cloned,
make targets available, the
model cache path exists, the VLM/LLM endpoints are reachable, and a GPU is
available. State any missing prerequisite as a blocker instead of assuming it.
If the stage images are not present locally, offer to pull the prebuilt
images from NGC with make pull instead of building them. Use
ngc-containers.
- Run a shipped example first to confirm the stack works end to end before
customizing. Pick the closest operator pipeline - video data augmentation,
event-and-person-attribute-search, or event-verification-reasoning - and
run its committed cookbook. Use the matching operator reference.
- Plan the target scenario: define modality, domain, intended consumer, and
required annotations, and get a minimal stage subset. Use
scenario-planning.
- Adapt the closest shipped cookbook to the new domain rather than authoring
from scratch. Use cookbook-authoring.
- Author the domain prompts and question banks. Use
prompt-authoring.
- Configure the per-stage settings for the domain (detector classes or SAM3
prompts, endpoints, windowing,
max_tokens). Use the relevant stage
reference, starting with
detection-and-tracking.
- Dry-run the adapted cookbook, then execute and validate the outputs. Use
workflow-runner-debugging.
Adopting an existing external annotation or dataset-generation repository into
CDF instead of starting from a shipped cookbook is a migration task; use
pipeline-migration for that path.
Examples
New user, new domain: "I cloned the repo and have my own warehouse-safety video.
How do I produce auto-labels for my domain?"
Guided path:
- Confirm env (model cache, VLM/LLM endpoints, GPU), then prove the stack on a
shipped example before customizing:
make run SCRIPT=workflow-runner:main \
ARGS='--cookbook-file cookbooks/video_data_augmentation/configs/pipeline_video.yaml --container-dry-run'
- Plan the domain (scenario-planning) -> subset
detection_and_tracking -> captioning -> visual_qa -> reasoning -> training_export
(add grounding_2d for caption→boxes or referring_expressions for boxes→phrases;
use grounding-2d /
referring-expressions).
- Copy the closest cookbook to
cookbooks/warehouse_safety/configs/pipeline.yaml
and adapt inputs, detector classes/SAM3 prompts, prompts, and question banks.
- Dry-run the new cookbook, then run for real and validate outputs:
make run SCRIPT=workflow-runner:main \
ARGS='--cookbook-file cookbooks/warehouse_safety/configs/pipeline.yaml --container-dry-run'
Guardrails
- Do not guess or fabricate the critical inputs enumerated in step 1; if any is
missing or ambiguous, ask the user and confirm before executing.
- Do not customize a cookbook before a shipped example runs clean; a broken base
makes domain debugging ambiguous.
- Keep the first custom pipeline minimal - only the stages needed for the
requested annotations - and expand later.
- Verify that every selected stage's service package and image exist in the
current branch before promising an end-to-end run.
- Do not put secrets, tokens, or absolute home paths in committed cookbooks; use
placeholders such as
<model-cache> and env vars for endpoint keys.
- For reasoning-capable models (for example Gemini 3 Flash), raise
max_tokens
on the visual_qa and reasoning LLM substages to avoid the thinking-token
tax; keep the default cap for non-reasoning models.
- Do not rely on non-CDF pipelines, commands, or file locations. A first run
must be reproducible through
workflow-runner:main inside this repo.
1---2name: cdf-auto-labeling3description: >-4 Use when a user needs to get started with CDF Auto-Labeling, plan a5 scenario, run or debug a shipped cookbook, author prompts or cookbooks,6 migrate a pipeline, configure a stage, or pull the prebuilt stage containers7 from NGC (nvcr.io). Confirm critical inputs (data8 path, output path, endpoints) and ask when any are missing. This is a9 router: read the matching reference instead of inventing a workflow.10license: Apache-2.011owner: NVIDIA12service: cosmos-data-factory13version: 1.1.014reviewed: 2026-09-1415author: "NVIDIA <opensource@nvidia.com>"16metadata:17 source: https://skillmd.com/skills/nvidia/cdf-auto-labeling18 author: "NVIDIA <opensource@nvidia.com>"19 tags: [getting-started, onboarding, new-domain, quickstart, new-use-case]20---2122# CDF Auto-Labeling2324Use this skill when a user wants to kick off CDF Auto-Labeling on their25own data, domain, or use case, or when the request matches a shipped cookbook,26stage, authoring, or migration task. This is a router: sequence the specialized27references instead of duplicating their detail.2829## Routing (Read First)3031| Request looks like | Read |32| --- | --- |33| New user, clean checkout, first validated run, "how do I get started" | This file, then the matching reference below |34| Choose annotation targets / stage subset for a domain | [`references/scenario-planning.md`](references/scenario-planning.md) |35| Create, review, or adapt a cookbook | [`references/cookbook-authoring.md`](references/cookbook-authoring.md) |36| Write or adapt VLM/LLM prompts or question banks | [`references/prompt-authoring.md`](references/prompt-authoring.md) |37| Migrate an existing annotation repo into this one | [`references/pipeline-migration.md`](references/pipeline-migration.md) |38| Run the video data augmentation cookbook | [`references/video-data-augmentation.md`](references/video-data-augmentation.md) |39| Run or choose an EPAS / PAS cookbook | [`references/event-and-person-attribute-search.md`](references/event-and-person-attribute-search.md) |40| Run event-verification reasoning | [`references/event-verification-reasoning.md`](references/event-verification-reasoning.md) |41| Debug an already-integrated workflow | [`references/workflow-runner-debugging.md`](references/workflow-runner-debugging.md) |42| Pull prebuilt stage images from NGC, log in to `nvcr.io`, or fix a missing `cdf-*` image | [`references/ngc-containers.md`](references/ngc-containers.md) |43| Implement or review a new stage or Dockerized service | [`references/workflow-stage-integration.md`](references/workflow-stage-integration.md) |44| Configure or debug one production stage | The matching file under [`references/stages/`](references/stages/) |4546Stage references: [super-resolution](references/stages/super-resolution.md),47[detection-and-tracking](references/stages/detection-and-tracking.md),48[captioning](references/stages/captioning.md),49[visual-qa](references/stages/visual-qa.md),50[reasoning](references/stages/reasoning.md),51[person-attribute-search](references/stages/person-attribute-search.md),52[grounding-2d](references/stages/grounding-2d.md),53[referring-expressions](references/stages/referring-expressions.md),54[training-export](references/stages/training-export.md).5556## Instructions57581. Confirm the critical run inputs with the user before doing anything else, and59 ask a concise question whenever one is missing or ambiguous - never guess or60 silently invent a default. At minimum confirm: input data path, output path,61 VLM/LLM endpoint URLs and model names, model cache path, GPU ids, and (for62 reasoning-capable models) the `max_tokens` cap. Restate the confirmed values63 back to the user before the first execution.642. Verify the environment: repository cloned, `make` targets available, the65 model cache path exists, the VLM/LLM endpoints are reachable, and a GPU is66 available. State any missing prerequisite as a blocker instead of assuming it.67 If the stage images are not present locally, offer to pull the prebuilt68 images from NGC with `make pull` instead of building them. Use69 [ngc-containers](references/ngc-containers.md).703. Run a shipped example first to confirm the stack works end to end before71 customizing. Pick the closest operator pipeline - video data augmentation,72 event-and-person-attribute-search, or event-verification-reasoning - and73 run its committed cookbook. Use the matching operator reference.744. Plan the target scenario: define modality, domain, intended consumer, and75 required annotations, and get a minimal stage subset. Use76 [scenario-planning](references/scenario-planning.md).775. Adapt the closest shipped cookbook to the new domain rather than authoring78 from scratch. Use [cookbook-authoring](references/cookbook-authoring.md).796. Author the domain prompts and question banks. Use80 [prompt-authoring](references/prompt-authoring.md).817. Configure the per-stage settings for the domain (detector classes or SAM382 prompts, endpoints, windowing, `max_tokens`). Use the relevant stage83 reference, starting with84 [detection-and-tracking](references/stages/detection-and-tracking.md).858. Dry-run the adapted cookbook, then execute and validate the outputs. Use86 [workflow-runner-debugging](references/workflow-runner-debugging.md).8788Adopting an existing external annotation or dataset-generation repository into89CDF instead of starting from a shipped cookbook is a migration task; use90[pipeline-migration](references/pipeline-migration.md) for that path.9192## Examples9394New user, new domain: "I cloned the repo and have my own warehouse-safety video.95How do I produce auto-labels for my domain?"9697Guided path:9899- Confirm env (model cache, VLM/LLM endpoints, GPU), then prove the stack on a100 shipped example before customizing:101102```bash103make run SCRIPT=workflow-runner:main \104 ARGS='--cookbook-file cookbooks/video_data_augmentation/configs/pipeline_video.yaml --container-dry-run'105```106107- Plan the domain ([scenario-planning](references/scenario-planning.md)) -> subset108 `detection_and_tracking -> captioning -> visual_qa -> reasoning -> training_export`109 (add `grounding_2d` for caption→boxes or `referring_expressions` for boxes→phrases;110 use [grounding-2d](references/stages/grounding-2d.md) /111 [referring-expressions](references/stages/referring-expressions.md)).112- Copy the closest cookbook to `cookbooks/warehouse_safety/configs/pipeline.yaml`113 and adapt inputs, detector classes/SAM3 prompts, prompts, and question banks.114- Dry-run the new cookbook, then run for real and validate outputs:115116```bash117make run SCRIPT=workflow-runner:main \118 ARGS='--cookbook-file cookbooks/warehouse_safety/configs/pipeline.yaml --container-dry-run'119```120121## Guardrails122123- Do not guess or fabricate the critical inputs enumerated in step 1; if any is124 missing or ambiguous, ask the user and confirm before executing.125- Do not customize a cookbook before a shipped example runs clean; a broken base126 makes domain debugging ambiguous.127- Keep the first custom pipeline minimal - only the stages needed for the128 requested annotations - and expand later.129- Verify that every selected stage's service package and image exist in the130 current branch before promising an end-to-end run.131- Do not put secrets, tokens, or absolute home paths in committed cookbooks; use132 placeholders such as `<model-cache>` and env vars for endpoint keys.133- For reasoning-capable models (for example Gemini 3 Flash), raise `max_tokens`134 on the `visual_qa` and `reasoning` LLM substages to avoid the thinking-token135 tax; keep the default cap for non-reasoning models.136- Do not rely on non-CDF pipelines, commands, or file locations. A first run137 must be reproducible through `workflow-runner:main` inside this repo.