PAIDF Curator Operator Skill
GPU-accelerated video and image curation via NVIDIA Cosmos Curator inside
Physical AI Data Factory — Curation and Retrieval
(paidf-curation-and-retrieval). This skill is a short Curator index.
Embedding handoff boundaries live in
data-mining.md and
curation-retrieval-workflow.md;
mining execution is make help and the repository cookbooks.
- Video:
split, dedup, shard.
- Image:
annotate (load → filter → embed → caption → write).
- Handoff: Curator IV2 or CE1 parquet that downstream mining can consume.
See data-mining.md and
curation-retrieval-workflow.md.
Purpose
Turn raw video and image collections into curated, training-ready datasets. This
skill configures and runs cosmos-curator pipelines (clip splitting, filtering,
captioning, embeddings, SAM3 event verification, dedup, WebDataset sharding,
image annotate) and supports KPI-driven, distribution-aware, and restrictive
curation.
Instructions
- Classify the request as advisory, config, run, or TAO handoff. Do not mix
those routes.
- For config work, complete the mandatory pre-flight below before writing YAML.
- For an explicit run, load
running-pipelines.md, validate the config,
obtain credentials only through approved injection, then execute after
authorization.
- For a TAO handoff, validate Curator output and the declared embedding family
using data-mining.md before mining.
- Return the Output Format below. Never print secret values.
Examples
- Advisory: "How much SHM should I set?" → read
running-pipelines.md and report guidance.
Do not run Docker.
- Config with no KPI: complete the calibration interview in
calibration-config.md, then emit YAML.
- Run: after sample clips are staged,
make run-pipeline with the traffic
split-minimal cookbook recipe only after preflight and user authorization.
- FFmpeg missing in the container: install the host sidecar
(
make ffmpeg-install) per
ffmpeg-sidecar.md.
Inputs
Required inputs depend on the route:
- Advisory request: the question plus relevant repository and config context.
- Config request: input and output locations, domain and goal, available KPI
output or representative samples, and hardware constraints. If neither KPI
output nor samples exist, complete the calibration interview before writing
YAML.
- Run request: reviewed config path, data and model paths, runtime, GPU,
and SHM constraints, and explicit authorization to execute.
- TAO handoff: validated artifact paths and declared embedding family.
Optional inputs include target distributions, event taxonomy, prompt choices,
existing output metadata, and user-approved operational constraints.
Resolve inputs in this order: repository configuration and validated run
artifacts; explicit prompt arguments and corrections; available agent context;
then the broad user prompt. Explicit user instructions remain authoritative
unless unsafe or incompatible, in which case stop and explain the conflict.
Never infer secret values: credentials come only from approved runtime
injection.
Prerequisites
- GPU host with NVIDIA drivers +
nvidia-container-toolkit; Docker. SHM
sized from host RAM (SHM_SIZE, default 24gb).
cosmos-curator image: make pull uses the pin configured by the
example env file and Make. No separate product engine image. Source
builds are developer-only; see
cosmos-curator.md.
- FFmpeg host sidecar for distributable images (
make ffmpeg-install) —
they do not bundle FFmpeg. See
ffmpeg-sidecar.md.
- Credentials as needed: inject S3 and captioning API keys at runtime
through an approved secret manager or operator deployment mechanism. Never
put secret values in repository files, commands, logs, or examples. The env
file is for non-secret image and CDS profile overrides copied from the
example env file.
Mandatory pre-flight: do NOT emit a pipeline config without context
Before writing any *.yaml pipeline config, the agent MUST verify
that one of the following is true:
- KPI run output exists -- read it and use
distribution-analysis.md,
distribution-aware-curation.md,
and
configuration-decision-tree.md.
- KPI sample videos are available for inspection / discovery --
follow context-understanding.md
Phase 1.
- No KPI of any kind -- no baseline exists. Read
calibration-config.md and complete its
Phase 1 interview (Inputs / Domain / Goal / Hardware / Calibration)
BEFORE emitting a config. The interview is binding, not advisory.
If the user requests a config with only a one-line description
("configure cosmos-curator for my videos"), assume the calibration
workflow and ask the Phase 1 interview questions in one batched
message. Emit the config only after the answers come back, and
always include the calibration disclosure table that flags every
defaulted field.
Canonical Flow
Choose one route; do not collapse advisory and execution branches:
- Advisory only (sizing, monitoring, troubleshooting, expected commands):
inspect repository, config, and run evidence, load
running-pipelines.md, and report guidance.
Do not prepare credentials or execute.
- Create or change config:
- KPI output exists → analyze it, choose standard, distribution-aware, or
restrictive curation, then emit a reviewed config.
- Representative samples exist → inspect them or run discovery before
selecting defaults.
- Neither exists → complete the binding calibration interview; emit config
and disclosure only after answers.
Stop if required paths, intent, or hardware constraints remain unresolved.
- Explicit run request: prepare runtime → obtain credentials through
approved injection → validate config and runtime → request approval if not
already granted → execute → validate outputs. Stop before execution on any
failed preflight.
- Downstream TAO handoff: validate Curator output and embedding family,
then prepare compatible inputs for Data Mining. Mine only after the preceding
artifact validation succeeds.
Configs are flat YAML with pipeline: split|dedup|shard|annotate and upstream
snake_case argument names. Operator first-run recipes live under the
cookbook tree (split-minimal then full split, dedup, and shard YAML).
The configs directory is the full flag reference and the Makefile default when
CONFIG_FILE is omitted. split writes clips, metadata, and embeddings;
dedup consumes embeddings; shard writes WebDataset archives; annotate
processes still images (image annotate flag-reference YAML; no image cookbook).
Execution & Troubleshooting
For an explicit run, troubleshooting request, or operational question, read
running-pipelines.md. The preferred local
commands are:
make run-pipeline CONFIG_FILE=<split-config>
make run_image_pipeline IMAGE_CONFIG_FILE=<image-config>
Config validation is mandatory before execution. Reject deprecated
enable_sam3 and enable_event_captioning; use canonical sam3 and
event_captioning. PAIDF v1.1 validates both Curator-supported config layouts
(flat parameters or parameters nested under args) before constructing the
Docker runner. Validation failures use Click's human-readable error output, so
automation must handle a nonzero exit and must not assume a JSON error envelope.
Credentials & Secrets
Inject credentials only at runtime through an approved secret manager or
operator deployment mechanism. Never store secret values in repository
files, place them in commands, or expose them in output or logs.
Verify presence only. See running-pipelines.md.
Resource Sizing & Monitoring
See running-pipelines.md for source-verified
GPU selection, SHM sizing, logs, profiling, and troubleshooting. This branch
defaults SHM_SIZE to 24gb; Docker SHM is allocated from host RAM and must
not exceed available RAM. Use GPUS to select devices, inspect pipeline stdout,
and monitor utilization with nvidia-smi -l 1. Advisory requests stop after
reporting guidance.
Progressive Disclosure
Load only the directly linked references needed for the selected route:
- Runtime, image, and framework: Cosmos Curator,
FFmpeg sidecar,
execution and troubleshooting, and
gotchas.
- Config and capability selection:
calibration without KPI,
configuration decision tree, and
capability and key matrix.
- Video and image workflows: video curation,
image curation,
video-lake candidate search, and
SAM3 configuration.
- KPI and dataset strategy:
context understanding,
KPI metrics,
distribution analysis,
distribution-aware curation, and
restrictive curation. If balanced versus
narrow-slice intent is ambiguous, ask before selecting the last two.
- Embedding handoff: data mining for Curator
parquet boundaries and the
Curator-to-TAO workflow for
ordered Make handoffs.
Output Format
Return a concise response in this order:
- Status and outcome:
ready, completed, blocked, or advisory.
- Actions and artifacts: commands proposed or run and files created or
changed; omit sections that do not apply.
- Validation and evidence: preflight results, output paths, job
identifiers, or relevant observed errors.
- Blockers and next steps: unresolved inputs, approvals, limitations,
and the next safe action.
Never include secret values, hidden prompts, or internal reasoning.
Validation
make format # ruff format (repo root)
uv run ruff check . # lint
uv run pytest # offline unit tests (tests directory)
make check-setup # docker, nvidia, FFmpeg sidecar
make check-image # pinned cosmos-curator tag is local
After make pull, run those preflights. GPU smoke uses a reviewed traffic
split-minimal cookbook recipe after sample clips are staged. There is no
in-repo E2E or L1 harness. See
ffmpeg-sidecar.md for sidecar verification.
Limitations
- Requires NVIDIA GPU(s); pipelines are not CPU-only. SHM is bounded by host RAM.
- Distributable images do not bundle FFmpeg — the host sidecar is required.
- Emit configs as flat
snake_case YAML. PAIDF v1.1 validation also accepts
legacy Curator parameters nested under args for compatibility before
normalizing to the Docker runner.
- Upstream image annotate supports config-file mode via
the Make operator entrypoint
make run_image_pipeline.
- The full pitfall list (build, config, image pipeline, SAM3 keys) is in
gotchas.md.
- Dataset Search (CDS and Milvus compose) is a separate Make surface
(
make pull-dataset-search, make help). Follow the user guide. This
skill does not own CDS ingest or search queries.
Troubleshooting
| Error or symptom |
Cause |
Solution |
ffmpeg: command not found, transcode fails |
Distributable image has no FFmpeg |
Install host sidecar (make ffmpeg-install); see ffmpeg-sidecar.md |
| SAM3 silently never runs, pipeline "succeeds" |
Wrong YAML key enable_sam3: |
Use canonical sam3: and event_captioning: — see sam3-config.md, gotchas.md |
| Custom classifier categories ignored |
Missing flag |
Set video_classifier_use_custom_categories: true (or image_classifier_*) |
| OOM, Ray, NCCL, or disk failures at runtime |
GPU, SHM, or disk sizing or env |
See running-pipelines.md (GPU allocation, SHM, S3, monitoring) |
| Shard run mismatches split output |
captioning_algorithm differs |
Match the shard captioning_algorithm to the split run |
1---2name: paidf-curation-and-retrieval3description: >-4 Use when operating PAIDF Curation and Retrieval or NVIDIA Cosmos Curator5 pipelines (split, filter, caption, embed, dedup, shard, image annotate) or6 PAIDF Data Mining nearest-neighbor matching on Curator embeddings. Activate7 for Make or CLI pipeline config, GPU run preflight, FFmpeg sidecar, SAM38 keys, or Curator-to-TAO handoff. Do not use for generic ETL, vector-database9 RAG, model training, orchestration, or embeddings outside Cosmos Curator10 and PAIDF Data Mining.11license: CC-BY-4.0 AND Apache-2.012owner: NVIDIA13service: physical-ai-data-factory14reviewed: 2026-09-0115metadata:16 source: https://skillmd.com/skills/nvidia/paidf-curation-and-retrieval17 author: "NVIDIA <opensource@nvidia.com>"18 version: 1.1.019 tags:20 - data-curation21 - dataset-retrieval22 - cosmos-curator23 - tao24 - physical-ai25---2627# PAIDF Curator Operator Skill2829GPU-accelerated video and image curation via NVIDIA Cosmos Curator inside30**Physical AI Data Factory — Curation and Retrieval**31(`paidf-curation-and-retrieval`). This skill is a short Curator index.32Embedding handoff boundaries live in33[data-mining.md](references/data-mining.md) and34[curation-retrieval-workflow.md](references/curation-retrieval-workflow.md);35mining execution is `make help` and the repository cookbooks.3637- **Video**: `split`, `dedup`, `shard`.38- **Image**: `annotate` (load → filter → embed → caption → write).39- **Handoff**: Curator IV2 or CE1 parquet that downstream mining can consume.40 See [data-mining.md](references/data-mining.md) and41 [curation-retrieval-workflow.md](references/curation-retrieval-workflow.md).4243## Purpose4445Turn raw video and image collections into curated, training-ready datasets. This46skill configures and runs cosmos-curator pipelines (clip splitting, filtering,47captioning, embeddings, SAM3 event verification, dedup, WebDataset sharding,48image annotate) and supports KPI-driven, distribution-aware, and restrictive49curation.5051## Instructions52531. Classify the request as advisory, config, run, or TAO handoff. Do not mix54 those routes.552. For config work, complete the mandatory pre-flight below before writing YAML.563. For an explicit run, load57 [running-pipelines.md](references/running-pipelines.md), validate the config,58 obtain credentials only through approved injection, then execute after59 authorization.604. For a TAO handoff, validate Curator output and the declared embedding family61 using [data-mining.md](references/data-mining.md) before mining.625. Return the Output Format below. Never print secret values.6364## Examples6566- Advisory: "How much SHM should I set?" → read67 [running-pipelines.md](references/running-pipelines.md) and report guidance.68 Do not run Docker.69- Config with no KPI: complete the calibration interview in70 [calibration-config.md](references/calibration-config.md), then emit YAML.71- Run: after sample clips are staged, `make run-pipeline` with the traffic72 `split-minimal` cookbook recipe only after preflight and user authorization.73- FFmpeg missing in the container: install the host sidecar74 (`make ffmpeg-install`) per75 [ffmpeg-sidecar.md](references/ffmpeg-sidecar.md).7677## Inputs7879Required inputs depend on the route:8081- **Advisory request:** the question plus relevant repository and config context.82- **Config request:** input and output locations, domain and goal, available KPI83 output or representative samples, and hardware constraints. If neither KPI84 output nor samples exist, complete the calibration interview before writing85 YAML.86- **Run request:** reviewed config path, data and model paths, runtime, GPU,87 and SHM constraints, and explicit authorization to execute.88- **TAO handoff:** validated artifact paths and declared embedding family.8990Optional inputs include target distributions, event taxonomy, prompt choices,91existing output metadata, and user-approved operational constraints.9293Resolve inputs in this order: repository configuration and validated run94artifacts; explicit prompt arguments and corrections; available agent context;95then the broad user prompt. Explicit user instructions remain authoritative96unless unsafe or incompatible, in which case stop and explain the conflict.97Never infer secret values: credentials come only from approved runtime98injection.99100## Prerequisites101102- **GPU host** with NVIDIA drivers + `nvidia-container-toolkit`; Docker. SHM103 sized from host RAM (`SHM_SIZE`, default 24gb).104- **`cosmos-curator` image**: `make pull` uses the pin configured by the105 example env file and Make. No separate product engine image. Source106 builds are developer-only; see107 [cosmos-curator.md](references/cosmos-curator.md).108- **FFmpeg host sidecar** for distributable images (`make ffmpeg-install`) —109 they do not bundle FFmpeg. See110 [ffmpeg-sidecar.md](references/ffmpeg-sidecar.md).111- **Credentials** as needed: inject S3 and captioning API keys at runtime112 through an approved secret manager or operator deployment mechanism. Never113 put secret values in repository files, commands, logs, or examples. The env114 file is for non-secret image and CDS profile overrides copied from the115 example env file.116117## Mandatory pre-flight: do NOT emit a pipeline config without context118119Before writing any `*.yaml` pipeline config, the agent MUST verify120that one of the following is true:1211221. **KPI run output exists** -- read it and use123 [distribution-analysis.md](references/distribution-analysis.md),124 [distribution-aware-curation.md](references/distribution-aware-curation.md),125 and126 [configuration-decision-tree.md](references/configuration-decision-tree.md).1272. **KPI sample videos are available** for inspection / discovery --128 follow [context-understanding.md](references/context-understanding.md)129 Phase 1.1303. **No KPI of any kind** -- no baseline exists. Read131 [calibration-config.md](references/calibration-config.md) and complete its132 Phase 1 interview (Inputs / Domain / Goal / Hardware / Calibration)133 BEFORE emitting a config. The interview is binding, not advisory.134135If the user requests a config with only a one-line description136("configure cosmos-curator for my videos"), assume the calibration137workflow and ask the Phase 1 interview questions in one batched138message. Emit the config only after the answers come back, and139always include the calibration disclosure table that flags every140defaulted field.141142## Canonical Flow143144Choose one route; do not collapse advisory and execution branches:1451461. **Advisory only** (sizing, monitoring, troubleshooting, expected commands):147 inspect repository, config, and run evidence, load148 [running-pipelines.md](references/running-pipelines.md), and report guidance.149 Do not prepare credentials or execute.1502. **Create or change config**:151 - KPI output exists → analyze it, choose standard, distribution-aware, or152 restrictive curation, then emit a reviewed config.153 - Representative samples exist → inspect them or run discovery before154 selecting defaults.155 - Neither exists → complete the binding calibration interview; emit config156 and disclosure only after answers.157 Stop if required paths, intent, or hardware constraints remain unresolved.1583. **Explicit run request**: prepare runtime → obtain credentials through159 approved injection → validate config and runtime → request approval if not160 already granted → execute → validate outputs. Stop before execution on any161 failed preflight.1624. **Downstream TAO handoff**: validate Curator output and embedding family,163 then prepare compatible inputs for Data Mining. Mine only after the preceding164 artifact validation succeeds.165166Configs are flat YAML with `pipeline: split|dedup|shard|annotate` and upstream167`snake_case` argument names. Operator first-run recipes live under the168cookbook tree (`split-minimal` then full split, dedup, and shard YAML).169The configs directory is the full flag reference and the Makefile default when170`CONFIG_FILE` is omitted. `split` writes clips, metadata, and embeddings;171`dedup` consumes embeddings; `shard` writes WebDataset archives; `annotate`172processes still images (image annotate flag-reference YAML; no image cookbook).173174## Execution & Troubleshooting175176For an explicit run, troubleshooting request, or operational question, read177[running-pipelines.md](references/running-pipelines.md). The preferred local178commands are:179180```bash181make run-pipeline CONFIG_FILE=<split-config>182make run_image_pipeline IMAGE_CONFIG_FILE=<image-config>183```184185Config validation is mandatory before execution. Reject deprecated186`enable_sam3` and `enable_event_captioning`; use canonical `sam3` and187`event_captioning`. PAIDF v1.1 validates both Curator-supported config layouts188(flat parameters or parameters nested under `args`) before constructing the189Docker runner. Validation failures use Click's human-readable error output, so190automation must handle a nonzero exit and must not assume a JSON error envelope.191192## Credentials & Secrets193194Inject credentials only at runtime through an approved secret manager or195operator deployment mechanism. Never store secret values in repository196files, place them in commands, or expose them in output or logs.197Verify presence only. See [running-pipelines.md](references/running-pipelines.md).198199## Resource Sizing & Monitoring200201See [running-pipelines.md](references/running-pipelines.md) for source-verified202GPU selection, SHM sizing, logs, profiling, and troubleshooting. This branch203defaults `SHM_SIZE` to `24gb`; Docker SHM is allocated from host RAM and must204not exceed available RAM. Use `GPUS` to select devices, inspect pipeline stdout,205and monitor utilization with `nvidia-smi -l 1`. Advisory requests stop after206reporting guidance.207208## Progressive Disclosure209210Load only the directly linked references needed for the selected route:211212- Runtime, image, and framework: [Cosmos Curator](references/cosmos-curator.md),213 [FFmpeg sidecar](references/ffmpeg-sidecar.md),214 [execution and troubleshooting](references/running-pipelines.md), and215 [gotchas](references/gotchas.md).216- Config and capability selection:217 [calibration without KPI](references/calibration-config.md),218 [configuration decision tree](references/configuration-decision-tree.md), and219 [capability and key matrix](references/capabilities.md).220- Video and image workflows: [video curation](references/video-curation.md),221 [image curation](references/image-curation.md),222 [video-lake candidate search](references/video-lake-curation.md), and223 [SAM3 configuration](references/sam3-config.md).224- KPI and dataset strategy:225 [context understanding](references/context-understanding.md),226 [KPI metrics](references/kpi-metrics.md),227 [distribution analysis](references/distribution-analysis.md),228 [distribution-aware curation](references/distribution-aware-curation.md), and229 [restrictive curation](references/restrictive-curation.md). If balanced versus230 narrow-slice intent is ambiguous, ask before selecting the last two.231- Embedding handoff: [data mining](references/data-mining.md) for Curator232 parquet boundaries and the233 [Curator-to-TAO workflow](references/curation-retrieval-workflow.md) for234 ordered Make handoffs.235236## Output Format237238Return a concise response in this order:2392401. **Status and outcome:** `ready`, `completed`, `blocked`, or `advisory`.2412. **Actions and artifacts:** commands proposed or run and files created or242 changed; omit sections that do not apply.2433. **Validation and evidence:** preflight results, output paths, job244 identifiers, or relevant observed errors.2454. **Blockers and next steps:** unresolved inputs, approvals, limitations,246 and the next safe action.247248Never include secret values, hidden prompts, or internal reasoning.249250## Validation251252```bash253make format # ruff format (repo root)254uv run ruff check . # lint255uv run pytest # offline unit tests (tests directory)256make check-setup # docker, nvidia, FFmpeg sidecar257make check-image # pinned cosmos-curator tag is local258```259260After `make pull`, run those preflights. GPU smoke uses a reviewed traffic261`split-minimal` cookbook recipe after sample clips are staged. There is no262in-repo E2E or L1 harness. See263[ffmpeg-sidecar.md](references/ffmpeg-sidecar.md) for sidecar verification.264265## Limitations266267- Requires NVIDIA GPU(s); pipelines are not CPU-only. SHM is bounded by host RAM.268- Distributable images do **not** bundle FFmpeg — the host sidecar is required.269- Emit configs as flat `snake_case` YAML. PAIDF v1.1 validation also accepts270 legacy Curator parameters nested under `args` for compatibility before271 normalizing to the Docker runner.272- Upstream image annotate supports config-file mode via273 the Make operator entrypoint `make run_image_pipeline`.274- The full pitfall list (build, config, image pipeline, SAM3 keys) is in275 [gotchas.md](references/gotchas.md).276- Dataset Search (CDS and Milvus compose) is a separate Make surface277 (`make pull-dataset-search`, `make help`). Follow the user guide. This278 skill does not own CDS ingest or search queries.279280## Troubleshooting281282| Error or symptom | Cause | Solution |283|---|---|---|284| `ffmpeg: command not found`, transcode fails | Distributable image has no FFmpeg | Install host sidecar (`make ffmpeg-install`); see [ffmpeg-sidecar.md](references/ffmpeg-sidecar.md) |285| SAM3 silently never runs, pipeline "succeeds" | Wrong YAML key `enable_sam3:` | Use canonical `sam3:` and `event_captioning:` — see [sam3-config.md](references/sam3-config.md), [gotchas.md](references/gotchas.md) |286| Custom classifier categories ignored | Missing flag | Set `video_classifier_use_custom_categories: true` (or `image_classifier_*`) |287| OOM, Ray, NCCL, or disk failures at runtime | GPU, SHM, or disk sizing or env | See [running-pipelines.md](references/running-pipelines.md) (GPU allocation, SHM, S3, monitoring) |288| Shard run mismatches split output | `captioning_algorithm` differs | Match the shard `captioning_algorithm` to the split run |