CDF Augmentation Pipeline Skill
Remote-API pipeline for captioning, generating, and evaluating augmented camera
data. Models are configured as HTTP endpoints; no local model weights are
included.
Purpose
Use it to select a supported generation mode, author and validate
PipelineConfig YAML, configure captioning/evaluators, and run the
cdf-augmentation:1.2.0 container. It covers Cosmos Transfer/Predict,
Cosmos3 WSM controls, image editing, image-to-video, BYOM endpoints, and
quality gates.
Do not use this skill for training or fine-tuning models, deploying clusters or NIM endpoints, or unrelated application/database development.
Prerequisites
| Requirement |
Detail |
| Docker |
docker --version. The image is remote-API only — it bundles no Cosmos/torch weights, so plain remote inference needs no GPU and no HF_TOKEN. |
| NVIDIA GPU (conditional) |
Only when a configured local stage requires CUDA or decodes H.264 in the augmentation container. See Limitations. |
| Endpoint URLs |
One reachable URL per role used by the selected config. The examples name local Qwen services (Qwen/Qwen3.6-27B-FP8 on vlm, Qwen/Qwen2.5-14B-Instruct on llm), but those are not guaranteed to be running. Resolve every required role independently: keep each reachable configured endpoint and ask only for the missing role URLs. |
| API keys (conditional) |
Only for endpoints requiring authentication. Pass the environment variable named by each endpoint's api_key_env; never hardcode values in YAML. Local unauthenticated endpoints need none. |
| Input media |
A video/image reachable by multistorageclient (s3://, msc://, gs://, az://, or HTTPS). Local paths are allowed only when no external Cosmos evaluator is enabled; external evaluation requires checker-readable remote URIs. |
| Approved release commit (execution only) |
A 40-character commit hash confirmed by the user as release-owner-approved or retrieved from the verified release manifest. Never invent it or rely on an unverified environment value. |
Inputs
Resolve each value in this precedence order: state file → explicit prompt arguments → agent context → user prompt. Ask the user only for what remains unresolved.
| Input |
Required |
Description |
config_path |
Yes |
Path to the pipeline YAML, e.g. configs/cookbook/video-data-augmentation/config_video_transfer_CT3_omni.yaml. If absent, pick a starting config from Supported Models and confirm with the user. |
input_media |
Conditional |
Source video/image → data[].inputs.rgb. Required for every mode except Cosmos Predict inference_type: text2world, where inputs may be null or rgb omitted. Overridable at run time via data.0.inputs.rgb=.... |
control_media |
For precomputed control |
Control video → data[].inputs.controls.<type>. For Cosmos3 WSM transfer, set controls.wsm; the client uploads it rather than exposing its local path to the server. |
output_paths |
Yes |
data[].output.{video,caption,metadata}; evaluation optional. |
model_name |
Yes |
augmentation.model.name — an endpoint id, a role, or a known model name. Free-form string, not an enum. |
endpoint_urls |
Yes |
One endpoints[] entry per role in use. |
api_key_env |
If auth |
Env-var name per endpoint; the value comes from the environment. |
target_attributes |
No |
captioning.llm.variables (e.g. weather_condition, lighting_condition). |
generation_params |
No |
augmentation.parameters — pass-through; only set knobs are sent. |
seed |
No |
Under augmentation.parameters. Before the first candidate, null resolves once to the generator/executor seed when available, otherwise the current Unix time. Retry n uses that resolved base seed plus n. pipeline.retry defaults to 1, so evaluation permits at most 2 candidates by default. |
BYOM model: endpoints, adapters, roles
Each endpoints: list entry declares role, url, wire model, and optional
id, adapter, api_key_env, and timeout. Roles are vlm, llm,
image_edit, video_transfer, video_predict, image2video, and evaluator.
augmentation.model.name resolves by endpoint id, then role, then the known
model-name mapping. Adapter contracts and full fields are in
configuration-schema.md.
Supported Models
When the user hasn't specified a model, choose from their input type and goal:
| Input Type → Goal |
model.name |
Role / default adapter |
Input → Output |
| Video — change scene attributes (weather, lighting, style) |
cosmos-transfer2.5 |
video_transfer / nim |
Video (+ controls) → Video |
| Video + precomputed WSM — follow world-state geometry/motion |
A video_transfer endpoint id, e.g. cosmos3-transfer-wsm |
video_transfer / openai.video.async |
RGB video + WSM control + prompt → Video |
| Video + text — extend or predict continuation |
cosmos-predict |
video_predict / nim |
Video+Text → Video |
| Text only — generate video from scratch |
cosmos-predict (inference_type: text2world) |
video_predict / nim |
Text → Video |
| Image — edit specific attributes |
image-edit |
image_edit / nim (or openai.chat.completions, openai.images.edits) |
Image → Image |
| Image — animate a first frame |
cosmos3-image2video (or your Veo endpoint id) |
image2video / openai.video.sync (Veo: openai.video.async) |
Image + prompt → Video |
Key rule: use Cosmos Transfer for video scene changes, Cosmos Predict for
new/continued video, image edit for still edits, and image-to-video to animate a
frame. For Cosmos3 WSM, set data[].inputs.controls.wsm. Config selection
details are in config-decision-tree.md.
All models run via remote HTTP through one BaseExecutor; there is no local torchrun and no executor_type field.
Canonical Workflow
Follow this ordered decision tree; open the linked references only for the
selected mode's details.
Config Correction Budget
config_attempts counts YAML versions, not validation or preflight calls. Keep
it across the whole workflow and never reset it.
- Set it to
1 after initial authoring.
- On a correctable error, if it is already
3, STOP and report the error.
Otherwise increment it once and make one correction pass containing all
currently reported YAML fixes.
- Validate and preflight that edited version without another increment. Repeat
rule 2 only if a later check finds another correctable YAML error.
STEP 0 — Mandatory Entry Gate
Execute this gate before every other step:
- Scope gate: if the request asks for training, fine-tuning, cluster/NIM
deployment, or unrelated development, decline it and STOP.
- Input gate: resolve values using Inputs. If required values remain
unresolved, request only those values and STOP until they are supplied.
- Proceed only when both gates pass and the input type and augmentation goal
are known.
- Select the model from Supported Models using that input type and goal.
- Author the YAML in this order:
data, endpoints, captioning,
augmentation, evaluators, pipeline, then data_processing. Omit
optional sections rather than creating placeholders. Set
config_attempts = 1 (config attempt 1 of 3).
- Branch on the requested outcome:
- Configuration only or “do not execute”: do not build, launch, run
preflight, or infer. A trusted runtime is an already-running container
whose recorded immutable image ID/digest, checkout commit, and config
mounts satisfy step 4. If validation was explicitly requested and such a
runtime exists, run only the validation command and apply step 5's failure
rules. Return the config (mark it unvalidated if validation was not run)
and STOP.
- Inference/run requested: proceed to step 4.
- Prepare the runtime in these verifiable sub-steps:
a. Obtain
EXPECTED_RELEASE_REF from the user as a release-owner-approved
full commit or retrieve it from the verified release manifest.
b. Verify the Git working tree is clean.
c. Verify HEAD matches EXPECTED_RELEASE_REF. If a release tag selected
the revision, verify the tag first, then compare its resolved full commit.
d. Build the image and capture its immutable sha256: image ID, or obtain a
verified registry digest.
e. Create the cdf Docker bridge if needed and attach local services.
f. Launch the container using the recorded image ID, the cdf network,
required volume mounts, --entrypoint /bin/bash, and only the endpoint
key variables named by api_key_env plus the scoped storage credential
variables required for the configured remote media.
If any sub-step fails, STOP and report the exact failure and relevant
remediation: clean a dirty tree, supply the approved revision, resolve a
revision mismatch, check the Docker daemon/build, fix network setup, or fix
the container launch. Do not continue with a partial runtime.
- Inside the container, run the validation-only command in Usage Step 2 and
classify its result while preserving
config_attempts (current config
attempt N of 3):
- Success: if no external Cosmos evaluator is enabled, proceed directly
to step 7. If one is enabled, proceed to step 6.
- Editable YAML/schema error: when required endpoint URL/model details are
known, apply the Config Correction Budget rule, then rerun validation on
that YAML version.
- Infrastructure/unknown input: for missing endpoint details, credentials,
DNS, connectivity, or unhealthy services, do not edit speculatively.
STOP and report the error or ask for the missing endpoint details.
- With external Cosmos evaluation enabled, preserve
config_attempts
(current config attempt N of 3) and run its hard preflight gate before
captioning/generation by querying /health, /checkers, and
/dependency-graph:
- Pass: require
/health.status == "healthy"; require every configured
checks[].name to exactly equal one string in the top-level checkers
array returned by /checkers and one key in the top-level checkers
object returned by /dependency-graph. Do not use substring, alias, or
nested-field matching. Then proceed to step 7.
- Config error: only a 4xx response from
/checkers or
/dependency-graph whose body explicitly identifies a configured checker
name or invalid config field qualifies. Apply the Config Correction
Budget rule, then return to step 5; revalidation does not increment the
counter.
- Infrastructure failure: any timeout, network/DNS error, 5xx response,
/health failure, unhealthy service, or missing deployed checker stops the
workflow before generation. Create no candidate and report the root cause.
- Enter inference only after step 5 succeeds and, when external evaluation is
enabled, step 6 passes. Run a candidate loop bounded to
pipeline.retry + 1 candidates (retry defaults to 1, so the default
maximum is 2). Resolve the base seed once before the first candidate as
defined in Inputs; retry n uses base_seed + n:
- Generate a candidate and run its configured evaluators.
- On pass, accept it and exit the loop.
- On quality failure with attempts remaining, set the next seed to
base_seed + next_retry_number. If regenerate_caption_on_retry: true,
regenerate the caption next; then generate the replacement. If false,
generate the replacement immediately.
- On exhausted quality retries, fail evaluation and exit the loop.
- On a service/contract failure, stop on the current candidate without
generating another. For external Cosmos evaluation, this means a
submission, polling, or discovery failure; a failed execution; a missing
or skipped result; or an invalid gating contract without a literal
top-level boolean
passed. With pipeline.evaluation.strict: true (the
default), apply retain_failures: its default true retains video, caption,
metadata, and evaluation outputs for inspection; explicit false deletes
them. With strict: false, retain the files and report the evaluator
failure without failing the sample.
- If provider metadata contains
error_code: ambiguous_post, preserve the
retained candidate, do not resubmit automatically, STOP, and defer
to operator recovery. Follow the
ambiguous-POST recovery flow.
Usage
Step 1: Verify, Build, and Launch the Docker Container
Set EXPECTED_RELEASE_REF to a release-owner-reviewed full 40-character commit,
never a tag or branch. If a signed release tag selects the revision, verify the
tag first and record its resolved full commit as EXPECTED_RELEASE_REF. Build
only from a clean matching checkout, then record the resulting immutable image
ID. For a prebuilt release, use its verified digest.
set -e
EXPECTED_RELEASE_REF="${EXPECTED_RELEASE_REF:?set a reviewed full commit ID}"
test "${#EXPECTED_RELEASE_REF}" -eq 40
case "$EXPECTED_RELEASE_REF" in *[!0-9a-fA-F]*) exit 1 ;; esac
test -z "$(git status --porcelain)"
test "$(git rev-parse HEAD)" = "$(git rev-parse "${EXPECTED_RELEASE_REF}^{commit}")"
DOCKER_BUILDKIT=0 docker build \
-t cdf-augmentation:1.2.0 \
-f docker/Dockerfile .
CDF_IMAGE_ID="$(docker image inspect --format '{{.Id}}' cdf-augmentation:1.2.0)"
case "$CDF_IMAGE_ID" in sha256:*) ;; *) exit 1 ;; esac
test "$(docker image inspect --format '{{.Id}}' cdf-augmentation:1.2.0)" = "$CDF_IMAGE_ID"
docker network inspect cdf >/dev/null 2>&1 || \
docker network create cdf
# Attach each local model container once, for example:
# docker network connect cdf vlm
# Example only: replace these with exactly the api_key_env names declared by
# the selected config; use an empty array when every endpoint is unauthenticated.
CDF_ENDPOINT_KEY_ARGS=()
# Authenticated example:
# CDF_ENDPOINT_KEY_ARGS=(-e VLM_API_KEY -e LLM_API_KEY -e VEO_API_KEY)
# Forward only credential names required by the configured remote media. Give
# checker containers their own scoped credentials in their deployment.
CDF_STORAGE_CREDENTIAL_ARGS=()
# S3-compatible example (include only the names your storage config requires):
# CDF_STORAGE_CREDENTIAL_ARGS=(-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
# -e AWS_DEFAULT_REGION -e AWS_ENDPOINT_URL)
docker run -it --rm \
--network cdf \
"${CDF_ENDPOINT_KEY_ARGS[@]}" \
"${CDF_STORAGE_CREDENTIAL_ARGS[@]}" \
-v "$(pwd)/modules:/workspace/modules" \
-v "$(pwd)/configs:/workspace/configs" \
-v "$(pwd)/data:/workspace/data" \
--entrypoint /bin/bash \
"$CDF_IMAGE_ID"
Persist the captured sha256: ID and use it directly for every later launch;
never re-resolve the mutable tag as the launch target. For registry releases,
verify the signed manifest and use name:tag@sha256:<manifest-digest>.
- Networking: augmentation is an outbound-only client and does not publish
ports. Use the user-defined
cdf bridge for the augmentation container and
every local model, Arbitrator, and checker container. cosmos-evaluator may
be a container/DNS hostname on that bridge; it is not a second bridge name.
Change endpoint URLs from host localhost addresses to container DNS names,
for example http://vlm:8000/v1. The default bridge is sufficient when every
endpoint is remote.
- API keys: prefer a platform secrets manager that injects the required
environment variables. Otherwise, export only the required keys and forward
their names with
-e VAR_NAME; never mount or load a broad credential file.
- No GPU needed for remote inference — add
--gpus for data_processing.alignment and any H.264 decode; pick a GPU not shared with a busy model server. Container runs as uid 10000; ensure data/ is writable (or --user "$(id -u):$(id -g)").
Warning: Never enable Docker host-network mode on shared, multi-tenant, or
production hosts. It is permitted only when all three conditions hold: (1) a
required host-local service cannot be moved onto the cdf bridge, (2) the
host is single-tenant and isolated with no other workloads, and (3) the user
sends an explicit message in the current conversation choosing host networking
after receiving a warning that it removes network namespace isolation. Silence,
prior documentation, or an earlier unrelated approval is not acceptance. Do
not select it automatically. Review
pipeline-operations.md.
Step 2: Validate, Then Run (Inside the Container)
# Schema and cross-section preflight only: no endpoint calls or inference.
uv run --no-sync modules/cli.py --config configs/<config_file>.yaml --validate-only
# Run only after validation succeeds.
uv run --no-sync modules/cli.py --config configs/<config_file>.yaml
# With OmegaConf CLI overrides (dot-list syntax)
uv run --no-sync modules/cli.py --config configs/<config_file>.yaml \
data.0.inputs.rgb=/workspace/data/input.mp4 \
augmentation.parameters.seed=42
Environment variables: generation/captioning keys resolve as the api_key_env
var → the role's default env var. External Cosmos evaluation reads only the
variable explicitly named by its endpoint and fails initialization when that
variable is unset. Leave api_key_env off for unauthenticated endpoints.
LOG_LEVEL sets logging.
Configuration Schema
Configs are validated against PipelineConfig (modules/aug_utils/schema/) and have seven top-level sections. Author them in the canonical order above; YAML key order does not change runtime semantics. Full per-section YAML is in configuration-schema.md; runtime flow and common editing tasks are in pipeline-operations.md.
Troubleshooting
Run all inference and schema validation inside the Docker container for a consistent environment. For config-validation errors, runtime/endpoint errors, and typical per-stage timings, see troubleshooting.md.
Limitations
- Remote inference only. All models run behind remote HTTP endpoints; no local weights, no
torchrun, no executor_type, no Gradio executor.
- GPU for configured local stages. Remote inference needs no augmentation-container GPU. A CUDA GPU is required by
data_processing.alignment (cupy) and by any local stage decoding H.264 because the image ships only the hardware h264_cuvid decoder (software AVC decode is off for licensing). VP9 decodes in software. Video output is VP9-only.
- External-evaluator media must be remote. When an external evaluator is enabled, schema validation rejects local
data[].inputs.rgb, control media, generated data[].output.video, and configured companion media. Use checker-readable s3://, msc://, gs://, az://, or HTTP(S) URIs; the pipeline does not upload local files solely for evaluation.
- Inference only. This pipeline augments and generates media — it does not train or fine-tune models.
- Auth varies by endpoint. Hosted endpoints (e.g. Veo) need a key via
api_key_env; local endpoints (e.g. vLLM) need none.
Reference files
- configuration-schema.md — full per-section YAML for every config section.
- config-decision-tree.md — which config to start from, model/captioning selection, alignment override rules.
- pipeline-operations.md — pipeline flow, worked example, common tasks, storage, security notes.
- captioning-strategy-guide.md — all 6 captioning modes with complete YAML.
- evaluator-setup-guide.md — built-in evaluator tuning plus external Cosmos discovery, gating, and failure semantics.
- troubleshooting.md — validation/runtime errors and per-stage timings.
- image-attribute-augmentation.md — Image Attribute Augmentation image-edit workflow and dataset packaging.
- event-video-gen.md — smart-space image-to-video event generation.
1---2name: cdf-augmentation3description: >-4 Use when authoring or validating Cosmos Data Factory (CDF) augmentation YAML5 configs, or running remote Cosmos Transfer (including Cosmos3 WSM controls),6 Cosmos Predict, image-edit, or image-to-video inference.7license: Apache-2.08metadata:9 source: https://skillmd.com/skills/nvidia/cdf-augmentation10 owner: NVIDIA11 service: cosmos-data-factory12 version: 1.2.013 reviewed: '2026-09-16'14 author: "NVIDIA <opensource@nvidia.com>"15 tags:16 - physical-ai17 - augmentation18 - cosmos19 - wsm20 - image-edit21---2223# CDF Augmentation Pipeline Skill2425Remote-API pipeline for captioning, generating, and evaluating augmented camera26data. Models are configured as HTTP endpoints; no local model weights are27included.2829## Purpose3031Use it to select a supported generation mode, author and validate32`PipelineConfig` YAML, configure captioning/evaluators, and run the33`cdf-augmentation:1.2.0` container. It covers Cosmos Transfer/Predict,34Cosmos3 WSM controls, image editing, image-to-video, BYOM endpoints, and35quality gates.3637Do **not** use this skill for training or fine-tuning models, deploying clusters or NIM endpoints, or unrelated application/database development.3839## Prerequisites4041| Requirement | Detail |42|-------------|--------|43| **Docker** | `docker --version`. The image is **remote-API only** — it bundles no Cosmos/torch weights, so plain remote inference needs **no GPU and no `HF_TOKEN`**. |44| **NVIDIA GPU** (conditional) | Only when a configured local stage requires CUDA or decodes **H.264** in the augmentation container. See Limitations. |45| **Endpoint URLs** | One reachable URL per role used by the selected config. The examples name local Qwen services (`Qwen/Qwen3.6-27B-FP8` on `vlm`, `Qwen/Qwen2.5-14B-Instruct` on `llm`), but those are not guaranteed to be running. Resolve every required role independently: keep each reachable configured endpoint and ask only for the missing role URLs. |46| **API keys** (conditional) | Only for endpoints requiring authentication. Pass the environment variable named by each endpoint's `api_key_env`; never hardcode values in YAML. Local unauthenticated endpoints need none. |47| **Input media** | A video/image reachable by `multistorageclient` (`s3://`, `msc://`, `gs://`, `az://`, or HTTPS). Local paths are allowed **only when no external Cosmos evaluator is enabled**; external evaluation requires checker-readable remote URIs. |48| **Approved release commit** (execution only) | A 40-character commit hash confirmed by the user as release-owner-approved or retrieved from the verified release manifest. Never invent it or rely on an unverified environment value. |4950## Inputs5152Resolve each value in this precedence order: **state file → explicit prompt arguments → agent context → user prompt.** Ask the user only for what remains unresolved.5354| Input | Required | Description |55|-------|----------|-------------|56| `config_path` | Yes | Path to the pipeline YAML, e.g. `configs/cookbook/video-data-augmentation/config_video_transfer_CT3_omni.yaml`. If absent, pick a starting config from *Supported Models* and confirm with the user. |57| `input_media` | Conditional | Source video/image → `data[].inputs.rgb`. Required for every mode except Cosmos Predict `inference_type: text2world`, where `inputs` may be null or `rgb` omitted. Overridable at run time via `data.0.inputs.rgb=...`. |58| `control_media` | For precomputed control | Control video → `data[].inputs.controls.<type>`. For Cosmos3 WSM transfer, set `controls.wsm`; the client uploads it rather than exposing its local path to the server. |59| `output_paths` | Yes | `data[].output.{video,caption,metadata}`; `evaluation` optional. |60| `model_name` | Yes | `augmentation.model.name` — an endpoint `id`, a role, or a known model name. Free-form string, not an enum. |61| `endpoint_urls` | Yes | One `endpoints[]` entry per role in use. |62| `api_key_env` | If auth | Env-var *name* per endpoint; the value comes from the environment. |63| `target_attributes` | No | `captioning.llm.variables` (e.g. `weather_condition`, `lighting_condition`). |64| `generation_params` | No | `augmentation.parameters` — pass-through; only set knobs are sent. |65| `seed` | No | Under `augmentation.parameters`. Before the first candidate, `null` resolves once to the generator/executor seed when available, otherwise the current Unix time. Retry `n` uses that resolved base seed plus `n`. `pipeline.retry` defaults to `1`, so evaluation permits at most 2 candidates by default. |6667## BYOM model: endpoints, adapters, roles6869Each `endpoints:` list entry declares `role`, `url`, wire `model`, and optional70`id`, `adapter`, `api_key_env`, and `timeout`. Roles are `vlm`, `llm`,71`image_edit`, `video_transfer`, `video_predict`, `image2video`, and `evaluator`.72`augmentation.model.name` resolves by endpoint `id`, then role, then the known73model-name mapping. Adapter contracts and full fields are in74[configuration-schema.md](references/configuration-schema.md).7576## Supported Models7778When the user hasn't specified a model, choose from their **input type and goal**:7980| Input Type → Goal | `model.name` | Role / default adapter | Input → Output |81|-------------------|--------------|------------------------|----------------|82| Video — change scene attributes (weather, lighting, style) | `cosmos-transfer2.5` | `video_transfer` / `nim` | Video (+ controls) → Video |83| Video + precomputed WSM — follow world-state geometry/motion | A `video_transfer` endpoint id, e.g. `cosmos3-transfer-wsm` | `video_transfer` / `openai.video.async` | RGB video + WSM control + prompt → Video |84| Video + text — extend or predict continuation | `cosmos-predict` | `video_predict` / `nim` | Video+Text → Video |85| Text only — generate video from scratch | `cosmos-predict` (`inference_type: text2world`) | `video_predict` / `nim` | Text → Video |86| Image — edit specific attributes | `image-edit` | `image_edit` / `nim` (or `openai.chat.completions`, `openai.images.edits`) | Image → Image |87| Image — animate a first frame | `cosmos3-image2video` (or your Veo endpoint id) | `image2video` / `openai.video.sync` (Veo: `openai.video.async`) | Image + prompt → Video |8889**Key rule:** use Cosmos Transfer for video scene changes, Cosmos Predict for90new/continued video, image edit for still edits, and image-to-video to animate a91frame. For Cosmos3 WSM, set `data[].inputs.controls.wsm`. Config selection92details are in [config-decision-tree.md](references/config-decision-tree.md).9394All models run via remote HTTP through one `BaseExecutor`; there is no local `torchrun` and no `executor_type` field.9596## Canonical Workflow9798Follow this ordered decision tree; open the linked references only for the99selected mode's details.100101### Config Correction Budget102103`config_attempts` counts YAML versions, not validation or preflight calls. Keep104it across the whole workflow and never reset it.1051061. Set it to `1` after initial authoring.1072. On a correctable error, if it is already `3`, **STOP** and report the error.108 Otherwise increment it once and make one correction pass containing all109 currently reported YAML fixes.1103. Validate and preflight that edited version without another increment. Repeat111 rule 2 only if a later check finds another correctable YAML error.112113### STEP 0 — Mandatory Entry Gate114115Execute this gate before every other step:116117- **Scope gate:** if the request asks for training, fine-tuning, cluster/NIM118 deployment, or unrelated development, decline it and **STOP**.119- **Input gate:** resolve values using *Inputs*. If required values remain120 unresolved, request only those values and **STOP** until they are supplied.121- Proceed only when both gates pass and the input type and augmentation goal122 are known.1231241. Select the model from *Supported Models* using that input type and goal.1252. Author the YAML in this order: `data`, `endpoints`, `captioning`,126 `augmentation`, `evaluators`, `pipeline`, then `data_processing`. Omit127 optional sections rather than creating placeholders. Set128 `config_attempts = 1` (**config attempt 1 of 3**).1293. Branch on the requested outcome:130 - **Configuration only or “do not execute”:** do not build, launch, run131 preflight, or infer. A trusted runtime is an already-running container132 whose recorded immutable image ID/digest, checkout commit, and config133 mounts satisfy step 4. If validation was explicitly requested and such a134 runtime exists, run only the validation command and apply step 5's failure135 rules. Return the config (mark it unvalidated if validation was not run)136 and **STOP**.137 - **Inference/run requested:** proceed to step 4.1384. Prepare the runtime in these verifiable sub-steps:139 a. Obtain `EXPECTED_RELEASE_REF` from the user as a release-owner-approved140 full commit or retrieve it from the verified release manifest.141 b. Verify the Git working tree is clean.142 c. Verify `HEAD` matches `EXPECTED_RELEASE_REF`. If a release tag selected143 the revision, verify the tag first, then compare its resolved full commit.144 d. Build the image and capture its immutable `sha256:` image ID, or obtain a145 verified registry digest.146 e. Create the `cdf` Docker bridge if needed and attach local services.147 f. Launch the container using the recorded image ID, the `cdf` network,148 required volume mounts, `--entrypoint /bin/bash`, and only the endpoint149 key variables named by `api_key_env` plus the scoped storage credential150 variables required for the configured remote media.151 If any sub-step fails, **STOP** and report the exact failure and relevant152 remediation: clean a dirty tree, supply the approved revision, resolve a153 revision mismatch, check the Docker daemon/build, fix network setup, or fix154 the container launch. Do not continue with a partial runtime.1555. Inside the container, run the validation-only command in Usage Step 2 and156 classify its result while preserving `config_attempts` (**current config157 attempt N of 3**):158 - **Success:** if no external Cosmos evaluator is enabled, proceed directly159 to step 7. If one is enabled, proceed to step 6.160 - **Editable YAML/schema error:** when required endpoint URL/model details are161 known, apply the *Config Correction Budget* rule, then rerun validation on162 that YAML version.163 - **Infrastructure/unknown input:** for missing endpoint details, credentials,164 DNS, connectivity, or unhealthy services, do not edit speculatively.165 **STOP** and report the error or ask for the missing endpoint details.1666. With external Cosmos evaluation enabled, preserve `config_attempts`167 (**current config attempt N of 3**) and run its hard preflight gate before168 captioning/generation by querying `/health`, `/checkers`, and169 `/dependency-graph`:170 - **Pass:** require `/health.status == "healthy"`; require every configured171 `checks[].name` to exactly equal one string in the top-level `checkers`172 array returned by `/checkers` and one key in the top-level `checkers`173 object returned by `/dependency-graph`. Do not use substring, alias, or174 nested-field matching. Then proceed to step 7.175 - **Config error:** only a 4xx response from `/checkers` or176 `/dependency-graph` whose body explicitly identifies a configured checker177 name or invalid config field qualifies. Apply the *Config Correction178 Budget* rule, then return to step 5; revalidation does not increment the179 counter.180 - **Infrastructure failure:** any timeout, network/DNS error, 5xx response,181 `/health` failure, unhealthy service, or missing deployed checker stops the182 workflow before generation. Create no candidate and report the root cause.1837. Enter inference only after step 5 succeeds and, when external evaluation is184 enabled, step 6 passes. Run a candidate loop bounded to185 **`pipeline.retry + 1` candidates** (`retry` defaults to `1`, so the default186 maximum is 2). Resolve the base seed once before the first candidate as187 defined in *Inputs*; retry `n` uses `base_seed + n`:188 - Generate a candidate and run its configured evaluators.189 - On pass, accept it and exit the loop.190 - On quality failure with attempts remaining, set the next seed to191 `base_seed + next_retry_number`. If `regenerate_caption_on_retry: true`,192 regenerate the caption next; then generate the replacement. If false,193 generate the replacement immediately.194 - On exhausted quality retries, fail evaluation and exit the loop.195 - On a service/contract failure, stop on the current candidate without196 generating another. For external Cosmos evaluation, this means a197 submission, polling, or discovery failure; a failed execution; a missing198 or skipped result; or an invalid gating contract without a literal199 top-level boolean `passed`. With `pipeline.evaluation.strict: true` (the200 default), apply `retain_failures`: its default `true` retains video, caption,201 metadata, and evaluation outputs for inspection; explicit `false` deletes202 them. With `strict: false`, retain the files and report the evaluator203 failure without failing the sample.204 - If provider metadata contains `error_code: ambiguous_post`, preserve the205 retained candidate, do not resubmit automatically, **STOP**, and defer206 to operator recovery. Follow the207 [ambiguous-POST recovery flow](references/evaluator-setup-guide.md#gating-and-failures).208209## Usage210211### Step 1: Verify, Build, and Launch the Docker Container212213Set `EXPECTED_RELEASE_REF` to a release-owner-reviewed full 40-character commit,214never a tag or branch. If a signed release tag selects the revision, verify the215tag first and record its resolved full commit as `EXPECTED_RELEASE_REF`. Build216only from a clean matching checkout, then record the resulting immutable image217ID. For a prebuilt release, use its verified digest.218219```bash220set -e221222EXPECTED_RELEASE_REF="${EXPECTED_RELEASE_REF:?set a reviewed full commit ID}"223test "${#EXPECTED_RELEASE_REF}" -eq 40224case "$EXPECTED_RELEASE_REF" in *[!0-9a-fA-F]*) exit 1 ;; esac225test -z "$(git status --porcelain)"226test "$(git rev-parse HEAD)" = "$(git rev-parse "${EXPECTED_RELEASE_REF}^{commit}")"227228DOCKER_BUILDKIT=0 docker build \229 -t cdf-augmentation:1.2.0 \230 -f docker/Dockerfile .231CDF_IMAGE_ID="$(docker image inspect --format '{{.Id}}' cdf-augmentation:1.2.0)"232case "$CDF_IMAGE_ID" in sha256:*) ;; *) exit 1 ;; esac233test "$(docker image inspect --format '{{.Id}}' cdf-augmentation:1.2.0)" = "$CDF_IMAGE_ID"234235docker network inspect cdf >/dev/null 2>&1 || \236 docker network create cdf237# Attach each local model container once, for example:238# docker network connect cdf vlm239240# Example only: replace these with exactly the api_key_env names declared by241# the selected config; use an empty array when every endpoint is unauthenticated.242CDF_ENDPOINT_KEY_ARGS=()243# Authenticated example:244# CDF_ENDPOINT_KEY_ARGS=(-e VLM_API_KEY -e LLM_API_KEY -e VEO_API_KEY)245246# Forward only credential names required by the configured remote media. Give247# checker containers their own scoped credentials in their deployment.248CDF_STORAGE_CREDENTIAL_ARGS=()249# S3-compatible example (include only the names your storage config requires):250# CDF_STORAGE_CREDENTIAL_ARGS=(-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \251# -e AWS_DEFAULT_REGION -e AWS_ENDPOINT_URL)252253docker run -it --rm \254 --network cdf \255 "${CDF_ENDPOINT_KEY_ARGS[@]}" \256 "${CDF_STORAGE_CREDENTIAL_ARGS[@]}" \257 -v "$(pwd)/modules:/workspace/modules" \258 -v "$(pwd)/configs:/workspace/configs" \259 -v "$(pwd)/data:/workspace/data" \260 --entrypoint /bin/bash \261 "$CDF_IMAGE_ID"262```263264Persist the captured `sha256:` ID and use it directly for every later launch;265never re-resolve the mutable tag as the launch target. For registry releases,266verify the signed manifest and use `name:tag@sha256:<manifest-digest>`.267268- **Networking:** augmentation is an outbound-only client and does not publish269 ports. Use the user-defined `cdf` bridge for the augmentation container and270 every local model, Arbitrator, and checker container. `cosmos-evaluator` may271 be a container/DNS hostname on that bridge; it is not a second bridge name.272 Change endpoint URLs from host `localhost` addresses to container DNS names,273 for example `http://vlm:8000/v1`. The default bridge is sufficient when every274 endpoint is remote.275- **API keys:** prefer a platform secrets manager that injects the required276 environment variables. Otherwise, export only the required keys and forward277 their names with `-e VAR_NAME`; never mount or load a broad credential file.278- **No GPU needed for remote inference** — add `--gpus` for `data_processing.alignment` and any **H.264** decode; pick a GPU not shared with a busy model server. Container runs as uid 10000; ensure `data/` is writable (or `--user "$(id -u):$(id -g)"`).279280> **Warning:** Never enable Docker host-network mode on shared, multi-tenant, or281> production hosts. It is permitted only when all three conditions hold: (1) a282> required host-local service cannot be moved onto the `cdf` bridge, (2) the283> host is single-tenant and isolated with no other workloads, and (3) the user284> sends an explicit message in the current conversation choosing host networking285> after receiving a warning that it removes network namespace isolation. Silence,286> prior documentation, or an earlier unrelated approval is not acceptance. Do287> not select it automatically. Review288> [pipeline-operations.md](references/pipeline-operations.md#security-notes).289290### Step 2: Validate, Then Run (Inside the Container)291292```bash293# Schema and cross-section preflight only: no endpoint calls or inference.294uv run --no-sync modules/cli.py --config configs/<config_file>.yaml --validate-only295296# Run only after validation succeeds.297uv run --no-sync modules/cli.py --config configs/<config_file>.yaml298299# With OmegaConf CLI overrides (dot-list syntax)300uv run --no-sync modules/cli.py --config configs/<config_file>.yaml \301 data.0.inputs.rgb=/workspace/data/input.mp4 \302 augmentation.parameters.seed=42303```304305Environment variables: generation/captioning keys resolve as the `api_key_env`306var → the role's default env var. External Cosmos evaluation reads only the307variable explicitly named by its endpoint and fails initialization when that308variable is unset. Leave `api_key_env` off for unauthenticated endpoints.309`LOG_LEVEL` sets logging.310311## Configuration Schema312313Configs are validated against `PipelineConfig` (`modules/aug_utils/schema/`) and have seven top-level sections. Author them in the canonical order above; YAML key order does not change runtime semantics. Full per-section YAML is in [configuration-schema.md](references/configuration-schema.md); runtime flow and common editing tasks are in [pipeline-operations.md](references/pipeline-operations.md).314315## Troubleshooting316317Run all inference and schema validation **inside the Docker container** for a consistent environment. For config-validation errors, runtime/endpoint errors, and typical per-stage timings, see [troubleshooting.md](references/troubleshooting.md).318319## Limitations320321- **Remote inference only.** All models run behind remote HTTP endpoints; no local weights, no `torchrun`, no `executor_type`, no Gradio executor.322- **GPU for configured local stages.** Remote inference needs no augmentation-container GPU. A CUDA GPU is required by `data_processing.alignment` (cupy) and by any local stage decoding H.264 because the image ships only the hardware `h264_cuvid` decoder (software AVC decode is off for licensing). VP9 decodes in software. Video **output** is VP9-only.323- **External-evaluator media must be remote.** When an external evaluator is enabled, schema validation rejects local `data[].inputs.rgb`, control media, generated `data[].output.video`, and configured companion media. Use checker-readable `s3://`, `msc://`, `gs://`, `az://`, or HTTP(S) URIs; the pipeline does not upload local files solely for evaluation.324- **Inference only.** This pipeline augments and generates media — it does not train or fine-tune models.325- **Auth varies by endpoint.** Hosted endpoints (e.g. Veo) need a key via `api_key_env`; local endpoints (e.g. vLLM) need none.326327## Reference files328329- [configuration-schema.md](references/configuration-schema.md) — full per-section YAML for every config section.330- [config-decision-tree.md](references/config-decision-tree.md) — which config to start from, model/captioning selection, alignment override rules.331- [pipeline-operations.md](references/pipeline-operations.md) — pipeline flow, worked example, common tasks, storage, security notes.332- [captioning-strategy-guide.md](references/captioning-strategy-guide.md) — all 6 captioning modes with complete YAML.333- [evaluator-setup-guide.md](references/evaluator-setup-guide.md) — built-in evaluator tuning plus external Cosmos discovery, gating, and failure semantics.334- [troubleshooting.md](references/troubleshooting.md) — validation/runtime errors and per-stage timings.335- [image-attribute-augmentation.md](references/image-attribute-augmentation.md) — Image Attribute Augmentation image-edit workflow and dataset packaging.336- [event-video-gen.md](references/event-video-gen.md) — smart-space image-to-video event generation.