Prepare a FastGen dataset
Apply fastgen-runtime. The
dataset README owns formats, preparation commands,
loader presets and key parameters; the loaders in
wds_dataloaders.py and
wds_utils.py document decoding, partitioning and
resume in their docstrings. Read loader contracts for
shard acceptance, coverage and resume accounting.
- Scope. Resolve source, permitted use, destination, task, the method's supervision
needs (fastgen-configure if unsettled), the
network's preprocessing and the available sample count. Inspect existing inputs before
downloading; keep provenance and counts. Follow the pretrained model's conditioning
requirements; recaption prompts to match its reference style when needed, preserving meaning.
- Loader. Choose the existing class-conditional or WebDataset loader. Shards need
unique sample keys with the extensions named in
key_map; constants come from
files_map and presets_map. Raw media and prompts use the *_raw keys and cached
latents and embeddings the plain keys; never label placeholders as model-space data,
and record the model revision and preprocessing of every cache. Rebuild affected caches
after edits.
- Pairing and geometry. Keep task pairing (source and target for editing, reference
frames for image-conditioned video, camera data, aligned audio). Derive frame counts,
FPS, aspect ratio and temporal compression from the network; the representable frame
count is not the nominal clip length.
- Splits and size. Separate training from held-out evaluation inputs before deriving
labels or trajectories. Estimate the unique examples needed from throughput, effective
batch, updates and acceptable repetition; a tiny dataset validates plumbing only.
- Register. Import and copy a preset from data.py
rather than reproducing its loader fields; add a shared preset there when needed.
Supply paths through the config or environment roots; keep data and preprocessing
artifacts outside the source tree. For data-dependent fine-tuning or distillation, consider
teacher-generated targets via fastgen-inference.
Save the method's required supervision, including
KD pairs or trajectories.
- Verify. Iterate the real loader and a prepared batch (
prepare_batch) in the
proper runtime; check keys, shapes, dtypes, ranges, captions, pairing, negative
conditions and modality lengths. Use enough batches and ranks to expose empty worker
or shard splits, and keep a manifest of usable counts. Before prompt-only training,
sample the teacher on a few real training prompts to confirm they are usable.
Report provenance, the split/count manifest, dataset location, data-config path and the
observed batch contract. For empty or short batches inspect tar members, key_map,
filters, shard/worker partitioning and files_map before touching the network; report
corrupt or excluded samples instead of counting them.
1---2name: fastgen-dataset3description: >-4 Prepare, register or repair datasets for NVIDIA FastGen loaders: class-conditional ZIPs, WebDataset shards of images, prompts, latents, videos, audio or camera data, and KD pairs or trajectories; then verify decoded batches. For experiment configuration over already prepared data, use fastgen-configure.5license: Apache-2.06metadata:7 source: https://skillmd.com/skills/nvidia/fastgen-dataset8 author: "Julius Berner <jberner@nvidia.com>"9 domain: ml10---1112# Prepare a FastGen dataset1314Apply [fastgen-runtime](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-runtime/SKILL.md). The15[dataset README](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/README.md) owns formats, preparation commands,16loader presets and key parameters; the loaders in17[wds_dataloaders.py](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/wds_dataloaders.py) and18[wds_utils.py](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/wds_utils.py) document decoding, partitioning and19resume in their docstrings. Read [loader contracts](references/loader-contract.md) for20shard acceptance, coverage and resume accounting.21221. **Scope.** Resolve source, permitted use, destination, task, the method's supervision23 needs ([fastgen-configure](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-configure/SKILL.md) if unsettled), the24 network's preprocessing and the available sample count. Inspect existing inputs before25 downloading; keep provenance and counts. Follow the pretrained model's conditioning26 requirements; recaption prompts to match its reference style when needed, preserving meaning.272. **Loader.** Choose the existing class-conditional or WebDataset loader. Shards need28 unique sample keys with the extensions named in `key_map`; constants come from29 `files_map` and `presets_map`. Raw media and prompts use the `*_raw` keys and cached30 latents and embeddings the plain keys; never label placeholders as model-space data,31 and record the model revision and preprocessing of every cache. Rebuild affected caches32 after edits.333. **Pairing and geometry.** Keep task pairing (source and target for editing, reference34 frames for image-conditioned video, camera data, aligned audio). Derive frame counts,35 FPS, aspect ratio and temporal compression from the network; the representable frame36 count is not the nominal clip length.374. **Splits and size.** Separate training from held-out evaluation inputs before deriving38 labels or trajectories. Estimate the unique examples needed from throughput, effective39 batch, updates and acceptable repetition; a tiny dataset validates plumbing only.405. **Register.** Import and copy a preset from [data.py](https://github.com/NVlabs/FastGen/blob/main/fastgen/configs/data.py)41 rather than reproducing its loader fields; add a shared preset there when needed.42 Supply paths through the config or environment roots; keep data and preprocessing43 artifacts outside the source tree. For data-dependent fine-tuning or distillation, consider44 teacher-generated targets via [fastgen-inference](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-inference/SKILL.md).45 Save the method's required supervision, including46 [KD pairs or trajectories](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/README.md#knowledge-distillation-loaders).476. **Verify.** Iterate the real loader and a prepared batch (`prepare_batch`) in the48 proper runtime; check keys, shapes, dtypes, ranges, captions, pairing, negative49 conditions and modality lengths. Use enough batches and ranks to expose empty worker50 or shard splits, and keep a manifest of usable counts. Before prompt-only training,51 sample the teacher on a few real training prompts to confirm they are usable.5253Report provenance, the split/count manifest, dataset location, data-config path and the54observed batch contract. For empty or short batches inspect tar members, `key_map`,55filters, shard/worker partitioning and `files_map` before touching the network; report56corrupt or excluded samples instead of counting them.