Add a FastGen modality
Apply fastgen-develop
and fastgen-runtime. Read the
network and dataset
READMEs, then modality boundaries. Use
fastgen-add-network alone for another model of
an already supported task.
- Role. Establish whether the modality is generated state, conditioning or control,
or a derived training target, from the upstream model and its source. Record
representation, units, normalisation, dimensions, temporal alignment, masks and decode
semantics; a checkpoint name or an extra channel is not evidence of support.
- Trace. Follow one sample through loader,
prepare_batch, network, method loss,
generator and output writer; note which boundaries already handle it and which assume
image, video, audio or a single tensor. Preserve raw-versus-encoded distinctions and
upstream packing, and do not turn a conditioning signal into diffusion state to fit an
existing multimodal example.
- Model. Use fastgen-add-network for encoder
and backbone changes; add only the needed shape inference, schedule dispatch and
conditioning paths. For joint generation check whether modalities share a clock, grid
and batch and keep the upstream relationship.
- Data. Use fastgen-dataset for acquisition,
serialization, pairing and loader registration; version learned normalisation with the
model artifacts and verify encode/decode and alignment before large preprocessing.
- Methods. Use fastgen-configure for SFT and
the requested distillation recipes; trace each method's tensor operations, since a
tree utility accepting a key does not make the method support it. Reject unsupported
combinations explicitly.
- Outputs. Extend an inference entry point and serializer where the existing ones
assume conventional media; keep native arrays in a usable format with shape and unit
metadata, separate from rendered previews, tracker logging and metrics.
- Validate. Use fastgen-validate for parity and
reviewable comparisons, then fastgen-train for short
SFT and the requested method; check per-modality gradients, conditioning sensitivity,
serialization of every generated modality, checkpoint reload and that existing modes
still work.
Report supported input and output modes, implementation, data and config paths, native
artifacts and evidence per tested combination. For plausible but wrong outputs check
units, packing, alignment and normalisation before tuning losses, and state whether the
result is a new conditioning path or a newly generatable modality.
1---2name: fastgen-add-modality3description: >-4 Extend NVIDIA FastGen to a new generated or conditioning modality beyond the existing image, video, audio, camera and text paths: representation, loader keys, model state, noise-schedule clock, losses, sampling, serialization and validation. Use it when a new signal type must cross the data, model and output interfaces.5license: Apache-2.06metadata:7 source: https://skillmd.com/skills/nvidia/fastgen-add-modality8 author: "Julius Berner <jberner@nvidia.com>"9 domain: ml10---1112# Add a FastGen modality1314Apply [fastgen-develop](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-develop/SKILL.md)15and [fastgen-runtime](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-runtime/SKILL.md). Read the16[network](https://github.com/NVlabs/FastGen/blob/main/fastgen/networks/README.md) and [dataset](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/README.md)17READMEs, then [modality boundaries](references/modality-contract.md). Use18[fastgen-add-network](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-add-network/SKILL.md) alone for another model of19an already supported task.20211. **Role.** Establish whether the modality is generated state, conditioning or control,22 or a derived training target, from the upstream model and its source. Record23 representation, units, normalisation, dimensions, temporal alignment, masks and decode24 semantics; a checkpoint name or an extra channel is not evidence of support.252. **Trace.** Follow one sample through loader, `prepare_batch`, network, method loss,26 generator and output writer; note which boundaries already handle it and which assume27 image, video, audio or a single tensor. Preserve raw-versus-encoded distinctions and28 upstream packing, and do not turn a conditioning signal into diffusion state to fit an29 existing multimodal example.303. **Model.** Use [fastgen-add-network](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-add-network/SKILL.md) for encoder31 and backbone changes; add only the needed shape inference, schedule dispatch and32 conditioning paths. For joint generation check whether modalities share a clock, grid33 and batch and keep the upstream relationship.344. **Data.** Use [fastgen-dataset](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-dataset/SKILL.md) for acquisition,35 serialization, pairing and loader registration; version learned normalisation with the36 model artifacts and verify encode/decode and alignment before large preprocessing.375. **Methods.** Use [fastgen-configure](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-configure/SKILL.md) for SFT and38 the requested distillation recipes; trace each method's tensor operations, since a39 tree utility accepting a key does not make the method support it. Reject unsupported40 combinations explicitly.416. **Outputs.** Extend an inference entry point and serializer where the existing ones42 assume conventional media; keep native arrays in a usable format with shape and unit43 metadata, separate from rendered previews, tracker logging and metrics.447. **Validate.** Use [fastgen-validate](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-validate/SKILL.md) for parity and45 reviewable comparisons, then [fastgen-train](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-train/SKILL.md) for short46 SFT and the requested method; check per-modality gradients, conditioning sensitivity,47 serialization of every generated modality, checkpoint reload and that existing modes48 still work.4950Report supported input and output modes, implementation, data and config paths, native51artifacts and evidence per tested combination. For plausible but wrong outputs check52units, packing, alignment and normalisation before tuning losses, and state whether the53result is a new conditioning path or a newly generatable modality.