NVIDIA DiffusionHarmonizer (NuRec post-processing)
Purpose
Run NVIDIA DiffusionHarmonizer on rendered images from neural
reconstructions. DiffusionHarmonizer is a single-step,
temporally-aware image diffusion enhancer for NeRF / 3DGS /
NuRec-style renderings. It improves realism, reduces
reconstruction artifacts, and harmonizes inserted dynamic
objects with the surrounding scene.
When to Use / When NOT to Use
Use this skill when the user has rendered frames from NRE,
NuRec, 3DGS, NeRF, or a similar reconstruction pipeline and
wants to enhance, harmonize, evaluate, or optionally fine-tune
the DiffusionHarmonizer model.
Do NOT use this skill when:
- The user wants to train or render the 3D reconstruction itself
(use
nre).
- The user wants to convert raw sensor data to NCore V4 (use
ncore).
- The user wants a generic photo enhancer. DiffusionHarmonizer
is tuned for neural-reconstruction artifacts and
object-insertion failures.
- The user only wants NRE inline rendering with
--enable-difix. That remains an NRE runtime feature; use the
nre skill for the complete serve-grpc / render-grpc
command shape.
What changed from the older Fixer skill
This skill follows the public NVIDIA/harmonizer release, not
the older NGC JIT .pt artifact recipe. Use these public
release artifacts:
- Code: https://github.com/NVIDIA/harmonizer
- Model:
nvidia/Harmonizer on Hugging Face (the paper checkpoint
models/diffusion_harmonizer.pkl), plus the base
nvidia/Cosmos-Predict2-0.6B-Text2Image model that inference
also requires.
- Checkpoint download:
./download_checkpoints.sh from the repo
root. It fetches the Harmonizer checkpoints into models/
(diffusion_harmonizer.pkl, harmonizer_nontemporal.pt) and
the base Cosmos DiT + tokenizer into
src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/.
- Runtime: the
harmonizer-cosmos-env image built from
Dockerfile.cosmos (base nvcr.io/nvidia/pytorch:25.10-py3).
- Inference entry:
src/inference_pix2pix_turbo_harmonizer.py,
run from inside /work/src so it can import its sibling
modules.
- Evaluation entry:
src/evaluate_test_dataset.py
- Training entry:
src/train_pix2pix_turbo_harmonizer.py
Do not use the obsolete standalone recipe that downloads
nvidia/nre/nurec-fixer:cosmos_3dgut_fixer_harmonizer, mounts
harmonizer_temporal.pt, or runs inference_jit_harmonizer.py
inside nvcr.io/nvidia/pytorch:24.10-py3 unless the user
explicitly asks for that older beta artifact. Do not run
inference_pretrained_model.py; the current README documents
inference_pix2pix_turbo_harmonizer.py as the inference entry
point.
Background
DiffusionHarmonizer is described in DiffusionHarmonizer:
Bridging Neural Reconstruction and Photorealistic Simulation
with Online Diffusion Enhancer
(arXiv 2602.24096, CVPR 2026).
It distills a pretrained multi-step diffusion model into a
single-step enhancer designed for online simulation and offline
data cleanup.
Two operating modes:
- Offline: clean pseudo-training views rendered from a
reconstruction, then distill the improved views back into the
3D representation.
- Online: enhance frames during simulation/inference by
harmonizing color and lighting, reconstructing missing or
inconsistent shadows for inserted actors, and reducing residual
reconstruction artifacts.
The public model card describes DiffusionHarmonizer-cosmos-0.6B,
a Cosmos Predict2 Diffusion Transformer post-trained at
576x1024 input and output resolution.
Inputs
- code_dir — checkout of
https://github.com/NVIDIA/harmonizer.
- model_dir — checkpoints fetched by
./download_checkpoints.sh
from the repo root. It places the paper checkpoint at
models/diffusion_harmonizer.pkl and the required base Cosmos
model under
src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/.
- input_dir — directory of rendered RGB frames (
.png,
.jpg, .jpeg) to enhance.
- output_dir —
inference_pix2pix_turbo_harmonizer.py does
not take an output flag; it writes to a sibling folder named
<input_dir>_<model_identifier> next to the input directory.
- HF_TOKEN — Hugging Face token with access to
nvidia/Harmonizer, nvidia/Cosmos-Predict2-0.6B-Text2Image,
and (if used) nvidia/Harmonizer-Dataset. Accept the
model/dataset license terms first.
- NGC_API_KEY — often needed to authenticate
docker pull
from nvcr.io. Use only for container pulls, not model
download.
Instructions
Validate the host. Have the agent execute
scripts/validate_setup.py via its standard script runner —
e.g. run_script("scripts/validate_setup.py") or
python scripts/validate_setup.py. It checks Docker, the
NVIDIA Container Toolkit, GPU architecture, git, the
Hugging Face CLI, token presence, and free disk space — and
exits non-zero on any missing prerequisite.
Clone the code and build (or pull) the runtime image.
Full commands and the Blackwell patch caveat live in
references/inference.md.
git clone https://github.com/NVIDIA/harmonizer.git
cd harmonizer
docker build -t harmonizer-cosmos-env -f Dockerfile.cosmos .
Download the checkpoints. From the repo root run the
helper, which fetches both the Harmonizer checkpoints and the
base Cosmos model into the paths the code expects.
export HF_TOKEN=<your-hugging-face-token>
hf auth login --token "$HF_TOKEN"
./download_checkpoints.sh
Verify models/diffusion_harmonizer.pkl and
src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/
exist.
Confirm input_dir exists and filenames sort into frame
order. The temporal inference script sorts frames with
natural sort and uses previous outputs as references; prefer
zero-padded names such as frame_000001.png.
Run inference inside the container with the repo mounted
at /work, then cd /work/src and run
inference_pix2pix_turbo_harmonizer.py. Pin
-u $(id -u):$(id -g) so outputs are owned by the host user.
Output frames land in
<input_dir>_<model_identifier>. Full docker run recipe and
flag matrix in
references/inference.md.
Validate that the output frame count matches the input
frame count, spot-check frames, and (if ground truth is
available) run paired evaluation with PSNR/LPIPS — see
references/evaluation.md.
(Optional) Train or fine-tune. Download the dataset (or
prepare JSON manifests in the documented format), then run
src/train_pix2pix_turbo_harmonizer.py with the recommended
hyperparameters. For fine-tuning, initialize from the released
checkpoint with --pretrained_path /path/to/diffusion_harmonizer.pkl. Full recipe + NuRec
data-pair recipes in
references/training.md.
(Optional) Teardown. Follow
references/teardown.md to remove
images, code clones, model weights, datasets, and outputs.
Examples
Example 1 — Enhance a folder of rendered frames
Run scripts/validate_setup.py, build the image once (see
references/wrapper-image.md), then
invoke inference_pix2pix_turbo_harmonizer.py inside the
harmonizer-cosmos-env container with the repo checkout mounted at
/work. From /work/src, point --input_image at the rendered
frames, --model_path at /work/models/diffusion_harmonizer.pkl,
set --model_identifier, and pass typical flags
--timestep 250 --resolution 1024 --use_sched. Enhanced frames are
written to <input_dir>_<model_identifier>. The canonical
docker run command and the full flag matrix live in
references/inference.md.
Example 2 — Quantitative PSNR / LPIPS evaluation
Prepare the paired test_dataset/{scene}/render +
test_dataset/{scene}/gt layout, then run
src/evaluate_test_dataset.py inside the container. See
references/evaluation.md for the
exact directory shape and the docker run command.
Example 3 — Fine-tune from the public checkpoint
Download nvidia/Harmonizer-Dataset, prepare the
training JSON, then run
src/train_pix2pix_turbo_harmonizer.py with the multi-GPU
accelerate launch recipe in
references/training.md. For
fine-tuning add --pretrained_path /path/to/diffusion_harmonizer.pkl and use
--fixing_data_weight 3 on the released dataset.
Example 4 — Non-temporal (frame-by-frame) enhancement
inference_pix2pix_turbo_harmonizer.py is temporal by default
and uses previous enhanced frames as references
(--offset_list -1 -2 -3 -4). When the user wants each frame
enhanced independently (e.g. unordered images), add
--nontemporal to disable temporal conditioning. Full command +
--offset_list defaults in
references/inference.md.
Consuming the inline Fixer via NRE
nre --enable-difix is still the right answer when the user
wants NRE to enhance frames as part of rendering without a
separate harmonizer checkout. That path is owned by the sibling
nre skill. Do not mix the standalone DiffusionHarmonizer
HF/Cosmos workflow with NRE's internal cache flags unless the
NRE documentation for the user's tag explicitly says they share
weights.
Use this standalone skill when the user wants the public
DiffusionHarmonizer code, model card, training/evaluation
scripts, or post-processing of frames that already exist on
disk.
Prerequisites
- OS: Linux host.
- GPU / driver: NVIDIA GPU Ampere or newer (compute
capability
>= 8.0; A100, A10, L40, H100, RTX 30/40/PRO,
B200, GB200).
- Container runtime: Docker with the NVIDIA Container
Toolkit.
- Tools:
git, python3, Hugging Face CLI (hf or
huggingface-cli).
- Secrets:
- Disk: at least ~120 GB free for the runtime image, build
cache, model weights, optional dataset, and outputs combined.
- Source / model / dataset:
The fail-fast check that enforces all of the above is
scripts/validate_setup.py.
Scripts
| Script |
Purpose |
Usage |
scripts/validate_setup.py |
Verify Docker, NVIDIA Container Toolkit, GPU architecture, git, Hugging Face CLI, token presence, and disk space. No network calls. |
run_script("scripts/validate_setup.py") or python scripts/validate_setup.py |
scripts/.env.example |
Template for HF_TOKEN and optional NGC_API_KEY. |
cp scripts/.env.example .env && set -a && . ./.env && set +a |
References
references/inference.md — container
build, raw-base fallback, Blackwell patches, checkpoint
download, inference_pix2pix_turbo_harmonizer.py flag matrix,
non-temporal mode.
references/evaluation.md — paired
test_dataset/ layout and evaluate_test_dataset.py command.
references/training.md — dataset
download, training JSON format, multi-GPU accelerate launch
recipe, fine-tuning flags, NuRec data-pair recipes.
references/wrapper-image.md —
build and run the project image for repeat inference.
references/troubleshooting.md
— extended diagnostic notes.
references/teardown.md — cleanup
inventory for images, code, Hugging Face caches, datasets,
and outputs.
- Public code: https://github.com/NVIDIA/harmonizer
- Model card: https://huggingface.co/nvidia/Harmonizer
- Dataset: https://huggingface.co/datasets/nvidia/Harmonizer-Dataset
- Paper: https://arxiv.org/abs/2602.24096
- Project page: https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/
Limitations
- Rendered inputs only. The model is tuned for
neural-reconstruction renderings and object-insertion
artifacts, not arbitrary real photos.
- Primary model resolution is 576x1024. The inference script
maps resolution key
1024 to 1024x576. Only 1024, 960,
and 1360 are supported keys; 1024 matches the model-card
operating point.
- Temporal references and filename order matter. The
inference script enhances frames in natural-sorted order and
feeds previous outputs back as temporal references. Use
zero-padded frame numbers, or pass
--nontemporal for
unordered images.
- Container builds can be large. The runtime image, build
cache, model weights, and optional dataset can exceed 100 GB.
- Training is multi-GPU by default. The README command
assumes 8 GPUs with bf16 mixed precision.
- Public code evolves. The current README documents
inference_pix2pix_turbo_harmonizer.py as the inference entry
point; if a future checkout renames it, prefer the script and
flags present in that checkout.
Troubleshooting (top 5)
| Error / symptom |
Most common cause |
docker: could not select device driver ... gpu |
NVIDIA Container Toolkit missing or Docker is not configured for the NVIDIA runtime. |
docker pull 401 / 403 from nvcr.io |
Docker is not authenticated to NGC, or the API key lacks container access. |
hf download ... 401 / 403 |
HF_TOKEN is missing/expired/lacks read scope, or the model/dataset license has not been accepted. |
diffusion_harmonizer.pkl missing |
Checkpoint download path is wrong or incomplete. Re-run ./download_checkpoints.sh from the repo root. |
Output files owned by root |
The docker run omitted -u $(id -u):$(id -g). |
Full matrix in
references/troubleshooting.md.
Teardown
A full workflow can leave large artifacts on disk: the Cosmos
image, project image, build cache, harmonizer code checkout,
Hugging Face model weights, optional dataset, evaluation
outputs, and enhanced frames. Reclaim them with the inventory
in references/teardown.md. Do not
revoke HF_TOKEN or NGC_API_KEY as normal cleanup. Rotate a
token only if you suspect it was leaked.
1---2name: nurec-fixer3description: >-4 Use to run NVIDIA DiffusionHarmonizer (public successor to the5 older Fixer recipes) to enhance, harmonize, evaluate, or6 fine-tune novel-view frames from NRE / NuRec / 3DGS / NeRF7 reconstructions. Do NOT use for training the 3D reconstruction8 itself (use `nre`) or for sensor-to-NCore conversion (use9 `ncore`).10version: "0.5.0"11tools:12 - Shell13 - Read14 - Write15license: CC-BY-4.0 AND Apache-2.016compatibility: >-17 Linux + NVIDIA GPU Ampere+ (compute capability >= 8.0; A100,18 A10, L40, H100, RTX 30/40/PRO, B200, GB200), Docker + NVIDIA19 Container Toolkit, ~120 GB free disk. HF_TOKEN required for the20 `nvidia/Harmonizer` checkpoints and the base21 `nvidia/Cosmos-Predict2-0.6B-Text2Image` model; NGC_API_KEY22 needed when pulling the `nvcr.io/nvidia/pytorch:25.10-py3` base23 image from nvcr.io. Code at github.com/NVIDIA/harmonizer;24 runtime is the `harmonizer-cosmos-env` image built from the25 project `Dockerfile.cosmos`, which layers cosmos-predict2 and26 the required patches on top of `nvcr.io/nvidia/pytorch:25.10-py3`.27dependencies:28 - bash29 - docker30 - nvidia-container-toolkit31 - python332 - git33 - huggingface_hub34metadata:35 source: https://skillmd.com/skills/nvidia/nurec-fixer36 author: NVIDIA NRS <nurec-skills@nvidia.com>37 tags:38 - nurec39 - fixer40 - harmonizer41 - diffusionharmonizer42 - reconstruction43 - autonomous-vehicles44 - post-processing45 - evaluation46 upstream: https://github.com/NVIDIA/harmonizer47 hf_model: https://huggingface.co/nvidia/Harmonizer48 hf_dataset: https://huggingface.co/datasets/nvidia/Harmonizer-Dataset49 container: nvcr.io/nvidia/pytorch:25.10-py350 paper: https://arxiv.org/abs/2602.2409651 product_page: https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/52 model_license: NVIDIA Open Model License Agreement53---5455# NVIDIA DiffusionHarmonizer (NuRec post-processing)5657## Purpose5859Run NVIDIA DiffusionHarmonizer on rendered images from neural60reconstructions. DiffusionHarmonizer is a single-step,61temporally-aware image diffusion enhancer for NeRF / 3DGS /62NuRec-style renderings. It improves realism, reduces63reconstruction artifacts, and harmonizes inserted dynamic64objects with the surrounding scene.6566## When to Use / When NOT to Use6768**Use this skill when** the user has rendered frames from NRE,69NuRec, 3DGS, NeRF, or a similar reconstruction pipeline and70wants to enhance, harmonize, evaluate, or optionally fine-tune71the DiffusionHarmonizer model.7273**Do NOT use this skill when:**7475- The user wants to train or render the 3D reconstruction itself76 (use `nre`).77- The user wants to convert raw sensor data to NCore V4 (use78 `ncore`).79- The user wants a generic photo enhancer. DiffusionHarmonizer80 is tuned for neural-reconstruction artifacts and81 object-insertion failures.82- The user only wants NRE inline rendering with83 `--enable-difix`. That remains an NRE runtime feature; use the84 `nre` skill for the complete `serve-grpc` / `render-grpc`85 command shape.8687## What changed from the older Fixer skill8889This skill follows the public `NVIDIA/harmonizer` release, not90the older NGC JIT `.pt` artifact recipe. Use these public91release artifacts:9293- Code: <https://github.com/NVIDIA/harmonizer>94- Model: `nvidia/Harmonizer` on Hugging Face (the paper checkpoint95 `models/diffusion_harmonizer.pkl`), plus the base96 `nvidia/Cosmos-Predict2-0.6B-Text2Image` model that inference97 also requires.98- Checkpoint download: `./download_checkpoints.sh` from the repo99 root. It fetches the Harmonizer checkpoints into `models/`100 (`diffusion_harmonizer.pkl`, `harmonizer_nontemporal.pt`) and101 the base Cosmos DiT + tokenizer into102 `src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/`.103- Runtime: the `harmonizer-cosmos-env` image built from104 `Dockerfile.cosmos` (base `nvcr.io/nvidia/pytorch:25.10-py3`).105- Inference entry: `src/inference_pix2pix_turbo_harmonizer.py`,106 run from inside `/work/src` so it can import its sibling107 modules.108- Evaluation entry: `src/evaluate_test_dataset.py`109- Training entry: `src/train_pix2pix_turbo_harmonizer.py`110111Do not use the obsolete standalone recipe that downloads112`nvidia/nre/nurec-fixer:cosmos_3dgut_fixer_harmonizer`, mounts113`harmonizer_temporal.pt`, or runs `inference_jit_harmonizer.py`114inside `nvcr.io/nvidia/pytorch:24.10-py3` unless the user115explicitly asks for that older beta artifact. Do not run116`inference_pretrained_model.py`; the current README documents117`inference_pix2pix_turbo_harmonizer.py` as the inference entry118point.119120## Background121122DiffusionHarmonizer is described in *DiffusionHarmonizer:123Bridging Neural Reconstruction and Photorealistic Simulation124with Online Diffusion Enhancer*125([arXiv 2602.24096](https://arxiv.org/abs/2602.24096), CVPR 2026).126It distills a pretrained multi-step diffusion model into a127single-step enhancer designed for online simulation and offline128data cleanup.129130Two operating modes:131132- **Offline:** clean pseudo-training views rendered from a133 reconstruction, then distill the improved views back into the134 3D representation.135- **Online:** enhance frames during simulation/inference by136 harmonizing color and lighting, reconstructing missing or137 inconsistent shadows for inserted actors, and reducing residual138 reconstruction artifacts.139140The public model card describes `DiffusionHarmonizer-cosmos-0.6B`,141a Cosmos Predict2 Diffusion Transformer post-trained at142`576x1024` input and output resolution.143144## Inputs145146- **code_dir** — checkout of147 <https://github.com/NVIDIA/harmonizer>.148- **model_dir** — checkpoints fetched by `./download_checkpoints.sh`149 from the repo root. It places the paper checkpoint at150 `models/diffusion_harmonizer.pkl` and the required base Cosmos151 model under152 `src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/`.153- **input_dir** — directory of rendered RGB frames (`.png`,154 `.jpg`, `.jpeg`) to enhance.155- **output_dir** — `inference_pix2pix_turbo_harmonizer.py` does156 not take an output flag; it writes to a sibling folder named157 `<input_dir>_<model_identifier>` next to the input directory.158- **HF_TOKEN** — Hugging Face token with access to159 `nvidia/Harmonizer`, `nvidia/Cosmos-Predict2-0.6B-Text2Image`,160 and (if used) `nvidia/Harmonizer-Dataset`. Accept the161 model/dataset license terms first.162- **NGC_API_KEY** — often needed to authenticate `docker pull`163 from `nvcr.io`. Use only for container pulls, not model164 download.165166## Instructions1671681. **Validate the host.** Have the agent execute169 `scripts/validate_setup.py` via its standard script runner —170 e.g. `run_script("scripts/validate_setup.py")` or171 `python scripts/validate_setup.py`. It checks Docker, the172 NVIDIA Container Toolkit, GPU architecture, `git`, the173 Hugging Face CLI, token presence, and free disk space — and174 exits non-zero on any missing prerequisite.1752. **Clone the code and build (or pull) the runtime image.**176 Full commands and the Blackwell patch caveat live in177 [`references/inference.md`](references/inference.md).178179 ```bash180 git clone https://github.com/NVIDIA/harmonizer.git181 cd harmonizer182 docker build -t harmonizer-cosmos-env -f Dockerfile.cosmos .183 ```1841853. **Download the checkpoints.** From the repo root run the186 helper, which fetches both the Harmonizer checkpoints and the187 base Cosmos model into the paths the code expects.188189 ```bash190 export HF_TOKEN=<your-hugging-face-token>191 hf auth login --token "$HF_TOKEN"192 ./download_checkpoints.sh193 ```194195 Verify `models/diffusion_harmonizer.pkl` and196 `src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/`197 exist.1984. **Confirm `input_dir` exists and filenames sort into frame199 order.** The temporal inference script sorts frames with200 natural sort and uses previous outputs as references; prefer201 zero-padded names such as `frame_000001.png`.2025. **Run inference inside the container** with the repo mounted203 at `/work`, then `cd /work/src` and run204 `inference_pix2pix_turbo_harmonizer.py`. Pin205 `-u $(id -u):$(id -g)` so outputs are owned by the host user.206 Output frames land in207 `<input_dir>_<model_identifier>`. Full `docker run` recipe and208 flag matrix in209 [`references/inference.md`](references/inference.md).2106. **Validate that the output frame count matches the input211 frame count,** spot-check frames, and (if ground truth is212 available) run paired evaluation with PSNR/LPIPS — see213 [`references/evaluation.md`](references/evaluation.md).2147. **(Optional) Train or fine-tune.** Download the dataset (or215 prepare JSON manifests in the documented format), then run216 `src/train_pix2pix_turbo_harmonizer.py` with the recommended217 hyperparameters. For fine-tuning, initialize from the released218 checkpoint with `--pretrained_path219 /path/to/diffusion_harmonizer.pkl`. Full recipe + NuRec220 data-pair recipes in221 [`references/training.md`](references/training.md).2228. **(Optional) Teardown.** Follow223 [`references/teardown.md`](references/teardown.md) to remove224 images, code clones, model weights, datasets, and outputs.225226## Examples227228### Example 1 — Enhance a folder of rendered frames229230Run `scripts/validate_setup.py`, build the image once (see231[`references/wrapper-image.md`](references/wrapper-image.md)), then232invoke `inference_pix2pix_turbo_harmonizer.py` inside the233`harmonizer-cosmos-env` container with the repo checkout mounted at234`/work`. From `/work/src`, point `--input_image` at the rendered235frames, `--model_path` at `/work/models/diffusion_harmonizer.pkl`,236set `--model_identifier`, and pass typical flags237`--timestep 250 --resolution 1024 --use_sched`. Enhanced frames are238written to `<input_dir>_<model_identifier>`. The canonical239`docker run` command and the full flag matrix live in240[`references/inference.md`](references/inference.md).241242### Example 2 — Quantitative PSNR / LPIPS evaluation243244Prepare the paired `test_dataset/{scene}/render` +245`test_dataset/{scene}/gt` layout, then run246`src/evaluate_test_dataset.py` inside the container. See247[`references/evaluation.md`](references/evaluation.md) for the248exact directory shape and the `docker run` command.249250### Example 3 — Fine-tune from the public checkpoint251252Download `nvidia/Harmonizer-Dataset`, prepare the253training JSON, then run254`src/train_pix2pix_turbo_harmonizer.py` with the multi-GPU255`accelerate launch` recipe in256[`references/training.md`](references/training.md). For257fine-tuning add `--pretrained_path258/path/to/diffusion_harmonizer.pkl` and use259`--fixing_data_weight 3` on the released dataset.260261### Example 4 — Non-temporal (frame-by-frame) enhancement262263`inference_pix2pix_turbo_harmonizer.py` is temporal by default264and uses previous enhanced frames as references265(`--offset_list -1 -2 -3 -4`). When the user wants each frame266enhanced independently (e.g. unordered images), add267`--nontemporal` to disable temporal conditioning. Full command +268`--offset_list` defaults in269[`references/inference.md`](references/inference.md).270271## Consuming the inline Fixer via NRE272273`nre --enable-difix` is still the right answer when the user274wants NRE to enhance frames as part of rendering without a275separate harmonizer checkout. That path is owned by the sibling276`nre` skill. Do not mix the standalone DiffusionHarmonizer277HF/Cosmos workflow with NRE's internal cache flags unless the278NRE documentation for the user's tag explicitly says they share279weights.280281Use this standalone skill when the user wants the public282DiffusionHarmonizer code, model card, training/evaluation283scripts, or post-processing of frames that already exist on284disk.285286## Prerequisites287288- **OS:** Linux host.289- **GPU / driver:** NVIDIA GPU Ampere or newer (compute290 capability `>= 8.0`; A100, A10, L40, H100, RTX 30/40/PRO,291 B200, GB200).292- **Container runtime:** Docker with the NVIDIA Container293 Toolkit.294- **Tools:** `git`, `python3`, Hugging Face CLI (`hf` or295 `huggingface-cli`).296- **Secrets:**297 - `HF_TOKEN` with the298 [`nvidia/Harmonizer`](https://huggingface.co/nvidia/Harmonizer)299 and300 [`nvidia/Cosmos-Predict2-0.6B-Text2Image`](https://huggingface.co/nvidia/Cosmos-Predict2-0.6B-Text2Image)301 licenses accepted (required to download model weights and302 the optional dataset).303 - `NGC_API_KEY` (often required for `docker login nvcr.io`304 before pulling `nvcr.io/nvidia/pytorch:25.10-py3`).305- **Disk:** at least ~120 GB free for the runtime image, build306 cache, model weights, optional dataset, and outputs combined.307- **Source / model / dataset:**308 - Code: <https://github.com/NVIDIA/harmonizer>.309 - Model: <https://huggingface.co/nvidia/Harmonizer>.310 - Base model:311 <https://huggingface.co/nvidia/Cosmos-Predict2-0.6B-Text2Image>.312 - Optional dataset:313 <https://huggingface.co/datasets/nvidia/Harmonizer-Dataset>.314315The fail-fast check that enforces all of the above is316`scripts/validate_setup.py`.317318## Scripts319320| Script | Purpose | Usage |321|--------|---------|-------|322| `scripts/validate_setup.py` | Verify Docker, NVIDIA Container Toolkit, GPU architecture, `git`, Hugging Face CLI, token presence, and disk space. No network calls. | `run_script("scripts/validate_setup.py")` or `python scripts/validate_setup.py` |323| `scripts/.env.example` | Template for `HF_TOKEN` and optional `NGC_API_KEY`. | `cp scripts/.env.example .env && set -a && . ./.env && set +a` |324325## References326327- [`references/inference.md`](references/inference.md) — container328 build, raw-base fallback, Blackwell patches, checkpoint329 download, `inference_pix2pix_turbo_harmonizer.py` flag matrix,330 non-temporal mode.331- [`references/evaluation.md`](references/evaluation.md) — paired332 `test_dataset/` layout and `evaluate_test_dataset.py` command.333- [`references/training.md`](references/training.md) — dataset334 download, training JSON format, multi-GPU `accelerate launch`335 recipe, fine-tuning flags, NuRec data-pair recipes.336- [`references/wrapper-image.md`](references/wrapper-image.md) —337 build and run the project image for repeat inference.338- [`references/troubleshooting.md`](references/troubleshooting.md)339 — extended diagnostic notes.340- [`references/teardown.md`](references/teardown.md) — cleanup341 inventory for images, code, Hugging Face caches, datasets,342 and outputs.343- Public code: <https://github.com/NVIDIA/harmonizer>344- Model card: <https://huggingface.co/nvidia/Harmonizer>345- Dataset: <https://huggingface.co/datasets/nvidia/Harmonizer-Dataset>346- Paper: <https://arxiv.org/abs/2602.24096>347- Project page: <https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/>348349## Limitations350351- **Rendered inputs only.** The model is tuned for352 neural-reconstruction renderings and object-insertion353 artifacts, not arbitrary real photos.354- **Primary model resolution is 576x1024.** The inference script355 maps resolution key `1024` to `1024x576`. Only `1024`, `960`,356 and `1360` are supported keys; `1024` matches the model-card357 operating point.358- **Temporal references and filename order matter.** The359 inference script enhances frames in natural-sorted order and360 feeds previous outputs back as temporal references. Use361 zero-padded frame numbers, or pass `--nontemporal` for362 unordered images.363- **Container builds can be large.** The runtime image, build364 cache, model weights, and optional dataset can exceed 100 GB.365- **Training is multi-GPU by default.** The README command366 assumes 8 GPUs with bf16 mixed precision.367- **Public code evolves.** The current README documents368 `inference_pix2pix_turbo_harmonizer.py` as the inference entry369 point; if a future checkout renames it, prefer the script and370 flags present in that checkout.371372## Troubleshooting (top 5)373374| Error / symptom | Most common cause |375|-----------------|-------------------|376| `docker: could not select device driver ... gpu` | NVIDIA Container Toolkit missing or Docker is not configured for the NVIDIA runtime. |377| `docker pull` 401 / 403 from `nvcr.io` | Docker is not authenticated to NGC, or the API key lacks container access. |378| `hf download ... 401 / 403` | `HF_TOKEN` is missing/expired/lacks read scope, or the model/dataset license has not been accepted. |379| `diffusion_harmonizer.pkl` missing | Checkpoint download path is wrong or incomplete. Re-run `./download_checkpoints.sh` from the repo root. |380| Output files owned by `root` | The `docker run` omitted `-u $(id -u):$(id -g)`. |381382Full matrix in383[`references/troubleshooting.md`](references/troubleshooting.md).384385## Teardown386387A full workflow can leave large artifacts on disk: the Cosmos388image, project image, build cache, `harmonizer` code checkout,389Hugging Face model weights, optional dataset, evaluation390outputs, and enhanced frames. Reclaim them with the inventory391in [`references/teardown.md`](references/teardown.md). Do not392revoke `HF_TOKEN` or `NGC_API_KEY` as normal cleanup. Rotate a393token only if you suspect it was leaked.