NVIDIA DiffusionHarmonizer (NuRec post-processing)
Purpose
Run NVIDIA DiffusionHarmonizer on rendered images from neural
reconstructions. DiffusionHarmonizer is a single-step,
temporally-aware image diffusion enhancer for NeRF / 3DGS /
NuRec-style renderings. It improves realism, reduces
reconstruction artifacts, and harmonizes inserted dynamic
objects with the surrounding scene.
When to Use / When NOT to Use
Use this skill when the user has rendered frames from
NuRec, 3DGS, NeRF, or a similar reconstruction pipeline and
wants to enhance, harmonize, evaluate, or optionally fine-tune
the DiffusionHarmonizer model.
Do NOT use this skill when:
- The user wants to train or render the 3D reconstruction itself
(use the
nurec skills).
- The user wants to convert raw sensor data to NCore V4 (use
ncore).
- The user wants a generic photo enhancer. DiffusionHarmonizer
is tuned for neural-reconstruction artifacts and
object-insertion failures.
- The user only wants NuRec inline rendering with
--enable-difix. That remains a NuRec runtime feature; use the
nurec skills for the complete serve-grpc / render-grpc
command shape.
What changed from the older Fixer skill
This skill follows the public NVIDIA/harmonizer release, not
the older NGC JIT .pt artifact recipe. Use these public
release artifacts:
- Code: https://github.com/NVIDIA/harmonizer
- Model:
nvidia/Harmonizer on Hugging Face (the paper checkpoint
models/diffusion_harmonizer.pkl), plus the base
nvidia/Cosmos-Predict2-0.6B-Text2Image model that inference
also requires.
- Checkpoint download:
./download_checkpoints.sh from the repo
root. It fetches the Harmonizer checkpoints into models/
(diffusion_harmonizer.pkl, harmonizer_nontemporal.pt) and
the base Cosmos DiT + tokenizer into
src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/.
- Runtime: the
harmonizer-cosmos-env image built from
Dockerfile.cosmos (base nvcr.io/nvidia/pytorch:25.10-py3).
- Inference entry:
src/inference_pix2pix_turbo_harmonizer.py,
run from inside /work/src so it can import its sibling
modules.
- Evaluation entry:
src/evaluate_test_dataset.py
- Training entry:
src/train_pix2pix_turbo_harmonizer.py
Do not run inference_pretrained_model.py; the current README
documents inference_pix2pix_turbo_harmonizer.py as the
inference entry point.
Background
DiffusionHarmonizer is described in DiffusionHarmonizer:
Bridging Neural Reconstruction and Photorealistic Simulation
with Online Diffusion Enhancer
(arXiv 2602.24096, CVPR 2026).
It distills a pretrained multi-step diffusion model into a
single-step enhancer designed for online simulation and offline
data cleanup.
Two operating modes:
- Offline: clean pseudo-training views rendered from a
reconstruction, then distill the improved views back into the
3D representation.
- Online: enhance frames during simulation/inference by
harmonizing color and lighting, reconstructing missing or
inconsistent shadows for inserted actors, and reducing residual
reconstruction artifacts.
The public model card describes DiffusionHarmonizer-cosmos-0.6B,
a Cosmos Predict2 Diffusion Transformer post-trained at
576x1024 input and output resolution.
Inputs
- code_dir — checkout of
https://github.com/NVIDIA/harmonizer.
- model_dir — checkpoints fetched by
./download_checkpoints.sh
from the repo root. It places the paper checkpoint at
models/diffusion_harmonizer.pkl and the required base Cosmos
model under
src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/.
- input_dir — directory of rendered RGB frames (
.png,
.jpg, .jpeg) to enhance.
- output_dir —
inference_pix2pix_turbo_harmonizer.py does
not take an output flag; it writes to a sibling folder named
<input_dir>_<model_identifier> next to the input directory.
- HF_TOKEN — Hugging Face token. Not required for
nvidia/Harmonizer, which is public and downloads anonymously.
Required, with licence acceptance, for the gated
nvidia/Cosmos-Predict2-0.6B-Text2Image and (if used)
nvidia/Harmonizer-Dataset. A cached hf auth login works in place
of the environment variable.
- NGC_API_KEY — often needed to authenticate
docker pull
from nvcr.io. Use only for container pulls, not model
download.
Instructions
Validate the host. Have the agent execute
this skill's scripts/validate_setup.py via its standard script
runner — e.g. run_script("scripts/validate_setup.py") or, from this
skill's own directory, python scripts/validate_setup.py. Paths
beginning scripts/ in this document are relative to the skill
directory; this repository has no scripts/ at its root, so the
command will not resolve from the checkout. It checks Docker, the
NVIDIA Container Toolkit, GPU architecture, git, the
Hugging Face CLI, token presence, and free disk space. It exits
non-zero for missing Docker, toolkit, GPU or git. A missing
Hugging Face CLI or token is only a warning that still exits
zero — pass --strict to make warnings fail. Two further gaps to
know: it accepts the legacy huggingface-cli, but
download_checkpoints.sh invokes hf only, so that combination
passes the check and then fails at download; and its disk check
covers inference only, not the ~1.76 TB dataset.
Clone the code and build (or pull) the runtime image.
Full commands and the Blackwell patch caveat live in
references/inference.md.
If you are already inside a harmonizer checkout — which is the case when
this skill ships in-repo — use it and skip the clone; cloning from
there nests a second checkout and switches work to upstream main.
# only when you do not already have a checkout
git clone https://github.com/NVIDIA/harmonizer.git
cd harmonizer
# from an existing checkout, start here
docker build -t harmonizer-cosmos-env -f Dockerfile.cosmos .
Download the checkpoints. From the repo root run the
helper, which fetches both the Harmonizer checkpoints and the
base Cosmos model into the paths the code expects.
export HF_TOKEN="hf_your_token_here" # your real token, no angle brackets
hf auth login --token "$HF_TOKEN"
./download_checkpoints.sh
Verify models/diffusion_harmonizer.pkl and
src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/
exist.
Confirm input_dir exists and filenames sort into frame
order. The temporal inference script sorts frames with
plain lexical sort (not natural sort, despite the video-assembly step
using natsorted) and uses previous outputs as references, so
fixed-width zero-padded filenames are required — frame_10.png
otherwise sorts before frame_2.png and corrupts the sequence; prefer
zero-padded names such as frame_000001.png.
Run inference inside the container with the repo mounted
at /work, then cd /work/src and run
inference_pix2pix_turbo_harmonizer.py. Pin
-u $(id -u):$(id -g) so outputs are owned by the host user.
Output frames land in
<input_dir>_<model_identifier>. Full docker run recipe and
flag matrix in
references/inference.md.
Validate that the output frame count matches the input
frame count, spot-check frames, and (if ground truth is
available) consult
references/evaluation.md for the paired
PSNR/LPIPS shape. Do not promise the user that evaluation will run —
evaluate_test_dataset.py has upstream blockers documented in that file.
(Optional) Train or fine-tune. Download the dataset (or
prepare JSON manifests in the documented format), then run
src/train_pix2pix_turbo_harmonizer.py with the recommended
hyperparameters. For fine-tuning, initialize from the released
checkpoint with --pretrained_path /path/to/diffusion_harmonizer.pkl. Full recipe + NuRec
data-pair recipes in
references/training.md.
(Optional) Teardown. Follow
references/teardown.md to remove
images, code clones, model weights, datasets, and outputs.
Examples
Example 1 — Enhance a folder of rendered frames
Run scripts/validate_setup.py, build the image once (see
references/wrapper-image.md), then
invoke inference_pix2pix_turbo_harmonizer.py inside the
harmonizer-cosmos-env container with the repo checkout mounted at
/work. From /work/src, point --input_image at the rendered
frames, --model_path at /work/models/diffusion_harmonizer.pkl,
set --model_identifier, and pass typical flags
--timestep 250 --resolution 1024 --use_sched. Enhanced frames are
written to <input_dir>_<model_identifier>. The canonical
docker run command and the full flag matrix live in
references/inference.md.
Example 2 — Quantitative PSNR / LPIPS evaluation
Prepare the paired test_dataset/{scene}/render +
test_dataset/{scene}/gt layout. src/evaluate_test_dataset.py is the
intended entry point but does not currently run as shipped — see
references/evaluation.md for the exact
directory shape, the docker run shape, and the two upstream blockers.
Example 3 — Fine-tune from the public checkpoint
Download nvidia/Harmonizer-Dataset, prepare the
training JSON, then run
src/train_pix2pix_turbo_harmonizer.py with the multi-GPU
accelerate launch recipe in
references/training.md. For
fine-tuning add --pretrained_path /path/to/diffusion_harmonizer.pkl. Do not add --fixing_data_weight
here: with the single /data/data.json source shown above it cannot do
anything, because every item in a source gets the same weight and the
multiplier normalises out
(src/train_pix2pix_turbo_harmonizer.py:200-208). Up-weighting needs a
comma-separated --dataset_folder with the correction data isolated in its
own source whose path contains nre_data, plus --weighted_sampler.
Example 4 — Non-temporal (frame-by-frame) enhancement
inference_pix2pix_turbo_harmonizer.py is temporal by default
and uses previous enhanced frames as references
(--offset_list -1 -2 -3 -4). When the user wants each frame
enhanced independently (e.g. unordered images), add
--nontemporal to disable temporal conditioning. Full command +
--offset_list defaults in
references/inference.md.
Prerequisites
- OS: Linux host.
- GPU / driver: NVIDIA GPU Ampere or newer (compute
capability
>= 8.0; A100, A10, L40, H100, RTX 30/40/PRO,
B200, GB200).
- Container runtime: Docker with the NVIDIA Container
Toolkit.
- Tools:
git, python3, Hugging Face CLI (hf or
huggingface-cli).
- Secrets:
HF_TOKEN with the gated
nvidia/Cosmos-Predict2-0.6B-Text2Image
licenses accepted (required to download model weights and
the optional dataset).
NGC_API_KEY (often required for docker login nvcr.io
before pulling nvcr.io/nvidia/pytorch:25.10-py3).
- Disk: at least
120 GB free for the runtime image, build cache,
model weights and outputs — this covers inference only. The optional
training dataset is a further **1.76 TB**, not included in that figure
nor checked by the prerequisite script; see
references/training.md before downloading it.
- Source / model / dataset:
scripts/validate_setup.py checks the above. It fails hard on missing
Docker, NVIDIA Container Toolkit, a supported GPU or git; Hugging Face
CLI and token issues are warnings unless you pass --strict. It does not
verify space for the optional dataset.
Scripts
| Script |
Purpose |
Usage |
scripts/validate_setup.py |
Verify Docker, NVIDIA Container Toolkit, GPU architecture, git, Hugging Face CLI, token presence, and disk space. No network calls. |
run_script("scripts/validate_setup.py") or python scripts/validate_setup.py |
scripts/.env.example |
Template for HF_TOKEN and optional NGC_API_KEY. Keep the filled-in file OUTSIDE the checkout. Dockerfile.cosmos:29 is COPY . /harmonizer-codebase, and this repository ships no .dockerignore, so anything in the checkout — including a .env — is uploaded to the Docker daemon or remote builder and retained in build cache. A .gitignore does not scope build context and will not prevent this. |
install -m 600 scripts/.env.example ~/.harmonizer.env && set -a && . ~/.harmonizer.env && set +a (the template is mode 0664 — a plain cp would leave your live tokens readable by other local users) |
References
references/inference.md — container
build, raw-base fallback, Blackwell patches, checkpoint
download, inference_pix2pix_turbo_harmonizer.py flag matrix,
non-temporal mode.
references/evaluation.md — paired
test_dataset/ layout and evaluate_test_dataset.py command, plus the
upstream blockers that stop it running today.
references/training.md — dataset
download, training JSON format, multi-GPU accelerate launch
recipe, fine-tuning flags, NuRec data-pair recipes.
references/wrapper-image.md —
build and run the project image for repeat inference.
references/troubleshooting.md
— extended diagnostic notes.
references/teardown.md — cleanup
inventory for images, code, Hugging Face caches, datasets,
and outputs.
- Public code: https://github.com/NVIDIA/harmonizer
- Model card: https://huggingface.co/nvidia/Harmonizer
- Dataset: https://huggingface.co/datasets/nvidia/Harmonizer-Dataset
- Paper: https://arxiv.org/abs/2602.24096
- Project page: https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/
Limitations
- Rendered inputs only. The model is tuned for
neural-reconstruction renderings and object-insertion
artifacts, not arbitrary real photos.
- Primary model resolution is 576x1024. The inference script
maps resolution key
1024 to 1024x576. Only 1024, 960,
and 1360 are supported keys; 1024 matches the model-card
operating point.
- Temporal references and filename order matter. The
inference script enhances frames in plain lexically sorted order
(zero-pad filenames to a fixed width) and
feeds previous outputs back as temporal references. Use
zero-padded frame numbers, or pass
--nontemporal for
unordered images.
- Container builds can be large. The runtime image, build
cache, model weights, and optional dataset can exceed 100 GB.
- Training is multi-GPU by default. The README command
assumes 8 GPUs with bf16 mixed precision.
- Public code evolves. The current README documents
inference_pix2pix_turbo_harmonizer.py as the inference entry
point; if a future checkout renames it, prefer the script and
flags present in that checkout.
Troubleshooting (top 5)
| Error / symptom |
Most common cause |
docker: could not select device driver ... gpu |
NVIDIA Container Toolkit missing or Docker is not configured for the NVIDIA runtime. |
docker pull 401 / 403 from nvcr.io |
Docker is not authenticated to NGC, or the API key lacks container access. |
hf download ... 401 / 403 |
Hitting a gated repo (nvidia/Cosmos-Predict2-0.6B-Text2Image, nvidia/Harmonizer-Dataset) without an accepted licence or a valid token. nvidia/Harmonizer itself is public and needs neither. |
diffusion_harmonizer.pkl missing |
Checkpoint download path is wrong or incomplete. Re-run ./download_checkpoints.sh from the repo root. |
Output files owned by root |
The docker run omitted -u $(id -u):$(id -g). |
Full matrix in
references/troubleshooting.md.
Teardown
A full workflow can leave large artifacts on disk: the Cosmos
image, project image, build cache, harmonizer code checkout,
Hugging Face model weights, optional dataset, evaluation
outputs, and enhanced frames. Reclaim them with the inventory
in references/teardown.md. Do not
revoke HF_TOKEN or NGC_API_KEY as normal cleanup. Rotate a
token only if you suspect it was leaked.
1---2name: harmonizer3description: >-4 Use to run NVIDIA DiffusionHarmonizer (public successor to the5 older Fixer recipes) to enhance, harmonize, evaluate, or6 fine-tune novel-view frames from NuRec / 3DGS / NeRF7 reconstructions. Do NOT use for training the 3D reconstruction8 itself (use the `nurec` skills) or for sensor-to-NCore9 conversion (use `ncore`).10version: "0.5.0"11tools:12 - Shell13 - Read14 - Write15license: CC-BY-4.0 AND Apache-2.016compatibility: >-17 Linux + NVIDIA GPU Ampere+ (compute capability >= 8.0), Docker + NVIDIA18 Container Toolkit, ~120 GB free disk for inference. Runtime is the19 `harmonizer-cosmos-env` image built from `Dockerfile.cosmos`. The20 `nvidia/Harmonizer` checkpoints are public; HF_TOKEN and licence21 acceptance are needed only for gated resources, and NGC_API_KEY for the22 nvcr.io base image. See Prerequisites for the full detail.23dependencies:24 - bash25 - docker26 - nvidia-container-toolkit27 - python328 - git29 - huggingface_hub30metadata:31 source: https://skillmd.com/skills/nvidia/harmonizer32 author: NVIDIA NRS <nurec-skills@nvidia.com>33 tags:34 - nurec35 - fixer36 - harmonizer37 - diffusionharmonizer38 - reconstruction39 - autonomous-vehicles40 - post-processing41 - evaluation42 upstream: https://github.com/NVIDIA/harmonizer43 hf_model: https://huggingface.co/nvidia/Harmonizer44 hf_dataset: https://huggingface.co/datasets/nvidia/Harmonizer-Dataset45 container: nvcr.io/nvidia/pytorch:25.10-py346 paper: https://arxiv.org/abs/2602.2409647 product_page: https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/48 model_license: NVIDIA Open Model License Agreement49---5051# NVIDIA DiffusionHarmonizer (NuRec post-processing)5253## Purpose5455Run NVIDIA DiffusionHarmonizer on rendered images from neural56reconstructions. DiffusionHarmonizer is a single-step,57temporally-aware image diffusion enhancer for NeRF / 3DGS /58NuRec-style renderings. It improves realism, reduces59reconstruction artifacts, and harmonizes inserted dynamic60objects with the surrounding scene.6162## When to Use / When NOT to Use6364**Use this skill when** the user has rendered frames from65NuRec, 3DGS, NeRF, or a similar reconstruction pipeline and66wants to enhance, harmonize, evaluate, or optionally fine-tune67the DiffusionHarmonizer model.6869**Do NOT use this skill when:**7071- The user wants to train or render the 3D reconstruction itself72 (use the `nurec` skills).73- The user wants to convert raw sensor data to NCore V4 (use74 `ncore`).75- The user wants a generic photo enhancer. DiffusionHarmonizer76 is tuned for neural-reconstruction artifacts and77 object-insertion failures.78- The user only wants NuRec inline rendering with79 `--enable-difix`. That remains a NuRec runtime feature; use the80 `nurec` skills for the complete `serve-grpc` / `render-grpc`81 command shape.8283## What changed from the older Fixer skill8485This skill follows the public `NVIDIA/harmonizer` release, not86the older NGC JIT `.pt` artifact recipe. Use these public87release artifacts:8889- Code: <https://github.com/NVIDIA/harmonizer>90- Model: `nvidia/Harmonizer` on Hugging Face (the paper checkpoint91 `models/diffusion_harmonizer.pkl`), plus the base92 `nvidia/Cosmos-Predict2-0.6B-Text2Image` model that inference93 also requires.94- Checkpoint download: `./download_checkpoints.sh` from the repo95 root. It fetches the Harmonizer checkpoints into `models/`96 (`diffusion_harmonizer.pkl`, `harmonizer_nontemporal.pt`) and97 the base Cosmos DiT + tokenizer into98 `src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/`.99- Runtime: the `harmonizer-cosmos-env` image built from100 `Dockerfile.cosmos` (base `nvcr.io/nvidia/pytorch:25.10-py3`).101- Inference entry: `src/inference_pix2pix_turbo_harmonizer.py`,102 run from inside `/work/src` so it can import its sibling103 modules.104- Evaluation entry: `src/evaluate_test_dataset.py`105- Training entry: `src/train_pix2pix_turbo_harmonizer.py`106107Do not run `inference_pretrained_model.py`; the current README108documents `inference_pix2pix_turbo_harmonizer.py` as the109inference entry point.110111## Background112113DiffusionHarmonizer is described in *DiffusionHarmonizer:114Bridging Neural Reconstruction and Photorealistic Simulation115with Online Diffusion Enhancer*116([arXiv 2602.24096](https://arxiv.org/abs/2602.24096), CVPR 2026).117It distills a pretrained multi-step diffusion model into a118single-step enhancer designed for online simulation and offline119data cleanup.120121Two operating modes:122123- **Offline:** clean pseudo-training views rendered from a124 reconstruction, then distill the improved views back into the125 3D representation.126- **Online:** enhance frames during simulation/inference by127 harmonizing color and lighting, reconstructing missing or128 inconsistent shadows for inserted actors, and reducing residual129 reconstruction artifacts.130131The public model card describes `DiffusionHarmonizer-cosmos-0.6B`,132a Cosmos Predict2 Diffusion Transformer post-trained at133`576x1024` input and output resolution.134135## Inputs136137- **code_dir** — checkout of138 <https://github.com/NVIDIA/harmonizer>.139- **model_dir** — checkpoints fetched by `./download_checkpoints.sh`140 from the repo root. It places the paper checkpoint at141 `models/diffusion_harmonizer.pkl` and the required base Cosmos142 model under143 `src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/`.144- **input_dir** — directory of rendered RGB frames (`.png`,145 `.jpg`, `.jpeg`) to enhance.146- **output_dir** — `inference_pix2pix_turbo_harmonizer.py` does147 not take an output flag; it writes to a sibling folder named148 `<input_dir>_<model_identifier>` next to the input directory.149- **HF_TOKEN** — Hugging Face token. **Not** required for150 `nvidia/Harmonizer`, which is public and downloads anonymously.151 Required, with licence acceptance, for the gated152 `nvidia/Cosmos-Predict2-0.6B-Text2Image` and (if used)153 `nvidia/Harmonizer-Dataset`. A cached `hf auth login` works in place154 of the environment variable.155- **NGC_API_KEY** — often needed to authenticate `docker pull`156 from `nvcr.io`. Use only for container pulls, not model157 download.158159## Instructions1601611. **Validate the host.** Have the agent execute162 this skill's `scripts/validate_setup.py` via its standard script163 runner — e.g. `run_script("scripts/validate_setup.py")` or, from this164 skill's own directory, `python scripts/validate_setup.py`. Paths165 beginning `scripts/` in this document are relative to the skill166 directory; this repository has no `scripts/` at its root, so the167 command will not resolve from the checkout. It checks Docker, the168 NVIDIA Container Toolkit, GPU architecture, `git`, the169 Hugging Face CLI, token presence, and free disk space. It exits170 non-zero for missing Docker, toolkit, GPU or `git`. A missing171 Hugging Face CLI or token is only a **warning** that still exits172 zero — pass `--strict` to make warnings fail. Two further gaps to173 know: it accepts the legacy `huggingface-cli`, but174 `download_checkpoints.sh` invokes `hf` only, so that combination175 passes the check and then fails at download; and its disk check176 covers inference only, not the ~1.76 TB dataset.1772. **Clone the code and build (or pull) the runtime image.**178 Full commands and the Blackwell patch caveat live in179 [`references/inference.md`](references/inference.md).180181 If you are already inside a harmonizer checkout — which is the case when182 this skill ships in-repo — **use it and skip the clone**; cloning from183 there nests a second checkout and switches work to upstream `main`.184185 ```bash186 # only when you do not already have a checkout187 git clone https://github.com/NVIDIA/harmonizer.git188 cd harmonizer189190 # from an existing checkout, start here191 docker build -t harmonizer-cosmos-env -f Dockerfile.cosmos .192 ```1931943. **Download the checkpoints.** From the repo root run the195 helper, which fetches both the Harmonizer checkpoints and the196 base Cosmos model into the paths the code expects.197198 ```bash199 export HF_TOKEN="hf_your_token_here" # your real token, no angle brackets200 hf auth login --token "$HF_TOKEN"201 ./download_checkpoints.sh202 ```203204 Verify `models/diffusion_harmonizer.pkl` and205 `src/checkpoints/nvidia/Cosmos-Predict2-0.6B-Text2Image/`206 exist.2074. **Confirm `input_dir` exists and filenames sort into frame208 order.** The temporal inference script sorts frames with209 plain lexical sort (not natural sort, despite the video-assembly step210 using `natsorted`) and uses previous outputs as references, so211 fixed-width zero-padded filenames are **required** — `frame_10.png`212 otherwise sorts before `frame_2.png` and corrupts the sequence; prefer213 zero-padded names such as `frame_000001.png`.2145. **Run inference inside the container** with the repo mounted215 at `/work`, then `cd /work/src` and run216 `inference_pix2pix_turbo_harmonizer.py`. Pin217 `-u $(id -u):$(id -g)` so outputs are owned by the host user.218 Output frames land in219 `<input_dir>_<model_identifier>`. Full `docker run` recipe and220 flag matrix in221 [`references/inference.md`](references/inference.md).2226. **Validate that the output frame count matches the input223 frame count,** spot-check frames, and (if ground truth is224 available) consult225 [`references/evaluation.md`](references/evaluation.md) for the paired226 PSNR/LPIPS shape. **Do not promise the user that evaluation will run** —227 `evaluate_test_dataset.py` has upstream blockers documented in that file.2287. **(Optional) Train or fine-tune.** Download the dataset (or229 prepare JSON manifests in the documented format), then run230 `src/train_pix2pix_turbo_harmonizer.py` with the recommended231 hyperparameters. For fine-tuning, initialize from the released232 checkpoint with `--pretrained_path233 /path/to/diffusion_harmonizer.pkl`. Full recipe + NuRec234 data-pair recipes in235 [`references/training.md`](references/training.md).2368. **(Optional) Teardown.** Follow237 [`references/teardown.md`](references/teardown.md) to remove238 images, code clones, model weights, datasets, and outputs.239240## Examples241242### Example 1 — Enhance a folder of rendered frames243244Run `scripts/validate_setup.py`, build the image once (see245[`references/wrapper-image.md`](references/wrapper-image.md)), then246invoke `inference_pix2pix_turbo_harmonizer.py` inside the247`harmonizer-cosmos-env` container with the repo checkout mounted at248`/work`. From `/work/src`, point `--input_image` at the rendered249frames, `--model_path` at `/work/models/diffusion_harmonizer.pkl`,250set `--model_identifier`, and pass typical flags251`--timestep 250 --resolution 1024 --use_sched`. Enhanced frames are252written to `<input_dir>_<model_identifier>`. The canonical253`docker run` command and the full flag matrix live in254[`references/inference.md`](references/inference.md).255256### Example 2 — Quantitative PSNR / LPIPS evaluation257258Prepare the paired `test_dataset/{scene}/render` +259`test_dataset/{scene}/gt` layout. `src/evaluate_test_dataset.py` is the260intended entry point but does not currently run as shipped — see261[`references/evaluation.md`](references/evaluation.md) for the exact262directory shape, the `docker run` shape, and the two upstream blockers.263264### Example 3 — Fine-tune from the public checkpoint265266Download `nvidia/Harmonizer-Dataset`, prepare the267training JSON, then run268`src/train_pix2pix_turbo_harmonizer.py` with the multi-GPU269`accelerate launch` recipe in270[`references/training.md`](references/training.md). For271fine-tuning add `--pretrained_path272/path/to/diffusion_harmonizer.pkl`. Do **not** add `--fixing_data_weight`273here: with the single `/data/data.json` source shown above it cannot do274anything, because every item in a source gets the same weight and the275multiplier normalises out276(`src/train_pix2pix_turbo_harmonizer.py:200-208`). Up-weighting needs a277comma-separated `--dataset_folder` with the correction data isolated in its278own source whose path contains `nre_data`, plus `--weighted_sampler`.279280### Example 4 — Non-temporal (frame-by-frame) enhancement281282`inference_pix2pix_turbo_harmonizer.py` is temporal by default283and uses previous enhanced frames as references284(`--offset_list -1 -2 -3 -4`). When the user wants each frame285enhanced independently (e.g. unordered images), add286`--nontemporal` to disable temporal conditioning. Full command +287`--offset_list` defaults in288[`references/inference.md`](references/inference.md).289290## Prerequisites291292- **OS:** Linux host.293- **GPU / driver:** NVIDIA GPU Ampere or newer (compute294 capability `>= 8.0`; A100, A10, L40, H100, RTX 30/40/PRO,295 B200, GB200).296- **Container runtime:** Docker with the NVIDIA Container297 Toolkit.298- **Tools:** `git`, `python3`, Hugging Face CLI (`hf` or299 `huggingface-cli`).300- **Secrets:**301 - `HF_TOKEN` with the gated302 [`nvidia/Cosmos-Predict2-0.6B-Text2Image`](https://huggingface.co/nvidia/Cosmos-Predict2-0.6B-Text2Image)303 licenses accepted (required to download model weights and304 the optional dataset).305 - `NGC_API_KEY` (often required for `docker login nvcr.io`306 before pulling `nvcr.io/nvidia/pytorch:25.10-py3`).307- **Disk:** at least ~120 GB free for the runtime image, build cache,308 model weights and outputs — this covers **inference only**. The optional309 training dataset is a further **~1.76 TB**, not included in that figure310 nor checked by the prerequisite script; see311 [`references/training.md`](references/training.md) before downloading it.312- **Source / model / dataset:**313 - Code: <https://github.com/NVIDIA/harmonizer>.314 - Model: <https://huggingface.co/nvidia/Harmonizer>.315 - Base model:316 <https://huggingface.co/nvidia/Cosmos-Predict2-0.6B-Text2Image>.317 - Optional dataset:318 <https://huggingface.co/datasets/nvidia/Harmonizer-Dataset>.319320`scripts/validate_setup.py` checks the above. It fails hard on missing321Docker, NVIDIA Container Toolkit, a supported GPU or `git`; Hugging Face322CLI and token issues are warnings unless you pass `--strict`. It does not323verify space for the optional dataset.324325## Scripts326327| Script | Purpose | Usage |328|--------|---------|-------|329| `scripts/validate_setup.py` | Verify Docker, NVIDIA Container Toolkit, GPU architecture, `git`, Hugging Face CLI, token presence, and disk space. No network calls. | `run_script("scripts/validate_setup.py")` or `python scripts/validate_setup.py` |330| `scripts/.env.example` | Template for `HF_TOKEN` and optional `NGC_API_KEY`. **Keep the filled-in file OUTSIDE the checkout.** `Dockerfile.cosmos:29` is `COPY . /harmonizer-codebase`, and this repository ships no `.dockerignore`, so anything in the checkout — including a `.env` — is uploaded to the Docker daemon or remote builder and retained in build cache. A `.gitignore` does **not** scope build context and will not prevent this. | `install -m 600 scripts/.env.example ~/.harmonizer.env && set -a && . ~/.harmonizer.env && set +a` (the template is mode 0664 — a plain `cp` would leave your live tokens readable by other local users) |331332## References333334- [`references/inference.md`](references/inference.md) — container335 build, raw-base fallback, Blackwell patches, checkpoint336 download, `inference_pix2pix_turbo_harmonizer.py` flag matrix,337 non-temporal mode.338- [`references/evaluation.md`](references/evaluation.md) — paired339 `test_dataset/` layout and `evaluate_test_dataset.py` command, plus the340 upstream blockers that stop it running today.341- [`references/training.md`](references/training.md) — dataset342 download, training JSON format, multi-GPU `accelerate launch`343 recipe, fine-tuning flags, NuRec data-pair recipes.344- [`references/wrapper-image.md`](references/wrapper-image.md) —345 build and run the project image for repeat inference.346- [`references/troubleshooting.md`](references/troubleshooting.md)347 — extended diagnostic notes.348- [`references/teardown.md`](references/teardown.md) — cleanup349 inventory for images, code, Hugging Face caches, datasets,350 and outputs.351- Public code: <https://github.com/NVIDIA/harmonizer>352- Model card: <https://huggingface.co/nvidia/Harmonizer>353- Dataset: <https://huggingface.co/datasets/nvidia/Harmonizer-Dataset>354- Paper: <https://arxiv.org/abs/2602.24096>355- Project page: <https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/>356357## Limitations358359- **Rendered inputs only.** The model is tuned for360 neural-reconstruction renderings and object-insertion361 artifacts, not arbitrary real photos.362- **Primary model resolution is 576x1024.** The inference script363 maps resolution key `1024` to `1024x576`. Only `1024`, `960`,364 and `1360` are supported keys; `1024` matches the model-card365 operating point.366- **Temporal references and filename order matter.** The367 inference script enhances frames in plain lexically sorted order368 (zero-pad filenames to a fixed width) and369 feeds previous outputs back as temporal references. Use370 zero-padded frame numbers, or pass `--nontemporal` for371 unordered images.372- **Container builds can be large.** The runtime image, build373 cache, model weights, and optional dataset can exceed 100 GB.374- **Training is multi-GPU by default.** The README command375 assumes 8 GPUs with bf16 mixed precision.376- **Public code evolves.** The current README documents377 `inference_pix2pix_turbo_harmonizer.py` as the inference entry378 point; if a future checkout renames it, prefer the script and379 flags present in that checkout.380381## Troubleshooting (top 5)382383| Error / symptom | Most common cause |384|-----------------|-------------------|385| `docker: could not select device driver ... gpu` | NVIDIA Container Toolkit missing or Docker is not configured for the NVIDIA runtime. |386| `docker pull` 401 / 403 from `nvcr.io` | Docker is not authenticated to NGC, or the API key lacks container access. |387| `hf download ... 401 / 403` | Hitting a gated repo (`nvidia/Cosmos-Predict2-0.6B-Text2Image`, `nvidia/Harmonizer-Dataset`) without an accepted licence or a valid token. `nvidia/Harmonizer` itself is public and needs neither. |388| `diffusion_harmonizer.pkl` missing | Checkpoint download path is wrong or incomplete. Re-run `./download_checkpoints.sh` from the repo root. |389| Output files owned by `root` | The `docker run` omitted `-u $(id -u):$(id -g)`. |390391Full matrix in392[`references/troubleshooting.md`](references/troubleshooting.md).393394## Teardown395396A full workflow can leave large artifacts on disk: the Cosmos397image, project image, build cache, `harmonizer` code checkout,398Hugging Face model weights, optional dataset, evaluation399outputs, and enhanced frames. Reclaim them with the inventory400in [`references/teardown.md`](references/teardown.md). Do not401revoke `HF_TOKEN` or `NGC_API_KEY` as normal cleanup. Rotate a402token only if you suspect it was leaked.