NRE — NVIDIA Omniverse NuRec (Neural Reconstruction Engine)
Purpose
Drive the public NVIDIA Omniverse NuRec / Neural Reconstruction
Engine containers (nvcr.io/nvidia/nre/nre,
nvcr.io/nvidia/nre/nre-tools) to train a 3DGUT/3DGRT Gaussian
reconstruction from an NCore V4 camera+LiDAR clip, render novel
views (locally or via gRPC), generate aux data, export
PLY/depth/mesh/ego-mask/tracks, package Asset Harvester output
into a USDZ, and evaluate rendering metrics.
This skill carries the host-side toolkit around the NRE CLI: NGC
credential resolution, cached-image notes, local render recipes,
MP4 encoding, warm serve-grpc boot/teardown scripts, a thin
Python gRPC client for repeated RGB renders, bundled rig JSONs,
pre-baked custom-rig trajectories, and bash / Hydra / OSMO
workflow templates.
When to Use / When NOT to Use
Use this skill when the user has an NCore V4 clip (or a USDZ +
NRE artifact pair) on a Linux x86_64 host with an NVIDIA GPU and
an NGC API key, and wants to train, render, generate aux data,
export artifacts, insert/remove actors, run the gRPC server, or
evaluate metrics. Concrete triggers:
- Train a multi-camera + LiDAR AV clip into a renderable USDZ
scene with 3DGUT (or 3DGRT ray-traced) Gaussians.
- Generate NuRec auxiliary data (seg, depth, ego mask, DINOv2,
LiDAR-seg visibility) using
nre-tools.
- Render frames locally (no server) along the training rig or a
custom rig + offsets.
- Render novel views via the sensorsim gRPC API (CARLA, Isaac
Sim, AlpaSim, custom simulator), optionally with Difix
artifact-removal.
- Render LiDAR sweeps via
render-grpc --lidar.
- Export PLY / ego masks / depth / Poisson mesh / ground mesh /
point clouds / cuboid tracks / NCore tracks / custom rig
trajectories.
- Insert / remove / replace 3D actors with
export-external-assets
render-grpc --edit-assets.
- Render the gated HF dataset
nvidia/PhysicalAI-Autonomous-Vehicles-NuRec.
- Upgrade an old USDZ once (
upgrade-artifact).
- Inspect / evaluate (
export-parsed-config, gaussian-statistics,
eval-rendering-metrics, compute-metrics,
eval-ground-mesh).
- Browse a USDZ or PLY in the in-container viewer.
Do NOT use this skill when:
- The user still needs to convert raw sensor data into NCore V4
(use the
ncore skill first; NRE consumes NCore-formatted
shards).
- The user wants per-object 3D asset extraction from sparse views
(use
asset-harvester; NRE only consumes AH outputs via
export-external-assets).
- The user only needs to clean up already-rendered frames using
the standalone Cosmos-based Fixer (use
nurec-fixer). NRE's
inline --enable-difix flag is still on this skill's surface,
but the standalone harmonizer pipeline is owned by
nurec-fixer.
Inputs
- dataset_dir — host directory holding the NCore shards
(
<NAME>.zarr.itar, <NAME>.json, and any pre-generated
<NAME>.aux.*.zarr auxiliary shards). Required.
- dataset_name — basename of the NCore dataset (the part
before
.zarr.itar). Required.
- output_dir — host directory NRE will fill with checkpoints,
parsed config, metrics, videos, and USDZ artifacts. Required.
- camera_ids / lidar_ids — sensor IDs from the NCore JSON to
include. Default: all sensors per recipe.
- config_name — Hydra config path resolved inside the
container. Pick by source dataset:
- Waymo Open Dataset →
configs/apps/AV/Waymo/3dgut_dynamic.yaml (and its
_mcmc / _road_semantic / _static siblings). These are
Waymo-only — they bake in the Waymo sensor rig and conventions.
- NVIDIA Physical AI Autonomous Vehicles (PAI) →
/apps/prod/Hyperion-8.1/car2sim_6cam.yaml (the Hyperion-8.1
car2sim 6-camera recipe used by the Maglev PAI pipeline).
Typically referenced via the small overlay shipped at
references/configs/pai.yaml,
which extends car2sim_6cam.yaml with PAI's
lidar_top_360fov ID, six-camera validation set, and lidar
intensity supervision; mount it as
{nre_config_dir}/external_overrides.yaml and pass
--config-name=external_overrides.
- PandaSet / NVIDIA AV (NV) / Tesla / Alpasim → see the
matching
configs/apps/AV/{PandaSet,NV,Tesla}/… or
configs/apps/Alpasim/… recipes in
references/configuration.md.
- Do not use the Waymo recipes for PAI clips — the
sensor rig, validation cameras, and lidar IDs differ.
- mode —
train, val, or trainval. Default: trainval.
- NGC_API_KEY — required env var. Generate at
https://org.ngc.nvidia.com/setup/api-keys.
Instructions
- Validate prerequisites. Have the agent execute
scripts/validate_setup.py via its standard script runner
(run_script("scripts/validate_setup.py"), or
python scripts/validate_setup.py [--strict]). It checks
Docker, NVIDIA Container Toolkit, GPU/driver, and
NGC_API_KEY. Resolve any FAIL line before pulling the image.
- Authenticate Docker to NGC + pull the public containers.
See
references/install.md. Pull both
nvcr.io/nvidia/nre/nre:latest and
nvcr.io/nvidia/nre/nre-tools:latest.
- Confirm input layout. The dataset directory must contain
<NAME>.zarr.itar, <NAME>.json, and any <NAME>.aux.*.zarr
shards. If the NCore data is fresh, generate auxiliary data
first — see references/aux-data.md.
- Train / validate the reconstruction. Run the train recipe
in
references/cookbook.md with the
chosen Hydra --config-name, mode, dataset.path, and
out_dir. For multi-GPU append trainer.world_size=<N> trainer.num_nodes=<M> (see Workflow D). Set
checkpoint.artifact.enabled=true if you intend to render or
serve the result.
- Export downstream artifacts. Use export sub-commands
(
export-gaussian-plys, export-mesh, export-ground-mesh,
export-ego-mask, export-depth, export-sequence-tracks,
export-ncore-tracks, …) — full surface in
references/cli-reference.md.
- Render novel views — pick the backend.
- Local CLI —
nre render --artifact-path <usdz> writes
frames on disk along the training trajectory, with optional
rig offsets or --custom-rig-trajectory. No gRPC server.
See references/local-render.md.
- Warm RGB service — boot
serve-grpc once with
scripts/session_warm_server.sh, extract protobuf stubs,
and use
references/NRE_RenderClient/scripts/thin_client.py for
repeated single-camera or batch_render_rgb calls.
- Remote CLI / simulator integration —
serve-grpc +
render-grpc (or your own client via nre.grpc.protos).
Required for LiDAR rendering, simulator loops, Difix, or
--edit-assets. See references/grpc-api.md and
references/physical-ai-render.md.
- Edit actors (optional). Run
export-external-assets to
repackage Asset-Harvester output into a new USDZ, then pass
the produced edit-assets.json to render-grpc --edit-assets
(with serve-grpc --enable-editing-actors). See
references/asset-editing.md.
- Validate the result. Confirm
<output_dir>/<RUN-ID>/usd-out/last.usdz opens, metrics.yaml
reports a reasonable test/psnr, and the generated MP4s
render. For more thorough metrics use Workflow I (eval) in
references/workflows.md. Tear
down any gRPC server (Ctrl-C or docker rm -f).
For any NRE task expected to run 5 minutes or longer (training,
OSMO jobs, multi-clip renders), follow
references/long-running-tasks.md: delegate to a subagent /
background job and report compact status at least every 5
minutes.
Examples
Example 1 — End-to-end NCore → USDZ → render
Walk Workflow A in
references/workflows.md: validate
host → generate aux data → train (cookbook recipe) → export →
local render or serve-grpc. Concrete commands live in the
referenced files; this index does not duplicate them.
Example 2 — Skip training, render the gated Physical AI dataset
Walk Workflow B: download
nvidia/PhysicalAI-Autonomous-Vehicles-NuRec from HuggingFace,
then jump to serve-grpc + a Python client. Coordinate-frame
conversion code is in references/physical-ai-render.md.
Example 3 — Insert Asset-Harvester actors into a USDZ
Walk Workflow C: run asset-harvester, then
export-external-assets, edit edit-assets.json, and call
serve-grpc --enable-editing-actors + render-grpc --edit-assets. Schema lives in references/asset-editing.md.
Example 4 — Warm-server thin-client for repeated RGB renders
Walk the warm-server quick start at the bottom of
references/workflows.md. Boot
scripts/session_warm_server.sh, render with
thin_client.py, tear down with scripts/session_teardown.sh.
Backend Selection
Pick the smallest backend that exposes the requested feature:
- Local Docker, single command. Use
nre render, render-grpc,
or an export sub-command directly. Simplest for one-off renders,
LiDAR sweeps, actor edits, rolling shutter, in-container video
export, or exact --replicate-training-views behavior. See
references/local-render.md, references/nre-image-notes.md,
and references/mp4-encoding.md.
- Local Docker, warm
serve-grpc + thin host client. Use for
render-heavy RGB sessions where repeated Docker/Python/CUDA
cold-start dominates latency, or where multiple cameras should
be rendered through one batch_render_rgb RPC. See
references/NRE_RenderClient/README.md and
scripts/session_warm_server.sh / scripts/session_teardown.sh.
- OSMO / cluster workflows. Use the templates under
references/example-workflows/osmo/ for multi-clip fan-out,
isolation from the local machine, or training jobs that should
not run on the user's workstation. Follow
references/ngc-and-registry.md for registry credentials and
references/long-running-tasks.md for polling discipline.
Output Format
Structured deliverables placed under ${output_dir}/${RUN_ID}/
by the NRE container (no JSON state file required from the
agent):
config/parsed.yaml — Hydra-resolved training config.
checkpoints/last.ckpt (plus periodic snapshots).
val/metrics.yaml — per-frame PSNR / SSIM / LPIPS under
test/*.
val/*.mp4, val/<frame>/*.png — depth, opacity, segmentation,
RGB visualisations.
usd-out/last.usdz — USDZ containing the trained reconstruction,
data_info.json, rig_trajectories.json,
sequence_tracks.json, parsed_config.yaml, checkpoint.ckpt,
optional mesh.ply, and map.xodr. Render with nre render,
serve-grpc + render-grpc, the in-container viewer, or hand
to a downstream simulator (CARLA, AlpaSim, Isaac Sim).
*.ply / ego_mask/* / depth/* / sequence_tracks.json /
ncore_tracks.json / mesh.ply / ground_mesh.ply — produced
by the matching export sub-command.
Scripts
| Script |
Purpose |
Usage |
scripts/validate_setup.py |
Verify Docker, NVIDIA Container Toolkit, GPU/driver R570+ (R535+ minimum), NGC login, and NGC_API_KEY env var. No network calls. |
run_script("scripts/validate_setup.py") or python scripts/validate_setup.py [--strict] |
scripts/session_warm_server.sh |
Idempotently boot a session-scoped nre serve-grpc container for the thin Python client. Discovers a cached 26.04+ renderer image, mounts the USDZ root, waits for readiness. |
NRE_GRPC_USDZ_HOST_DIR=/path/to/usdz/root bash scripts/session_warm_server.sh |
scripts/session_teardown.sh |
Stop and remove the warm serve-grpc container and clear its state file without racing the next boot. |
bash scripts/session_teardown.sh |
References
Read these on demand; keep SKILL.md as the routing layer.
references/install.md — docker login nvcr.io, image pull, full prerequisite matrix, and safe
secret-handling for NGC_API_KEY / HF_TOKEN.
references/cookbook.md — most-used
docker run invocations: train + validate, re-validate with
shift, local render at quarter or native res, serve-grpc boot,
LiDAR sweep, in-container --help.
references/workflows.md — workflows
A – I end-to-end, plus the warm-server thin-client quick start.
references/troubleshooting.md
— extended error matrix (OOM, wandb blocking,
--artifact-glob mismatches, deprecated flags, gRPC LiDAR size,
etc.).
references/teardown.md — disk
cleanup, post-teardown verification, ownership-recovery.
references/cli-reference.md — full sub-command surface of the
NRE container (training, validation, render, serve-grpc,
render-grpc, render-novel-trajectory, every export-*,
upgrade-config / upgrade-artifact, gaussian-statistics,
eval-rendering-metrics, compute-metrics, viewer,
ply_viewer, profile-dataloader, run-script, the
nre-tools aux-data + AH entry points).
references/configuration.md — Hydra recipe map for Waymo / NV
/ PandaSet / Tesla / Alpasim, plus override matrix.
references/aux-data.md — nre-tools auxiliary-data CLI.
references/local-render.md — host-side docker run … render
recipes for rig offsets and export-custom-rig-trajectory.
references/carline-adaptation.md — augmented target-rig trajectory export,
local render, and optional nurec-fixer Harmonizer handoff.
references/NRE_RenderClient/README.md — warm-server thin
Python gRPC client.
references/grpc-api.md — sensorsim gRPC server flags + Python
client cookbook.
references/nre-image-notes.md — cached-image discovery, 26.04+
vs 26.03 vs pre-26.03 flags.
references/ngc-and-registry.md — NGC API key resolution.
references/mp4-encoding.md — host-side ffmpeg recipe.
references/asset-editing.md — export-external-assets +
edit-assets.json schema.
references/physical-ai-render.md — recipe for rendering the
HuggingFace NuRec dataset.
references/example-workflows/ — bash, Hydra, and OSMO
templates.
references/rig-json/ — bundled rig.json and
augmented_rig.json.
references/custom-rig-trajectories/ — pre-baked
export-custom-rig-trajectory outputs.
references/long-running-tasks.md — background-job + 5-minute
status reporting convention.
references/nurec-skill-catalog.md — routing table for sibling
NuRec-stack skills.
- Public product page: https://www.nvidia.com/en-us/omniverse/nurec/
- HF dataset: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec
- HF Fixer model: https://huggingface.co/nvidia/Difix3D
- NGC Fixer model card: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/models/nurec-fixer
Prerequisites
Linux x86_64 + NVIDIA GPU + Docker 23+ + NVIDIA Container Toolkit
1.13+ + NGC_API_KEY. Full matrix (driver minimums per arch,
shm-size, file ownership, GPU-tier guidance) lives in
references/install.md. Always verify
via scripts/validate_setup.py before pulling the image.
Limitations
- Linux x86_64 only. aarch64 (e.g. Jetson) is not supported.
- Internal source not redistributable. Use only the public
NGC containers and the public NuRec docs.
- Multi-GPU defaults are conservative. Set
trainer.world_size / trainer.num_nodes explicitly to scale
out; SLURM is auto-detected when both are 0. Quality plateaus
past ~6 GPUs; per release notes, multi-GPU +
dataset.aux_data=false is a known crash combination.
--config-name paths differ between train and val/export.
Training uses container-bundled recipes; validation and exports
re-pass the parsed.yaml written under
<output_dir>/<RUN-ID>/config/.
- Validation may prompt for
wandb. Choose option 3 to skip
in non-interactive runs, or pass logger=tensorboard /
logger=dummy.
- Render gRPC is data-format-pinned. Older releases warn /
reject artifacts that pre-date them; check release notes when
mixing client / server versions.
- Asset-Harvester input only.
export-external-assets
requires AH outputs; raw .ply files won't carry the
per-asset cuboid metadata.
- Difix variants are pluggable. The container ships both the
Cosmos Difix variant (default since 25.09 —
difix=cosmos_difix) and the legacy Stable-Diffusion variant
(difix=sd_difix). The newer Cosmos-Predict-based Fixer
variants live in the nurec-fixer skill.
render ↔ render-grpc overlap. render runs
in-container without a server; render-grpc requires an active
serve-grpc. Use render for batch novel-view jobs and
render-grpc when you need actor editing, LiDAR rendering, or
a long-lived service.
Troubleshooting (top 4)
| Error |
Cause |
Fix |
Unable to find image 'nvcr.io/nvidia/nre/nre:latest' |
Docker not authenticated to NGC. |
docker login nvcr.io with Username: $oauthtoken. |
OOM Killed / CUDA out of memory during training |
Default recipe needs >= 48 GB VRAM. |
Reduce dataset.camera_ids, lower trainer.max_epochs, switch to trainer.precision=16-mixed, or use a 48 GB+ GPU. |
serve-grpc fails to find the USDZ |
--artifact-glob must end in .usdz and be quoted. |
Use e.g. --artifact-glob /workdir/output/<RUN-ID>/usd-out/last.usdz. |
Edits silently ignored from render-grpc --edit-assets |
Server started without --enable-editing-actors. |
Restart serve-grpc with that flag. |
Full matrix in
references/troubleshooting.md.
Teardown
Full inventory, ownership-recovery, and post-teardown verification
commands live in references/teardown.md.
Headline: stop serve-grpc containers, docker image rm
nre/nre-tools, rm -rf ${HOME}/.cache/nre and your
<output_dir>/<RUN-ID>/. Do not revoke NGC_API_KEY unless you
suspect it has been leaked.
1---2name: nre3description: >-4 Use to drive NVIDIA Omniverse NuRec / Neural Reconstruction5 Engine (NRE) via the public NGC containers nvcr.io/nvidia/nre/nre6 and nvcr.io/nvidia/nre/nre-tools (NGC_API_KEY required) — train7 3DGUT Gaussian reconstructions from NCore clips, generate aux8 data, adapt an existing USDZ to an augmented target-vehicle rig9 (carline adaptation), render frames or LiDAR sweeps (local or warm10 `serve-grpc`), export PLY/depth/mesh/USDZ, edit actors, and evaluate11 metrics. Do NOT use for per-object asset capture (use12 `asset-harvester`) or sensor-to-NCore conversion (use `ncore`).13version: "0.2.2"14tools:15 - Shell16 - Read17 - Write18license: CC-BY-4.0 AND Apache-2.019compatibility: >-20 Linux x86_64, 1+ NVIDIA GPU (Ampere A100/A10/A40/RTX A6000, Ada21 L20/L40/L40S, Hopper H100/H20, or Blackwell RTX Pro 6000D) with22 CUDA 12.8 and >= 24 GB VRAM (48+ GB recommended); driver R570+23 recommended (R580+ on Blackwell, R535+ minimum for Fixer-only).24 Docker >= 23.0.1 + NVIDIA Container Toolkit >= 1.13.5, NGC25 account with NGC_API_KEY exported.26dependencies:27 - bash28 - docker29 - python330metadata:31 source: https://skillmd.com/skills/nvidia/nre32 author: NVIDIA NRS <nurec-skills@nvidia.com>33 tags:34 - nurec35 - autonomous-vehicles36 - neural-reconstruction37 - rendering38 - container39 product_page: https://www.nvidia.com/en-us/omniverse/nurec/40 ngc_container: nvcr.io/nvidia/nre/nre:latest41 ngc_tools_container: nvcr.io/nvidia/nre/nre-tools:latest42 ngc_fixer_model: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/models/nurec-fixer43 hf_datasets: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec44 hf_fixer_model: https://huggingface.co/nvidia/Difix3D45 release_date: "2026-04-30"46 time-estimate: "2h"47---4849# NRE — NVIDIA Omniverse NuRec (Neural Reconstruction Engine)5051## Purpose5253Drive the public NVIDIA Omniverse NuRec / Neural Reconstruction54Engine containers (`nvcr.io/nvidia/nre/nre`,55`nvcr.io/nvidia/nre/nre-tools`) to train a 3DGUT/3DGRT Gaussian56reconstruction from an NCore V4 camera+LiDAR clip, render novel57views (locally or via gRPC), generate aux data, export58PLY/depth/mesh/ego-mask/tracks, package Asset Harvester output59into a USDZ, and evaluate rendering metrics.6061This skill carries the host-side toolkit around the NRE CLI: NGC62credential resolution, cached-image notes, local render recipes,63MP4 encoding, warm `serve-grpc` boot/teardown scripts, a thin64Python gRPC client for repeated RGB renders, bundled rig JSONs,65pre-baked custom-rig trajectories, and bash / Hydra / OSMO66workflow templates.6768## When to Use / When NOT to Use6970**Use this skill when** the user has an NCore V4 clip (or a USDZ +71NRE artifact pair) on a Linux x86_64 host with an NVIDIA GPU and72an NGC API key, and wants to train, render, generate aux data,73export artifacts, insert/remove actors, run the gRPC server, or74evaluate metrics. Concrete triggers:7576- Train a multi-camera + LiDAR AV clip into a renderable USDZ77 scene with 3DGUT (or 3DGRT ray-traced) Gaussians.78- Generate NuRec auxiliary data (seg, depth, ego mask, DINOv2,79 LiDAR-seg visibility) using `nre-tools`.80- Render frames locally (no server) along the training rig or a81 custom rig + offsets.82- Render novel views via the sensorsim gRPC API (CARLA, Isaac83 Sim, AlpaSim, custom simulator), optionally with Difix84 artifact-removal.85- Render LiDAR sweeps via `render-grpc --lidar`.86- Export PLY / ego masks / depth / Poisson mesh / ground mesh /87 point clouds / cuboid tracks / NCore tracks / custom rig88 trajectories.89- Insert / remove / replace 3D actors with `export-external-assets`90 + `render-grpc --edit-assets`.91- Render the gated HF dataset92 `nvidia/PhysicalAI-Autonomous-Vehicles-NuRec`.93- Upgrade an old USDZ once (`upgrade-artifact`).94- Inspect / evaluate (`export-parsed-config`, `gaussian-statistics`,95 `eval-rendering-metrics`, `compute-metrics`,96 `eval-ground-mesh`).97- Browse a USDZ or PLY in the in-container viewer.9899**Do NOT use this skill when:**100101- The user still needs to convert raw sensor data into NCore V4102 (use the `ncore` skill first; NRE consumes NCore-formatted103 shards).104- The user wants per-object 3D asset extraction from sparse views105 (use `asset-harvester`; NRE only *consumes* AH outputs via106 `export-external-assets`).107- The user only needs to clean up already-rendered frames using108 the standalone Cosmos-based Fixer (use `nurec-fixer`). NRE's109 inline `--enable-difix` flag is still on this skill's surface,110 but the standalone harmonizer pipeline is owned by111 `nurec-fixer`.112113## Inputs114115- **dataset_dir** — host directory holding the NCore shards116 (`<NAME>.zarr.itar`, `<NAME>.json`, and any pre-generated117 `<NAME>.aux.*.zarr` auxiliary shards). Required.118- **dataset_name** — basename of the NCore dataset (the part119 before `.zarr.itar`). Required.120- **output_dir** — host directory NRE will fill with checkpoints,121 parsed config, metrics, videos, and USDZ artifacts. Required.122- **camera_ids / lidar_ids** — sensor IDs from the NCore JSON to123 include. Default: all sensors per recipe.124- **config_name** — Hydra config path resolved inside the125 container. Pick by source dataset:126 - **Waymo Open Dataset** →127 `configs/apps/AV/Waymo/3dgut_dynamic.yaml` (and its128 `_mcmc` / `_road_semantic` / `_static` siblings). These are129 Waymo-only — they bake in the Waymo sensor rig and conventions.130 - **NVIDIA Physical AI Autonomous Vehicles (PAI)** →131 `/apps/prod/Hyperion-8.1/car2sim_6cam.yaml` (the Hyperion-8.1132 car2sim 6-camera recipe used by the Maglev PAI pipeline).133 Typically referenced via the small overlay shipped at134 [`references/configs/pai.yaml`](references/configs/pai.yaml),135 which extends `car2sim_6cam.yaml` with PAI's136 `lidar_top_360fov` ID, six-camera validation set, and lidar137 `intensity` supervision; mount it as138 `{nre_config_dir}/external_overrides.yaml` and pass139 `--config-name=external_overrides`.140 - **PandaSet / NVIDIA AV (NV) / Tesla / Alpasim** → see the141 matching `configs/apps/AV/{PandaSet,NV,Tesla}/…` or142 `configs/apps/Alpasim/…` recipes in143 `references/configuration.md`.144 - Do **not** use the Waymo recipes for PAI clips — the145 sensor rig, validation cameras, and lidar IDs differ.146- **mode** — `train`, `val`, or `trainval`. Default: `trainval`.147- **NGC_API_KEY** — required env var. Generate at148 <https://org.ngc.nvidia.com/setup/api-keys>.149150## Instructions1511521. **Validate prerequisites.** Have the agent execute153 `scripts/validate_setup.py` via its standard script runner154 (`run_script("scripts/validate_setup.py")`, or155 `python scripts/validate_setup.py [--strict]`). It checks156 Docker, NVIDIA Container Toolkit, GPU/driver, and157 `NGC_API_KEY`. Resolve any FAIL line before pulling the image.1582. **Authenticate Docker to NGC + pull the public containers.**159 See [`references/install.md`](references/install.md). Pull both160 `nvcr.io/nvidia/nre/nre:latest` and161 `nvcr.io/nvidia/nre/nre-tools:latest`.1623. **Confirm input layout.** The dataset directory must contain163 `<NAME>.zarr.itar`, `<NAME>.json`, and any `<NAME>.aux.*.zarr`164 shards. If the NCore data is fresh, generate auxiliary data165 first — see `references/aux-data.md`.1664. **Train / validate the reconstruction.** Run the train recipe167 in [`references/cookbook.md`](references/cookbook.md) with the168 chosen Hydra `--config-name`, `mode`, `dataset.path`, and169 `out_dir`. For multi-GPU append `trainer.world_size=<N>170 trainer.num_nodes=<M>` (see Workflow D). Set171 `checkpoint.artifact.enabled=true` if you intend to render or172 serve the result.1735. **Export downstream artifacts.** Use export sub-commands174 (`export-gaussian-plys`, `export-mesh`, `export-ground-mesh`,175 `export-ego-mask`, `export-depth`, `export-sequence-tracks`,176 `export-ncore-tracks`, …) — full surface in177 `references/cli-reference.md`.1786. **Render novel views — pick the backend.**179 - **Local CLI** — `nre render --artifact-path <usdz>` writes180 frames on disk along the training trajectory, with optional181 rig offsets or `--custom-rig-trajectory`. No gRPC server.182 See `references/local-render.md`.183 - **Warm RGB service** — boot `serve-grpc` once with184 `scripts/session_warm_server.sh`, extract protobuf stubs,185 and use186 `references/NRE_RenderClient/scripts/thin_client.py` for187 repeated single-camera or `batch_render_rgb` calls.188 - **Remote CLI / simulator integration** — `serve-grpc` +189 `render-grpc` (or your own client via `nre.grpc.protos`).190 Required for LiDAR rendering, simulator loops, Difix, or191 `--edit-assets`. See `references/grpc-api.md` and192 `references/physical-ai-render.md`.1937. **Edit actors (optional).** Run `export-external-assets` to194 repackage Asset-Harvester output into a new USDZ, then pass195 the produced `edit-assets.json` to `render-grpc --edit-assets`196 (with `serve-grpc --enable-editing-actors`). See197 `references/asset-editing.md`.1988. **Validate the result.** Confirm199 `<output_dir>/<RUN-ID>/usd-out/last.usdz` opens, `metrics.yaml`200 reports a reasonable `test/psnr`, and the generated MP4s201 render. For more thorough metrics use Workflow I (eval) in202 [`references/workflows.md`](references/workflows.md). Tear203 down any gRPC server (`Ctrl-C` or `docker rm -f`).204205For any NRE task expected to run 5 minutes or longer (training,206OSMO jobs, multi-clip renders), follow207`references/long-running-tasks.md`: delegate to a subagent /208background job and report compact status at least every 5209minutes.210211## Examples212213### Example 1 — End-to-end NCore → USDZ → render214215Walk Workflow A in216[`references/workflows.md`](references/workflows.md): validate217host → generate aux data → train (cookbook recipe) → export →218local render or `serve-grpc`. Concrete commands live in the219referenced files; this index does not duplicate them.220221### Example 2 — Skip training, render the gated Physical AI dataset222223Walk Workflow B: download224`nvidia/PhysicalAI-Autonomous-Vehicles-NuRec` from HuggingFace,225then jump to `serve-grpc` + a Python client. Coordinate-frame226conversion code is in `references/physical-ai-render.md`.227228### Example 3 — Insert Asset-Harvester actors into a USDZ229230Walk Workflow C: run `asset-harvester`, then231`export-external-assets`, edit `edit-assets.json`, and call232`serve-grpc --enable-editing-actors` + `render-grpc233--edit-assets`. Schema lives in `references/asset-editing.md`.234235### Example 4 — Warm-server thin-client for repeated RGB renders236237Walk the warm-server quick start at the bottom of238[`references/workflows.md`](references/workflows.md). Boot239`scripts/session_warm_server.sh`, render with240`thin_client.py`, tear down with `scripts/session_teardown.sh`.241242## Backend Selection243244Pick the smallest backend that exposes the requested feature:245246- **Local Docker, single command.** Use `nre render`, `render-grpc`,247 or an export sub-command directly. Simplest for one-off renders,248 LiDAR sweeps, actor edits, rolling shutter, in-container video249 export, or exact `--replicate-training-views` behavior. See250 `references/local-render.md`, `references/nre-image-notes.md`,251 and `references/mp4-encoding.md`.252- **Local Docker, warm `serve-grpc` + thin host client.** Use for253 render-heavy RGB sessions where repeated Docker/Python/CUDA254 cold-start dominates latency, or where multiple cameras should255 be rendered through one `batch_render_rgb` RPC. See256 `references/NRE_RenderClient/README.md` and257 `scripts/session_warm_server.sh` / `scripts/session_teardown.sh`.258- **OSMO / cluster workflows.** Use the templates under259 `references/example-workflows/osmo/` for multi-clip fan-out,260 isolation from the local machine, or training jobs that should261 not run on the user's workstation. Follow262 `references/ngc-and-registry.md` for registry credentials and263 `references/long-running-tasks.md` for polling discipline.264265## Output Format266267Structured deliverables placed under `${output_dir}/${RUN_ID}/`268by the NRE container (no JSON state file required from the269agent):270271- `config/parsed.yaml` — Hydra-resolved training config.272- `checkpoints/last.ckpt` (plus periodic snapshots).273- `val/metrics.yaml` — per-frame PSNR / SSIM / LPIPS under274 `test/*`.275- `val/*.mp4`, `val/<frame>/*.png` — depth, opacity, segmentation,276 RGB visualisations.277- `usd-out/last.usdz` — USDZ containing the trained reconstruction,278 `data_info.json`, `rig_trajectories.json`,279 `sequence_tracks.json`, `parsed_config.yaml`, `checkpoint.ckpt`,280 optional `mesh.ply`, and `map.xodr`. Render with `nre render`,281 `serve-grpc` + `render-grpc`, the in-container `viewer`, or hand282 to a downstream simulator (CARLA, AlpaSim, Isaac Sim).283- `*.ply` / `ego_mask/*` / `depth/*` / `sequence_tracks.json` /284 `ncore_tracks.json` / `mesh.ply` / `ground_mesh.ply` — produced285 by the matching export sub-command.286287## Scripts288289| Script | Purpose | Usage |290|--------|---------|-------|291| `scripts/validate_setup.py` | Verify Docker, NVIDIA Container Toolkit, GPU/driver R570+ (R535+ minimum), NGC login, and `NGC_API_KEY` env var. No network calls. | `run_script("scripts/validate_setup.py")` or `python scripts/validate_setup.py [--strict]` |292| `scripts/session_warm_server.sh` | Idempotently boot a session-scoped `nre serve-grpc` container for the thin Python client. Discovers a cached 26.04+ renderer image, mounts the USDZ root, waits for readiness. | `NRE_GRPC_USDZ_HOST_DIR=/path/to/usdz/root bash scripts/session_warm_server.sh` |293| `scripts/session_teardown.sh` | Stop and remove the warm `serve-grpc` container and clear its state file without racing the next boot. | `bash scripts/session_teardown.sh` |294295## References296297Read these on demand; keep `SKILL.md` as the routing layer.298299- [`references/install.md`](references/install.md) — `docker300 login nvcr.io`, image pull, full prerequisite matrix, and safe301 secret-handling for `NGC_API_KEY` / `HF_TOKEN`.302- [`references/cookbook.md`](references/cookbook.md) — most-used303 `docker run` invocations: train + validate, re-validate with304 shift, local render at quarter or native res, `serve-grpc` boot,305 LiDAR sweep, in-container `--help`.306- [`references/workflows.md`](references/workflows.md) — workflows307 A – I end-to-end, plus the warm-server thin-client quick start.308- [`references/troubleshooting.md`](references/troubleshooting.md)309 — extended error matrix (`OOM`, `wandb` blocking,310 `--artifact-glob` mismatches, deprecated flags, gRPC LiDAR size,311 etc.).312- [`references/teardown.md`](references/teardown.md) — disk313 cleanup, post-teardown verification, ownership-recovery.314- `references/cli-reference.md` — full sub-command surface of the315 NRE container (training, validation, `render`, `serve-grpc`,316 `render-grpc`, `render-novel-trajectory`, every `export-*`,317 `upgrade-config` / `upgrade-artifact`, `gaussian-statistics`,318 `eval-rendering-metrics`, `compute-metrics`, `viewer`,319 `ply_viewer`, `profile-dataloader`, `run-script`, the320 `nre-tools` aux-data + AH entry points).321- `references/configuration.md` — Hydra recipe map for Waymo / NV322 / PandaSet / Tesla / Alpasim, plus override matrix.323- `references/aux-data.md` — `nre-tools` auxiliary-data CLI.324- `references/local-render.md` — host-side `docker run … render`325 recipes for rig offsets and `export-custom-rig-trajectory`.326- `references/carline-adaptation.md` — augmented target-rig trajectory export,327 local render, and optional `nurec-fixer` Harmonizer handoff.328- `references/NRE_RenderClient/README.md` — warm-server thin329 Python gRPC client.330- `references/grpc-api.md` — sensorsim gRPC server flags + Python331 client cookbook.332- `references/nre-image-notes.md` — cached-image discovery, 26.04+333 vs 26.03 vs pre-26.03 flags.334- `references/ngc-and-registry.md` — NGC API key resolution.335- `references/mp4-encoding.md` — host-side ffmpeg recipe.336- `references/asset-editing.md` — `export-external-assets` +337 `edit-assets.json` schema.338- `references/physical-ai-render.md` — recipe for rendering the339 HuggingFace NuRec dataset.340- `references/example-workflows/` — bash, Hydra, and OSMO341 templates.342- `references/rig-json/` — bundled `rig.json` and343 `augmented_rig.json`.344- `references/custom-rig-trajectories/` — pre-baked345 `export-custom-rig-trajectory` outputs.346- `references/long-running-tasks.md` — background-job + 5-minute347 status reporting convention.348- `references/nurec-skill-catalog.md` — routing table for sibling349 NuRec-stack skills.350- Public product page: <https://www.nvidia.com/en-us/omniverse/nurec/>351- HF dataset: <https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec>352- HF Fixer model: <https://huggingface.co/nvidia/Difix3D>353- NGC Fixer model card: <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/models/nurec-fixer>354355## Prerequisites356357Linux x86_64 + NVIDIA GPU + Docker 23+ + NVIDIA Container Toolkit3581.13+ + `NGC_API_KEY`. Full matrix (driver minimums per arch,359shm-size, file ownership, GPU-tier guidance) lives in360[`references/install.md`](references/install.md). Always verify361via `scripts/validate_setup.py` before pulling the image.362363## Limitations364365- **Linux x86_64 only.** aarch64 (e.g. Jetson) is not supported.366- **Internal source not redistributable.** Use only the public367 NGC containers and the public NuRec docs.368- **Multi-GPU defaults are conservative.** Set369 `trainer.world_size` / `trainer.num_nodes` explicitly to scale370 out; SLURM is auto-detected when both are `0`. Quality plateaus371 past ~6 GPUs; per release notes, multi-GPU +372 `dataset.aux_data=false` is a known crash combination.373- **`--config-name` paths differ between train and val/export.**374 Training uses container-bundled recipes; validation and exports375 re-pass the `parsed.yaml` written under376 `<output_dir>/<RUN-ID>/config/`.377- **Validation may prompt for `wandb`.** Choose option 3 to skip378 in non-interactive runs, or pass `logger=tensorboard` /379 `logger=dummy`.380- **Render gRPC is data-format-pinned.** Older releases warn /381 reject artifacts that pre-date them; check release notes when382 mixing client / server versions.383- **Asset-Harvester input only.** `export-external-assets`384 requires AH outputs; raw `.ply` files won't carry the385 per-asset cuboid metadata.386- **Difix variants are pluggable.** The container ships both the387 Cosmos Difix variant (default since 25.09 —388 `difix=cosmos_difix`) and the legacy Stable-Diffusion variant389 (`difix=sd_difix`). The newer Cosmos-Predict-based Fixer390 variants live in the `nurec-fixer` skill.391- **`render` ↔ `render-grpc` overlap.** `render` runs392 in-container without a server; `render-grpc` requires an active393 `serve-grpc`. Use `render` for batch novel-view jobs and394 `render-grpc` when you need actor editing, LiDAR rendering, or395 a long-lived service.396397## Troubleshooting (top 4)398399| Error | Cause | Fix |400|-------|-------|-----|401| `Unable to find image 'nvcr.io/nvidia/nre/nre:latest'` | Docker not authenticated to NGC. | `docker login nvcr.io` with `Username: $oauthtoken`. |402| `OOM Killed` / `CUDA out of memory` during training | Default recipe needs >= 48 GB VRAM. | Reduce `dataset.camera_ids`, lower `trainer.max_epochs`, switch to `trainer.precision=16-mixed`, or use a 48 GB+ GPU. |403| `serve-grpc` fails to find the USDZ | `--artifact-glob` must end in `.usdz` and be quoted. | Use e.g. `--artifact-glob /workdir/output/<RUN-ID>/usd-out/last.usdz`. |404| Edits silently ignored from `render-grpc --edit-assets` | Server started without `--enable-editing-actors`. | Restart `serve-grpc` with that flag. |405406Full matrix in407[`references/troubleshooting.md`](references/troubleshooting.md).408409## Teardown410411Full inventory, ownership-recovery, and post-teardown verification412commands live in [`references/teardown.md`](references/teardown.md).413Headline: stop `serve-grpc` containers, `docker image rm`414nre/nre-tools, `rm -rf ${HOME}/.cache/nre` and your415`<output_dir>/<RUN-ID>/`. Do not revoke `NGC_API_KEY` unless you416suspect it has been leaked.