NCore V4 Data Conversion
Purpose
Convert any sensor recording (cameras, LiDAR, radar, IMU, depth,
stereo, COLMAP/SfM, ROS 2 bag) into a valid NVIDIA NCore V4 store
so it can be consumed by NuRec / Asset Harvester / ncore_vis, or
wired into a robotics-to-sim ("r2s") pipeline. Drive the existing
in-tree converters (PAI, Waymo, COLMAP/ScanNet++) or author a new
converter from ncore_template/.
Use this skill when: the user has raw sensor data (any rig) that
NuRec or Asset Harvester needs to ingest, or when an existing
converter is failing validate.py / producing NuRec data-quality
complaints.
Do NOT use this skill when:
- The user is already on V4 and only wants to train or render
(use the
nre skill).
- The user wants per-object 3D assets from sparse views (use
asset-harvester).
- The user only needs to browse / pick an existing NVIDIA dataset
(use
physical-ai-datasets).
This skill teaches an agent to take any sensor dataset and produce a valid
NCore V4 store that NuRec / Asset Harvester / ncore_vis will accept. It covers
both driving the existing in-tree converters (PAI, Waymo, COLMAP/ScanNet++)
and writing a new one for unsupported formats (PandaSet, NuScenes, KITTI,
stereo, mono+depth, mono+lidar, custom robotics rigs).
Table of Contents
- When to use which path
- Install & references
- Mental model — the V4 store
- Path A — drive an existing in-tree converter
- Path B — author a new converter from the template
- V4 conventions you must obey
- Format recipes (AV)
- Format recipes (non-AV / sensor-only)
- Robotics pipeline shards (r2s)
- Validation & end-to-end NuRec
- Common failure modes (and the fix file)
- Additional resources
When to use which path
| You have… |
Use |
| PAI clip on HuggingFace, or a local PAI clip directory |
Path A — tools/data_converter/pai:convert (pai-stream-v4 / pai-v4) |
Waymo .tfrecord files |
Path A — tools/data_converter/waymo:convert (waymo-v4) |
| COLMAP scene (or ScanNet++ DSLR) |
Path A — tools/data_converter/colmap:convert (colmap-v4 / scannetpp-v4) |
| Mono RGB images, no poses |
Path A.5 — run COLMAP first, then colmap-v4 |
| PandaSet, NuScenes, KITTI, Argoverse, custom AV rig |
Path B — author from ncore_template/impl/data_converter/example_converter.py |
| Stereo / mono+depth / mono+lidar / robotics |
Path B |
| Already have parsed numpy/torch arrays in memory |
Path B but skip Bazel — use ncore.data.v4 API directly |
If a candidate path exists in upstream, prefer it. Hand-rolling a Waymo or
PAI converter on top of the template is wasted work and almost certainly wrong
(rolling-shutter timing, FTheta intrinsics, Waymo camera-frame rotation, etc).
Prerequisites
- Linux host with Python ≥ 3.10 and
pip.
git for the upstream converter sources.
bazel only if you run the in-tree converters (Path A); pure-Python
in-process writes (ncore.data.v4) need no Bazel.
- Disk: budget tens of GB per converted clip; pre-zarr scratch can be
larger than the final
.zarr.itar.
- HuggingFace token (
HF_TOKEN) only if pulling a gated PAI clip.
Verifying secrets safely
Always verify prerequisites with the upstream validate.py or by
running the converter against a tiny test slice; never write ad-hoc
bash that interpolates HF_TOKEN values. The common one-liner
# BAD — leaks the secret to the terminal when the variable is set
echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"
prints yes<token-value> whenever HF_TOKEN is set, because
${VAR:-no} only falls back to "no" when the variable is empty. Use
a length-only check, which never echoes the value:
# OK — prints "set (N chars)" or "missing", never the value
test -n "$HF_TOKEN" && echo "HF_TOKEN: set (${#HF_TOKEN} chars)" || echo "HF_TOKEN: missing"
Rotate any token you suspect was echoed at
https://huggingface.co/settings/tokens.
Install & references
pip install nvidia-ncore # pure-Python API for in-process writes
git clone --depth 1 https://github.com/NVIDIA/ncore.git # for upstream converters (Bazel)
Mental model — the V4 store
Every V4 sequence is one store (itar archive or plain directory) holding
component groups. The required components for NuRec are:
| Component |
What it carries |
API class |
Poses |
Dynamic T_rig_world (per timestamp); static T_sensor_rig per camera/lidar/radar; static T_world_world_global |
PosesComponent |
Intrinsics |
Per-camera model (Pinhole / Fisheye / FTheta) + per-LiDAR spinning model |
IntrinsicsComponent |
CameraSensor |
Encoded image bytes + per-frame [exposure_start, exposure_end] µs |
CameraSensorComponent |
LidarSensor |
Per-ray unit direction + per-ray µs timestamp + distance(s) + intensity + model_element=(row,col) |
LidarSensorComponent |
RadarSensor (optional) |
Same shape as LiDAR minus intensity / model_element |
RadarSensorComponent |
Cuboids (optional, recommended) |
CuboidTrackObservation list referencing rig / world / sensor frames |
CuboidsComponent |
Masks |
Per-camera dict of {name: PIL.Image} (NuRec requires ego masks) |
MasksComponent |
PointClouds (optional) |
Pre-computed dense or SfM points (e.g. COLMAP sfm_points, depth-derived) |
PointCloudsComponent |
Frames of reference (these are non-negotiable — wrong frames = silent NuRec failure):
- Rig:
+X forward, +Y left, +Z up. Origin at the middle of the rear axle on nominal ground for AV, or any natural body-fixed point for non-AV. All extrinsics are T_sensor → rig.
- Camera sensor:
+X right, +Y down, +Z forward (optical axis). NCore's convention. Waymo (X-fwd) and similar must be rotated before storing extrinsics.
- LiDAR model frame: azimuth 0° =
+X, 90° = +Y, +Z up. Independent of the raw sensor's native azimuth — you choose how column_azimuths_rad maps physical columns.
- World: sequence-local. Re-reference all
T_rig_world to the first ego pose so origin is near the vehicle start (raw UTM/ECEF at 10+ km loses precision in float32). Carry the original first pose into T_world_world_global (float64) if you need a global anchor.
- Image pixels:
u right, v down, origin at the top-left corner of the top-left pixel (so pixel centers are at 0.5, 0.5).
- Units: timestamps in µs everywhere, distances in metres, angles in radians.
The single source of truth is the spec — when in doubt, open it:
https://nvidia.github.io/ncore/data/conventions.html.
Sequence-level metadata
Some downstream pipelines need metadata beyond the per-component data — carry
it on the sequence's generic_meta_data (passed to
SequenceComponentGroupsWriter(...)):
Stereo pairs — required by stereo-depth modules (Foundation Stereo) and
multi-camera training configs to discover left/right pairings:
{"stereo_pairs": [{"left": "camera_front_left", "right": "camera_front_right"}]}
Multiple pairs are allowed for surround-stereo rigs.
Source tag — distinguishes real from synthetic data. Renderer-output
shards (NuRec sim) set {"source": "simulation", "model_checkpoint": "..."};
real-sensor shards omit the key or set {"source": "real"}. Downstream
validators key off this to skip "expected vs measured" checks on sim data.
Calibration / egomotion provenance — the example template writes
calibration_type and egomotion_type on the PosesComponent's
generic_meta_data. Use this to track the upstream tool (e.g.
egomotion_type: "cuvslam-stereo" or "kiss-icp" or "vio:orbslam3") so
later modules can pick refinement strategies that match the input quality.
Instructions
Pick one of the two paths below.
- Path A if the user's dataset format is already supported in
ncore/tools/data_converter/ (PAI, Waymo, COLMAP/ScanNet++) —
bootstrap the upstream repo and drive the existing binary.
- Path B if the format is unsupported (PandaSet, NuScenes, KITTI,
custom rig, robotics bag, …) — copy
ncore_template/ next to the
dataset and fill in the four hand-written hooks. The V4 conventions
in the Mental model section above are mandatory; the recipes
further down show typical configurations per rig.
After conversion, always run the Validation & end-to-end NuRec
section below before handing the store to NRE.
Path A — drive an existing in-tree converter
The upstream tools/data_converter/<format> modules are Bazel targets. Build
once, run per dataset.
Bootstrap
git clone --depth 1 https://github.com/NVIDIA/ncore.git
cd ncore
bazel build //tools/data_converter/pai:convert # or waymo, or colmap
Each convert binary takes shared base flags (--root-dir, --output-dir,
--no-cameras, --camera-id, --no-lidars, --lidar-id, --verbose) followed
by a subcommand (pai-v4, pai-stream-v4, waymo-v4, colmap-v4,
scannetpp-v4) with format-specific flags.
Standard sub-flags worth knowing
| Flag |
Default |
Meaning |
--store-type {itar,directory} |
itar |
itar is fastest for NuRec; directory is debuggable |
--profile {default,separate-sensors,separate-all} |
varies |
NuRec wants separate-sensors |
--sequence-meta / --no-sequence-meta |
enabled |
Writes <sequence>.json next to the store — NuRec/ncore_vis need it |
--world-global-mode {none,identity,localized} |
varies |
For NuRec releases that require the world→world_global edge, use identity (or localized to keep a real global anchor) |
When to script vs run interactively
For a one-off conversion, the bare bazel run form in each format recipe
below is enough. For repeatable cluster runs, wrap the same two steps
(bazel build then bazel run) inside an OSMO / Slurm / Kubernetes task that
clones NCore at a pinned ref, runs the convert step, and chains the result
into the nre training and aux-data containers (see
nre's Workflow A). The upstream
NVIDIA/ncore repo ships reference
converter targets that you can pin by Git commit for reproducibility.
Path B — author a new converter from the template
The scaffold is intentionally minimal but writes every required component
type with placeholder data. Treat it as a checklist: every # FILL IN is a
correctness gate — none can be skipped.
Scaffold
# Copy the scaffold next to your dataset (run from this skill's folder)
cp -r ncore_template /path/to/ncore-myformat
Then rename the package and class (ExampleConverter → MyFormatConverter) and
implement the contract:
| Method |
Contract |
get_sequence_ids(config) -> list[str] |
Discover sequence IDs from config.root_dir (or wherever your dataset lives — manifest CSV, HF clip index, ROS bag glob) |
from_config(config) -> Converter |
One-time setup (load calibration, open dataset index, init shared interpolators). Heavy lifting that all sequences share goes here |
convert_sequence(sequence_id) -> None |
Per-sequence work: open SequenceComponentGroupsWriter, register component writers, write data, finalize(), write <sequence_id>.json |
Inside convert_sequence the canonical order is Poses → Intrinsics → Masks →
Camera → LiDAR → Radar → Cuboids → finalize. This order is not required by the
writer but it surfaces calibration / pose / timing bugs before you've spent
minutes encoding image bytes.
The skeleton walks each step explicitly and lists every silent-correctness
trap inline — read these before filling them in:
- Spinning-LiDAR pitfalls (
spinning_direction, non-uniform row_elevations_rad,
column_azimuths_rad ordering, Ouster row_azimuth_offsets_rad):
example_converter.py:61-120
- Pose trajectory density + float64 → float32 + re-referencing rules:
example_converter.py:336-492
- Camera intrinsics for Pinhole / Fisheye / FTheta + shutter type:
example_converter.py:524-578
- Per-ray LiDAR timestamps and the three data shapes (range image / sensor-frame
XYZ / world-frame XYZ requiring decompensation):
example_converter.py:746-829
- Cuboid centroid convention (geometric center, not bottom-center):
example_converter.py:876-980
In-process API (no Bazel)
If you already have parsed arrays in Python and don't need a CLI, skip
FileBasedDataConverter entirely and call the V4 writer directly:
from ncore.data.v4 import SequenceComponentGroupsWriter, PosesComponent, ...
writer = SequenceComponentGroupsWriter(
output_dir_path=out / seq_id,
store_base_name=seq_id,
sequence_id=seq_id,
sequence_timestamp_interval_us=interval,
store_type="itar",
)
poses_writer = writer.register_component_writer(PosesComponent.Writer, ...)
# … write each component …
paths = writer.finalize()
The contract (component order, dtype rules, timestamp constraints) is identical.
V4 conventions you must obey
These are the rules that turn into runtime asserts (or worse: silent NuRec
artefacts). Cross-reference the spec before relaxing any of them.
Time
- Microseconds, uint64, everywhere.
np.uint64, not np.int64.
- The sequence interval is half-closed
[start, stop). Build it with
HalfClosedInterval.from_start_end(start, end_inclusive) — do not
pre-add 1 to end.
- Dynamic poses must exactly span the interval:
timestamps[0] == start and
timestamps[-1] == stop - 1. Sensors with timestamps slightly outside this
range are clamped at write time.
- Per-sensor frame timestamps are
[exposure_start, exposure_end] (cameras) or
[sweep_start, sweep_end] (LiDAR/radar). They must lie within the sequence
interval and the end must be unique within that sensor's writer.
- Rolling-shutter cameras:
start = trigger + half_shutter,
end = readout_done - half_shutter. Global shutter: start == end is OK.
- LiDAR
frame_timestamps_us[0] is the sweep start (not midpoint). Per-ray
timestamp_us must lie in [sweep_start, sweep_end]. Treating the dataset's
frame timestamp as the midpoint and subtracting half a sweep duration
introduces a ~50 ms shift on a 10 Hz LiDAR and produces motion-comp blur.
Pose graph
- All intermediate pose math is float64. Cast to float32 only as the very
last step before
store_dynamic_pose / store_static_pose.
Exception: world → world_global stays float64 (NuRec's
RigTrajectories.T_world_base is float64).
- Re-reference dynamic poses to the first ego pose:
poses_f64 = inv(poses_f64[0]) @ poses_f64. Without this, GPS/UTM/ENU at
10+ km loses sub-cm precision once cast to float32.
- The pose trajectory must be dense (waypoint spacing < LiDAR sweep
duration, e.g. <50 ms for 10 Hz). Combine every available source: per-camera
per-frame poses, per-LiDAR per-sweep poses, IMU/GPS/odometry. Concatenate,
np.unique by timestamp, sort.
T_camera_rig and T_lidar_rig are float32 (NuRec's transform_poses leaks
input dtype through an internal matmul; float64 here collides with the
float32 trajectory and crashes "Get Lidar Point Clouds" with
RuntimeError: double != float).
LiDAR
- Direction vectors are unit-norm, in sensor coordinates, at each ray's
measurement time (not the sweep-start time). Three input shapes:
- Polar range image with beam geometry → derive directly from
column_azimuths_rad[col], row_elevations_rad[row],
row_azimuth_offsets_rad[row]. Direction:
(cos(elev)*cos(azi), cos(elev)*sin(azi), sin(elev)).
- Sensor-frame XYZ →
direction = xyz / |xyz|.
- World-frame or ego-compensated XYZ → decompensate with
MotionCompensator.motion_decompensate_points, then normalise.
- Spinning LiDAR parameter traps (silent — wrong values pass write but fail
NuRec):
spinning_direction: nearly all automotive spinning LiDARs (Velodyne,
Hesai, Ouster, Robosense) are "cw". The template defaults to
"ccw" to force the question. Wrong value mirrors Y → Z-flip in NuRec.
row_elevations_rad: non-uniform on every common sensor. Read real
per-beam angles from the sensor calibration. Strictly descending
(highest beam first); reverse if your source is ascending. NCore asserts
np.diff > 0 after projecting through relative_angle.
column_azimuths_rad: must reflect the actual per-column azimuth at
frame start (not a synthetic linspace). NCore validates strict ordering
via relative_angle(azim[0], azim, spinning_direction). If raw
per-column heading is available, use it; otherwise linspace(0, ±2π, N, endpoint=False) is acceptable for "ccw" / "cw" respectively.
row_azimuth_offsets_rad: zero for most sensors; non-zero on Ouster
(beam_azimuth_angles from the HTTP API). Skipping this on Ouster
misaligns LiDAR-to-camera projection.
- Per-ray
timestamp_us is required for motion compensation; supply real
per-column firing times (typically column-linear across the sweep).
Cuboids
BBox3.centroid is the geometric center. Many AV datasets use
bottom-center → add dim_z / 2 to z. Verify empirically: mean(z) - mean(h)/2
near 0 = geometric center; near mean(h)/2 above ground = bottom-center.
reference_frame_id ∈ {"rig", "world", <sensor_id>}.
reference_frame_timestamp_us and timestamp_us must lie within the
sequence interval.
LabelSource: tag the origin pipeline, not quality. Third-party
dataset labels (Waymo, NuScenes, PandaSet) → EXTERNAL, even if the upstream
is human GT. Use GT_ANNOTATION only when this converter's team owns the
annotation.
Camera
- Image bytes stored verbatim (JPEG/PNG); NCore does not re-encode.
- Use the model that matches your sensor:
OpenCVPinholeCameraModelParameters
(most AV cameras), OpenCVFisheyeCameraModelParameters (Kannala-Brandt
fisheye, e.g. ScanNet++ / GoPro), FThetaCameraModelParameters (NVIDIA
Hyperion / equidistant-radial). Resolution must match the actual image bytes
bit-for-bit.
- Map shutter direction by enum name, not integer cast. The five
ShutterType values are ROLLING_TOP_TO_BOTTOM=1, ROLLING_LEFT_TO_RIGHT=2,
ROLLING_BOTTOM_TO_TOP=3, ROLLING_RIGHT_TO_LEFT=4, GLOBAL=5. Build a
source-enum-to-name dict.
- Provide ego masks. NuRec's data-quality guide requires binary
ego-vehicle masks per camera. Without them the hood/roof rack leaks into the
reconstruction.
NuRec data-quality minimums
Per the NuRec "Ensure Data Quality" guide, hitting these is the difference
between a clean reconstruction and visible artefacts:
- Camera extrinsics: < 0.5° rotation, < 2 cm translation (relative to rig).
- Camera intrinsics: < 1 px reprojection error.
- Egomotion: < 0.5° / < 2.5 cm consecutive-frame error; trajectory must cover
every sensor frame's start and end timestamp.
- Cuboids: < 1° / < 5 cm position / < 5 cm dimension; per-track stable
track_ids.
- Original sensor resolution at full FPS, no undistortion / rectification.
Format recipes (AV)
PAI (NVIDIA Physical AI Autonomous Vehicles, HuggingFace)
Built-in. Streaming or local. Uses Hyperion 8 / 8.1 sensor IDs.
# Streaming (no download — recommended for cloud / OSMO)
bazel run //tools/data_converter/pai:convert -- \
--output-dir <OUT> \
--camera-id camera_front_wide_120fov \
--camera-id camera_cross_left_120fov \
--camera-id camera_cross_right_120fov \
--camera-id camera_front_tele_30fov \
pai-stream-v4 \
--clip-id <CLIP_ID> \
--hf-token "$HF_TOKEN" \
--store-type itar \
--profile separate-sensors \
--sequence-meta
Notes:
- HF dataset:
nvidia/PhysicalAI-Autonomous-Vehicles. License must be accepted
on HF before HF_TOKEN works.
- 7 cameras (FTheta + Fisheye + Pinhole depending on FOV), 1 top LiDAR
(
lidar_top_360fov, spinning, see datasheet for elevations).
- Camera intrinsics include
shutter_delay_us for per-row rolling-shutter
timestamping, plus optional BivariateWindshieldModelParameters for
windshield refraction.
- Output path:
<output-dir>/pai_<clip-id>/pai_<clip-id>.ncore4.zarr.itar.
- Discover clip IDs via the
clip_index.parquet blob in the HF dataset
(nvidia/PhysicalAI-Autonomous-Vehicles) — the
physical-ai-datasets skill has the
download recipe and toolkit pointers.
- Subset by time with
--seek-sec / --duration-sec; subset sensors with
--no-lidars / --camera-id (repeatable).
- car2sim_6cam sim configs target CARLA sensor names; PAI exports use real
Hyperion IDs — when feeding NRE training, override
dataset.camera_ids /
dataset.lidar_ids on the NRE Hydra command line (see
nre Workflow A and references/configuration.md).
Waymo Open
Built-in. Reads .tfrecord segment files.
bazel run //tools/data_converter/waymo:convert -- \
--root-dir <DIR_OF_TFRECORDS> \
--output-dir <OUT> \
waymo-v4 \
--store-type itar \
--profile separate-sensors \
--world-global-mode localized # or 'identity' / 'none'
Notes:
- 5 cameras (
camera_front_50fov, camera_front_left_50fov,
camera_front_right_50fov, camera_side_left_50fov,
camera_side_right_50fov) + 1 top LiDAR (lidar_top).
- Camera intrinsics →
OpenCVPinholeCameraModelParameters. Waymo's local
camera frame is +X principal axis; the converter rotates to NCore's
+Z principal axis. If you re-derive extrinsics manually, apply the same
rotation to T_camera_rig.
- Cuboid classes:
unknown, vehicle, pedestrian, sign, cyclist. Map → NCore
class_id strings; tag LabelSource.EXTERNAL.
- Multi-return LiDAR (primary + secondary). Stack into the
[R, N] distance /
intensity arrays.
PandaSet
No upstream converter — author with Path B. PandaSet (Hesai +
Scale-AI-labelled) ships JSON metadata with each sequence.
Sensor inventory:
- 6 cameras (
front_camera, front_left_camera, front_right_camera,
back_camera, left_camera, right_camera) — pinhole, global shutter.
Intrinsics in meta/intrinsics/<id>.json.
- 2 LiDARs:
front_lidar: Hesai PandarGT (mechanical, 60° HFOV, 150° VFOV
forward-facing). Treat as a partial spinning sensor (column_azimuths_rad
spans the 60° wedge).
top_lidar (key sensor for AV reconstruction): Hesai Pandar64,
spinning_direction="cw", 64 beams with non-uniform elevations
(Pandar64 datasheet table — copy the 64 angles, sort descending),
row_azimuth_offsets_rad = zeros(64), 1800 columns.
Data layout per sequence (extract):
<seq_id>/
├── meta/
│ ├── intrinsics/<camera_id>.json # focal, principal point, distortion
│ ├── timestamps.json # per-camera-frame µs
│ └── poses.json # per-frame ego pose (UTM)
├── camera/<camera_id>/<frame>.jpg
├── camera/<camera_id>/poses.json # per-frame camera pose in world
├── lidar/<frame>.pkl.gz # XYZ + intensity + timestamp + ring
└── cuboids/<frame>.pkl.gz # bottom-center xyz + dim + yaw
Conversion checklist:
- Pose trajectory: union of per-camera per-frame poses (6 × ~80 = ~480
waypoints) and per-LiDAR-frame poses, deduped + sorted, re-referenced to
the first ego pose.
- LiDAR points are world-frame XYZ already ego-compensated → run
MotionCompensator.motion_decompensate_points to recover sensor-frame
per-ray XYZ, then normalise to direction. Per-ray µs from the timestamp
column.
- Map raw Pandar64
ring → NCore model_element[:, 0] via a
ring_id → row_index permutation that sorts elevations descending (raw
ring IDs are firing order, not beam index).
- Cuboids: bottom-center → geometric center (
centroid_z += dim_z/2).
Yaw-only rotation → rot=(0, 0, yaw_rad). LabelSource.EXTERNAL.
- Cameras: global shutter (
ShutterType.GLOBAL). frame_timestamps_us = [t, t].
NuScenes
No upstream converter — author with Path B. NuScenes ships nested JSON
tables (sample, sample_data, ego_pose, calibrated_sensor, sensor,
sample_annotation, instance).
Sensor inventory:
- 6 cameras (
CAM_FRONT, CAM_FRONT_LEFT, CAM_FRONT_RIGHT, CAM_BACK,
CAM_BACK_LEFT, CAM_BACK_RIGHT) — pinhole. Rolling shutter; consult the
device datasheet for shutter direction (rename to lowercase NCore IDs).
- 1 LiDAR (
LIDAR_TOP): Velodyne HDL-32E, spinning_direction="cw",
32 beams, 1800 columns at 0.2°. Use VLP/HDL-32 datasheet elevations
(descending). row_azimuth_offsets_rad = zeros(32).
- 5 radars (
RADAR_FRONT, RADAR_FRONT_LEFT, RADAR_FRONT_RIGHT,
RADAR_BACK_LEFT, RADAR_BACK_RIGHT). Continental ARS 408. Optional —
NuRec ignores radar.
Conversion checklist:
- A NuScenes "scene" → one V4 sequence. Iterate
samples in the scene to
enumerate sample_data per sensor.
- Timestamps in
sample_data.timestamp are µs already.
ego_pose is rig→world (translation + quaternion). Build the dense
trajectory from the union of ego_pose entries across all sample_data
(cameras at 12 Hz, LiDAR at 20 Hz, radars at 13 Hz → ~50 ms spacing).
calibrated_sensor.translation + .rotation is T_sensor_rig (named
T_calib in their docs — verify direction by transforming a sensor-frame
test point and checking it lands in the expected rig position).
- LiDAR
.pcd.bin files are already-ego-compensated sensor-frame XYZ + ring
index + intensity. Per-ray timestamps are not stored — synthesise from
column_index (azimuth bin) and the sweep duration (50 ms at 20 Hz):
t_ray = sweep_start + (azim_bin / 1800) * sweep_duration.
sample_annotation: bottom-of-box origin → add size[2]/2 to z.
rotation quaternion → XYZ-Euler radians. instance_token → track_id,
category_name → class_id. LabelSource.EXTERNAL.
- Ego masks: NuScenes does not ship them. Generate via SAM2 / off-the-shelf
ego segmentation per camera — strongly recommended for NuRec quality.
Format recipes (non-AV / sensor-only)
These do not have a vehicle rig; pick a body-fixed origin and apply the same
rig conventions (+X forward, +Y left, +Z up).
Mono camera (no depth, no LiDAR) → COLMAP track
You only have RGB images. Run COLMAP first to produce poses + sparse points,
then feed COLMAP into the upstream converter:
colmap automatic_reconstructor \
--workspace_path <SCENE> --image_path <SCENE>/images \
--camera_model OPENCV --single_camera 1
bazel run //tools/data_converter/colmap:convert -- \
--root-dir <SCENE> --output-dir <OUT> \
colmap-v4 --include-3d-points --start-time-sec 0
Notes:
- COLMAP timestamps are synthetic (1 FPS by default; tune with
--start-time-sec and the FPS embedded in your image filenames).
- The COLMAP camera frame already matches NCore (
+Z optical) — no rotation
required.
- SfM points become a
PointCloudsComponent named sfm_points. Use this when
no LiDAR is available so NuRec has a sparse geometric prior.
- Ego masks (
<image_basename>_mask.png next to images, or --masks-dir).
Stereo cameras
Two synchronised cameras at known baseline. No LiDAR.
- Rig origin: midpoint between the two camera optical centres (or the left
camera — pick one and stay consistent).
- Static extrinsics:
T_left_rig, T_right_rig from your stereo calibration
(OpenCV stereoCalibrate outputs R, T from right-to-left → invert/compose
to get rig-relative).
- Trajectory: stereo-VIO (ORB-SLAM3, OpenVSLAM) or COLMAP run on left
images → propagate to rig with
T_left_rig.
- No LiDAR component. Two options:
- Skip LiDAR entirely (set
lidar_ids = []). NuRec falls back to
image-only reconstruction (lower quality, more views needed).
- Compute disparity per stereo pair → 3-D point cloud per frame in the
left-camera frame → write as a
PointCloudsComponent per frame
(analogous to COLMAP sfm_points but dense). This gives NuRec a
geometric prior without faking a spinning LiDAR.
- Ego masks: render or hand-paint a static mask of the rig body if visible in
the FOV.
Multi-stereo rig (surround stereo)
Multiple synchronised stereo pairs on one rig (e.g. NVIDIA Hyperion 8.1
surround stereo, AV1, custom inspection robots). Each pair feeds Foundation
Stereo independently to produce dense depth around the platform.
Encode each camera as its own CameraSensorComponent with a standard
T_camera_rig extrinsic — do not pre-rectify or fuse a pair into a
single virtual sensor. NCore does not have a "stereo pair" component
type; pairing is metadata, not structure.
Declare the pairings on the sequence-level generic_meta_data so
downstream tools (Foundation Stereo, NuRec aux-data) can discover them:
{"stereo_pairs": [
{"left": "camera_front_left", "right": "camera_front_right"},
{"left": "camera_rear_left", "right": "camera_rear_right"},
{"left": "camera_side_left_a", "right": "camera_side_left_b"}
]}
Pose trajectory must include every (camera × frame timestamp) sample —
surround stereo at 8 cameras × 100 frames yields 800 trajectory waypoints,
more than dense enough for per-ray motion compensation. Do not subsample.
Calibration: stereo intrinsics + extrinsics from cv2.stereoCalibrate give
you R, T from right-to-left. Compose with your chosen rig origin to get
T_left_rig and T_right_rig separately; do not store only the
baseline.
Scale is metric by construction (calibrated baseline). Skip scale refinement
(or run it as a sanity check only).
Mono + depth (RGB-D / learned depth)
Single RGB camera + per-frame depth (sensor: RealSense, Kinect, ZED depth, or
learned mono-depth like Marigold / DepthAnythingV2 / MoGe-2).
RGB-D specifics:
RealSense D4xx / L515: active IR stereo (D4xx) or LiDAR-class TOF (L515).
Depth and RGB are co-triggered but not pixel-aligned out of the box —
use the rs2_align filter (or pre-aligned topics) before treating depth
as RGB-frame metric. Intrinsics: read the colour stream's intrinsics for
the RGB component; ignore the depth-stream intrinsics (depth is aligned
into the colour frame).
Microsoft Kinect Azure / Kinect v2: TOF depth with non-trivial
invalidation near object edges. Mask invalidated pixels (depth == 0) before
encoding as a point cloud.
Stereolabs ZED 2 / X: stereo with a built-in disparity engine. Either
store both raw left/right images and treat as a stereo rig (preferred —
Foundation Stereo can re-derive depth at higher quality), or store the
ZED-native depth as a per-frame PointCloudsComponent.
Learned mono-depth (Marigold, DepthAnythingV2, MoGe-2) is scale
ambiguous. Either anchor with one absolute reference (a known object
size, ground-plane height, IMU + visual-inertial scale) before storing,
or accept that scale refinement (r2s module 6) will run downstream.
Rig origin: camera optical centre (or device body if you have a static
IMU offset).
Trajectory: ARKit/ARCore pose stream, IMU+camera VIO, or depth-aided
RGB-D SLAM (Open3D, Spectacular AI).
No LiDAR — depth is not a LiDAR. Two valid encodings:
- Per-frame
PointCloudsComponent (preferred for learned/stereo-quality
depth, where reliability is uneven). Convert depth + intrinsics →
camera-frame XYZ, transform to world via T_camera_world(t), store as
a point cloud per frame. Carry the dense depth into generic_data if
downstream consumers want it.
- Synthetic spinning LiDAR (only if your depth is dense and reliable).
Sample a fixed grid of azimuths/elevations, ray-cast against the depth
map at each frame, and write a
LidarSensorComponent. This is more
work and less honest than option 1 — prefer point clouds unless NuRec
specifically needs a LiDAR component.
Camera intrinsics: pinhole or fisheye depending on the lens. Depth-camera
manufacturers ship calibration JSON — copy fx, fy, cx, cy and distortion
coefficients verbatim.
Frame timestamps: depth and RGB are usually co-triggered. Use the RGB
exposure timestamp; depth has no separate component.
Mono + LiDAR (handheld / robot)
Single camera + spinning or solid-state LiDAR, e.g. handheld scanner, ground
robot, drone.
- Rig origin: choice driven by mechanical mounting. If the LiDAR is the
reference for ego-motion (LiDAR-inertial SLAM), set rig = LiDAR (so
T_lidar_rig = I). Otherwise pick the IMU body frame or camera centre.
- Trajectory: LIO-SAM, FAST-LIO, or any LiDAR-inertial pipeline.
Re-reference to the first frame; cast last (float64 → float32).
- Static extrinsics:
T_camera_rig from camera-LiDAR calibration
(kalibr, lidar_align). Apply the NCore camera-frame rotation if your
calibration target uses a different convention.
- LiDAR:
- Spinning (Velodyne / Ouster / Hesai / Robosense / Livox Mid-360 in
repetitive mode): use
RowOffsetStructuredSpinningLidarModelParameters
with the sensor's real elevations (datasheet) and "cw". For Ouster
populate row_azimuth_offsets_rad from beam_azimuth_angles.
- Solid-state non-repetitive (Livox Avia, Mid-40, Mid-70 in
non-repetitive mode): the spinning model does not fit. Either replay
each scan as a fake "spinning" sweep (stash directions in
column_azimuths_rad per-scan — fragile) or, preferred, write the
points as a PointCloudsComponent per frame in the sensor frame and
skip LidarSensorComponent. NuRec can consume point clouds.
- Cuboids: usually unavailable for non-AV — leave the component empty or
unregistered.
- Ego masks: render the rig body if it intrudes on the FOV (drone arm,
robot chassis); empty
{} if not.
Solid-state / non-repetitive LiDAR (Livox)
Livox Avia / Mid-40 / Mid-70 — and Mid-360 in non-repetitive mode — produce
a point cloud per scan that does not lay out on a row-major spinning
grid. The RowOffsetStructuredSpinningLidarModel does not fit; forcing it
(synthetic columns, fake row bins) breaks NCore validation and
motion-compensation alignment.
Encoding rules:
- Preferred — write each scan as a per-frame
PointCloudsComponent
instance in the sensor frame, with per-point µs timestamps. Skip
LidarSensorComponent entirely. NuRec consumes point clouds and r2s depth
refinement (module 7) treats them as it would LiDAR sweeps.
- Fallback for repetitive Mid-360 — the Mid-360 in repetitive mode does
produce a structured grid; treat it as a spinning LiDAR with the
datasheet's beam table and
spinning_direction="cw". This is the only
Livox variant the spinning model fits.
Per-point timestamps: Livox custom messages carry offset_time (ns from
sweep start). Convert to absolute µs:
timestamp_us = sweep_start_us + offset_time_ns // 1000.
IMU + camera (visual-inertial)
IMU is not stored as its own NCore component — it densifies the pose
trajectory (and, optionally, anchors metric scale on monocular setups). Two
paths:
- VIO trajectory (preferred): run a visual-inertial pipeline (cuVSLAM
stereo-inertial, ORB-SLAM3, OpenVSLAM, OpenVINS, Spectacular AI) →
IMU-rate (100–200 Hz)
T_world_rig poses → store as the dynamic pose.
The trajectory is already dense enough that per-ray motion compensation
works without further densification.
- IMU integration only (bootstrap / fallback): pre-integrate IMU
(linear accel + angular vel) over short windows, anchor each window with
the next available camera- or LiDAR-rate pose. Useful for filling gaps or
extrapolating to per-LiDAR sweep timestamps when the SLAM stack only
emits poses at frame rate.
Either way, IMU sample timestamps go into the pose trajectory, not into
a separate component. If a downstream consumer wants raw IMU, stash the
samples in the sequence-level generic_meta_data["imu_samples"] (compact)
or in a sidecar file referenced from there.
Calibration: IMU-to-camera and IMU-to-LiDAR extrinsics live in the static
pose graph as T_imu_rig = I if you take the IMU body frame as the rig
(common for legged robots and drones), with T_camera_rig /
T_lidar_rig from kalibr / lidar_align outputs. Pick rig = IMU when
the IMU is the trajectory reference; otherwise pick the mechanical body
frame and store T_imu_rig for downstream tools that want IMU-frame data.
ROS2 bag (MCAP / SQLite3)
Most robotics datasets ship as ROS2 bags. The standard path is the
rosbags Python library — no rclpy and no native ROS install required:
pip install rosbags pyav # pyav decodes H.264 video chunks; rosbags handles .mcap and .db3
Convert in two passes: first enumerate sensors and collect calibration /
the static TF tree, then stream frames into the V4 writer. Common topic →
component mapping:
| ROS2 message type |
NCore mapping |
sensor_msgs/Image (raw) |
CameraSensorComponent.store_frame — r |
…(truncated)
1---2name: ncore3description: >-4 Use when converting any sensor dataset into NVIDIA NCore V4 format5 (and feeding it to NuRec or a robotics-to-sim "r2s" pipeline).6 Covers ingesting raw cameras, LiDARs, radars, IMUs, depth or stereo7 into V4 sequences; authoring a new converter from the template;8 adapting PAI / Waymo / PandaSet / NuScenes to V4; handling non-AV9 rigs (mono+depth, mono+lidar, stereo, multi-stereo, RGB-D, COLMAP /10 SfM, ROS2 bag); and diagnosing a broken converter against11 `validate.py`. Do NOT use to train reconstructions (use `nre`) or12 to extract per-object 3D assets (use `asset-harvester`). Trigger13 keywords: ncore, ncore v4, convert, ingest, zarr, itar, nurec,14 waymo, pandaset, nuscenes, pai, hyperion, colmap, scannetpp,15 stereo, multi-stereo, mono+depth, mono+lidar, kitti, sfm, camera,16 lidar, radar, imu, cuboid, poses, intrinsics, ego mask, ros2,17 rosbag, mcap, realsense, zed, rgb-d, r2s, robotics, sam2.18version: "0.1.0"19tools:20 - Shell21 - Read22 - Write23license: CC-BY-4.0 AND Apache-2.024metadata:25 source: https://skillmd.com/skills/nvidia/ncore26 author: NVIDIA NCore27 tags:28 - ncore29 - data-conversion30 - autonomous-vehicles31 - robotics32 - sensors33 upstream: https://github.com/NVIDIA/ncore34 spec_docs: https://nvidia.github.io/ncore/data/conventions.html35 release_tag: "2026.04"36---3738# NCore V4 Data Conversion3940## Purpose4142Convert any sensor recording (cameras, LiDAR, radar, IMU, depth,43stereo, COLMAP/SfM, ROS 2 bag) into a valid NVIDIA **NCore V4** store44so it can be consumed by NuRec / Asset Harvester / `ncore_vis`, or45wired into a robotics-to-sim ("r2s") pipeline. Drive the existing46in-tree converters (PAI, Waymo, COLMAP/ScanNet++) or author a new47converter from `ncore_template/`.4849**Use this skill when:** the user has raw sensor data (any rig) that50NuRec or Asset Harvester needs to ingest, or when an existing51converter is failing `validate.py` / producing NuRec data-quality52complaints.5354**Do NOT use this skill when:**5556- The user is **already on V4** and only wants to train or render57 (use the `nre` skill).58- The user wants per-object 3D assets from sparse views (use59 `asset-harvester`).60- The user only needs to browse / pick an existing NVIDIA dataset61 (use `physical-ai-datasets`).6263This skill teaches an agent to take **any** sensor dataset and produce a valid64NCore V4 store that NuRec / Asset Harvester / `ncore_vis` will accept. It covers65both **driving the existing in-tree converters** (PAI, Waymo, COLMAP/ScanNet++)66and **writing a new one** for unsupported formats (PandaSet, NuScenes, KITTI,67stereo, mono+depth, mono+lidar, custom robotics rigs).6869## Table of Contents70711. [When to use which path](#when-to-use-which-path)722. [Install & references](#install--references)733. [Mental model — the V4 store](#mental-model--the-v4-store)744. [Path A — drive an existing in-tree converter](#path-a--drive-an-existing-in-tree-converter)755. [Path B — author a new converter from the template](#path-b--author-a-new-converter-from-the-template)766. [V4 conventions you must obey](#v4-conventions-you-must-obey)777. [Format recipes (AV)](#format-recipes-av)78 - [PAI (NVIDIA Physical AI Autonomous Vehicles, HuggingFace)](#pai-nvidia-physical-ai-autonomous-vehicles-huggingface)79 - [Waymo Open](#waymo-open)80 - [PandaSet](#pandaset)81 - [NuScenes](#nuscenes)828. [Format recipes (non-AV / sensor-only)](#format-recipes-non-av--sensor-only)83 - [Mono camera (no depth, no LiDAR) → COLMAP track](#mono-camera-no-depth-no-lidar--colmap-track)84 - [Stereo cameras](#stereo-cameras)85 - [Multi-stereo rig (surround stereo)](#multi-stereo-rig-surround-stereo)86 - [Mono + depth (RGB-D / learned depth)](#mono--depth-rgb-d--learned-depth)87 - [Mono + LiDAR (handheld / robot)](#mono--lidar-handheld--robot)88 - [Solid-state / non-repetitive LiDAR (Livox)](#solid-state--non-repetitive-lidar-livox)89 - [IMU + camera (visual-inertial)](#imu--camera-visual-inertial)90 - [ROS2 bag (MCAP / SQLite3)](#ros2-bag-mcap--sqlite3)91 - [Aerial / drone](#aerial--drone)929. [Robotics pipeline shards (r2s)](#robotics-pipeline-shards-r2s)9310. [Validation & end-to-end NuRec](#validation--end-to-end-nurec)9411. [Common failure modes (and the fix file)](#common-failure-modes-and-the-fix-file)9512. [Additional resources](#additional-resources)9697---9899## When to use which path100101| You have… | Use |102|-----------|-----|103| PAI clip on HuggingFace, or a local PAI clip directory | **Path A** — `tools/data_converter/pai:convert` (`pai-stream-v4` / `pai-v4`) |104| Waymo `.tfrecord` files | **Path A** — `tools/data_converter/waymo:convert` (`waymo-v4`) |105| COLMAP scene (or ScanNet++ DSLR) | **Path A** — `tools/data_converter/colmap:convert` (`colmap-v4` / `scannetpp-v4`) |106| Mono RGB images, no poses | **Path A.5** — run COLMAP first, then `colmap-v4` |107| PandaSet, NuScenes, KITTI, Argoverse, custom AV rig | **Path B** — author from `ncore_template/impl/data_converter/example_converter.py` |108| Stereo / mono+depth / mono+lidar / robotics | **Path B** |109| Already have parsed numpy/torch arrays in memory | **Path B** but skip Bazel — use `ncore.data.v4` API directly |110111If a candidate path exists in upstream, **prefer it**. Hand-rolling a Waymo or112PAI converter on top of the template is wasted work and almost certainly wrong113(rolling-shutter timing, FTheta intrinsics, Waymo camera-frame rotation, etc).114115---116117## Prerequisites118119- Linux host with Python ≥ 3.10 and `pip`.120- `git` for the upstream converter sources.121- `bazel` only if you run the in-tree converters (Path A); pure-Python122 in-process writes (`ncore.data.v4`) need no Bazel.123- Disk: budget tens of GB per converted clip; pre-zarr scratch can be124 larger than the final `.zarr.itar`.125- HuggingFace token (`HF_TOKEN`) only if pulling a gated PAI clip.126127### Verifying secrets safely128129**Always verify prerequisites with the upstream `validate.py` or by130running the converter against a tiny test slice; never write ad-hoc131bash that interpolates `HF_TOKEN` values.** The common one-liner132133```bash134# BAD — leaks the secret to the terminal when the variable is set135echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"136```137138prints `yes<token-value>` whenever `HF_TOKEN` is set, because139`${VAR:-no}` only falls back to "no" when the variable is empty. Use140a length-only check, which never echoes the value:141142```bash143# OK — prints "set (N chars)" or "missing", never the value144test -n "$HF_TOKEN" && echo "HF_TOKEN: set (${#HF_TOKEN} chars)" || echo "HF_TOKEN: missing"145```146147Rotate any token you suspect was echoed at148<https://huggingface.co/settings/tokens>.149150## Install & references151152```bash153pip install nvidia-ncore # pure-Python API for in-process writes154git clone --depth 1 https://github.com/NVIDIA/ncore.git # for upstream converters (Bazel)155```156157- Source + upstream converters: <https://github.com/NVIDIA/ncore>158- V4 spec / conventions: <https://nvidia.github.io/ncore/data/conventions.html>159- API reference: <https://nvidia.github.io/ncore/apis/data.v4.html>160- Conversion guides: <https://nvidia.github.io/ncore/conversions/index.html>161- Sensor models (camera, LiDAR, windshield): <https://nvidia.github.io/ncore/data/sensor_models.html>162- The template scaffold (every method documented inline):163 [`ncore_template/impl/data_converter/example_converter.py`](ncore_template/impl/data_converter/example_converter.py)164- End-to-end NCore → NRE training and rendering: see the sibling165 [`nre`](../nre/SKILL.md) skill (Workflow A). NVIDIA's reference166 OSMO recipe that wires PAI → NCore → NuRec → USDZ training lives167 in the upstream NCore repo at168 <https://github.com/NVIDIA/ncore/tree/main/tools/data_converter/pai>169 and the NRE container docs at170 <https://www.nvidia.com/en-us/omniverse/nurec/>.171172---173174## Mental model — the V4 store175176Every V4 sequence is one **store** (`itar` archive or plain directory) holding177**component groups**. The required components for NuRec are:178179| Component | What it carries | API class |180|-----------|-----------------|-----------|181| `Poses` | Dynamic `T_rig_world` (per timestamp); static `T_sensor_rig` per camera/lidar/radar; static `T_world_world_global` | `PosesComponent` |182| `Intrinsics` | Per-camera model (Pinhole / Fisheye / FTheta) + per-LiDAR spinning model | `IntrinsicsComponent` |183| `CameraSensor` | Encoded image bytes + per-frame `[exposure_start, exposure_end]` µs | `CameraSensorComponent` |184| `LidarSensor` | Per-ray unit direction + per-ray µs timestamp + distance(s) + intensity + `model_element=(row,col)` | `LidarSensorComponent` |185| `RadarSensor` *(optional)* | Same shape as LiDAR minus intensity / model_element | `RadarSensorComponent` |186| `Cuboids` *(optional, recommended)* | `CuboidTrackObservation` list referencing `rig` / `world` / sensor frames | `CuboidsComponent` |187| `Masks` | Per-camera dict of `{name: PIL.Image}` (NuRec **requires** ego masks) | `MasksComponent` |188| `PointClouds` *(optional)* | Pre-computed dense or SfM points (e.g. COLMAP `sfm_points`, depth-derived) | `PointCloudsComponent` |189190Frames of reference (these are non-negotiable — wrong frames = silent NuRec failure):191192- **Rig**: `+X` forward, `+Y` left, `+Z` up. Origin at the middle of the rear axle on nominal ground for AV, or any natural body-fixed point for non-AV. All extrinsics are `T_sensor → rig`.193- **Camera sensor**: `+X` right, `+Y` down, `+Z` forward (optical axis). NCore's convention. Waymo (X-fwd) and similar must be **rotated** before storing extrinsics.194- **LiDAR model frame**: azimuth 0° = `+X`, 90° = `+Y`, `+Z` up. Independent of the raw sensor's native azimuth — you choose how `column_azimuths_rad` maps physical columns.195- **World**: sequence-local. Re-reference all `T_rig_world` to the **first** ego pose so origin is near the vehicle start (raw UTM/ECEF at 10+ km loses precision in float32). Carry the original first pose into `T_world_world_global` (float64) if you need a global anchor.196- **Image pixels**: `u` right, `v` down, origin at the top-left **corner** of the top-left pixel (so pixel centers are at `0.5, 0.5`).197- **Units**: timestamps in µs everywhere, distances in metres, angles in radians.198199The single source of truth is the spec — when in doubt, open it:200<https://nvidia.github.io/ncore/data/conventions.html>.201202### Sequence-level metadata203204Some downstream pipelines need metadata beyond the per-component data — carry205it on the sequence's `generic_meta_data` (passed to206`SequenceComponentGroupsWriter(...)`):207208- **Stereo pairs** — required by stereo-depth modules (Foundation Stereo) and209 multi-camera training configs to discover left/right pairings:210211 ```json212 {"stereo_pairs": [{"left": "camera_front_left", "right": "camera_front_right"}]}213 ```214215 Multiple pairs are allowed for surround-stereo rigs.216217- **Source tag** — distinguishes real from synthetic data. Renderer-output218 shards (NuRec sim) set `{"source": "simulation", "model_checkpoint": "..."}`;219 real-sensor shards omit the key or set `{"source": "real"}`. Downstream220 validators key off this to skip "expected vs measured" checks on sim data.221222- **Calibration / egomotion provenance** — the example template writes223 `calibration_type` and `egomotion_type` on the `PosesComponent`'s224 `generic_meta_data`. Use this to track the upstream tool (e.g.225 `egomotion_type: "cuvslam-stereo"` or `"kiss-icp"` or `"vio:orbslam3"`) so226 later modules can pick refinement strategies that match the input quality.227228---229230## Instructions231232Pick one of the two paths below.233234- **Path A** if the user's dataset format is already supported in235 `ncore/tools/data_converter/` (PAI, Waymo, COLMAP/ScanNet++) —236 bootstrap the upstream repo and drive the existing binary.237- **Path B** if the format is unsupported (PandaSet, NuScenes, KITTI,238 custom rig, robotics bag, …) — copy `ncore_template/` next to the239 dataset and fill in the four hand-written hooks. The V4 conventions240 in the **Mental model** section above are mandatory; the recipes241 further down show typical configurations per rig.242243After conversion, always run the **Validation & end-to-end NuRec**244section below before handing the store to NRE.245246## Path A — drive an existing in-tree converter247248The upstream `tools/data_converter/<format>` modules are Bazel targets. Build249once, run per dataset.250251### Bootstrap252253```bash254git clone --depth 1 https://github.com/NVIDIA/ncore.git255cd ncore256bazel build //tools/data_converter/pai:convert # or waymo, or colmap257```258259Each `convert` binary takes shared **base** flags (`--root-dir`, `--output-dir`,260`--no-cameras`, `--camera-id`, `--no-lidars`, `--lidar-id`, `--verbose`) followed261by a **subcommand** (`pai-v4`, `pai-stream-v4`, `waymo-v4`, `colmap-v4`,262`scannetpp-v4`) with format-specific flags.263264### Standard sub-flags worth knowing265266| Flag | Default | Meaning |267|------|---------|---------|268| `--store-type {itar,directory}` | `itar` | `itar` is fastest for NuRec; `directory` is debuggable |269| `--profile {default,separate-sensors,separate-all}` | varies | NuRec wants `separate-sensors` |270| `--sequence-meta` / `--no-sequence-meta` | enabled | Writes `<sequence>.json` next to the store — NuRec/`ncore_vis` need it |271| `--world-global-mode {none,identity,localized}` | varies | For NuRec releases that require the `world→world_global` edge, use `identity` (or `localized` to keep a real global anchor) |272273### When to script vs run interactively274275For a **one-off conversion**, the bare `bazel run` form in each format recipe276below is enough. For repeatable cluster runs, wrap the same two steps277(`bazel build` then `bazel run`) inside an OSMO / Slurm / Kubernetes task that278clones NCore at a pinned ref, runs the convert step, and chains the result279into the [`nre`](../nre/SKILL.md) training and aux-data containers (see280`nre`'s Workflow A). The upstream281[`NVIDIA/ncore`](https://github.com/NVIDIA/ncore) repo ships reference282converter targets that you can pin by Git commit for reproducibility.283284---285286## Path B — author a new converter from the template287288The scaffold is intentionally minimal but writes **every** required component289type with placeholder data. Treat it as a checklist: every `# FILL IN` is a290correctness gate — none can be skipped.291292### Scaffold293294```bash295# Copy the scaffold next to your dataset (run from this skill's folder)296cp -r ncore_template /path/to/ncore-myformat297```298299Then rename the package and class (`ExampleConverter` → `MyFormatConverter`) and300implement the contract:301302| Method | Contract |303|--------|----------|304| `get_sequence_ids(config) -> list[str]` | Discover sequence IDs from `config.root_dir` (or wherever your dataset lives — manifest CSV, HF clip index, ROS bag glob) |305| `from_config(config) -> Converter` | One-time setup (load calibration, open dataset index, init shared interpolators). Heavy lifting that all sequences share goes here |306| `convert_sequence(sequence_id) -> None` | Per-sequence work: open `SequenceComponentGroupsWriter`, register component writers, write data, `finalize()`, write `<sequence_id>.json` |307308Inside `convert_sequence` the canonical order is **Poses → Intrinsics → Masks →309Camera → LiDAR → Radar → Cuboids → finalize**. This order is not required by the310writer but it surfaces calibration / pose / timing bugs **before** you've spent311minutes encoding image bytes.312313The skeleton walks each step explicitly and lists every silent-correctness314trap inline — read these before filling them in:315316- Spinning-LiDAR pitfalls (`spinning_direction`, non-uniform `row_elevations_rad`,317 `column_azimuths_rad` ordering, Ouster `row_azimuth_offsets_rad`):318 [`example_converter.py:61-120`](ncore_template/impl/data_converter/example_converter.py#L61-L120)319- Pose trajectory density + float64 → float32 + re-referencing rules:320 [`example_converter.py:336-492`](ncore_template/impl/data_converter/example_converter.py#L336-L492)321- Camera intrinsics for Pinhole / Fisheye / FTheta + shutter type:322 [`example_converter.py:524-578`](ncore_template/impl/data_converter/example_converter.py#L524-L578)323- Per-ray LiDAR timestamps and the three data shapes (range image / sensor-frame324 XYZ / world-frame XYZ requiring decompensation):325 [`example_converter.py:746-829`](ncore_template/impl/data_converter/example_converter.py#L746-L829)326- Cuboid centroid convention (geometric center, not bottom-center):327 [`example_converter.py:876-980`](ncore_template/impl/data_converter/example_converter.py#L876-L980)328329### In-process API (no Bazel)330331If you already have parsed arrays in Python and don't need a CLI, skip332`FileBasedDataConverter` entirely and call the V4 writer directly:333334```python335from ncore.data.v4 import SequenceComponentGroupsWriter, PosesComponent, ...336writer = SequenceComponentGroupsWriter(337 output_dir_path=out / seq_id,338 store_base_name=seq_id,339 sequence_id=seq_id,340 sequence_timestamp_interval_us=interval,341 store_type="itar",342)343poses_writer = writer.register_component_writer(PosesComponent.Writer, ...)344# … write each component …345paths = writer.finalize()346```347348The contract (component order, dtype rules, timestamp constraints) is identical.349350---351352## V4 conventions you must obey353354These are the rules that turn into runtime asserts (or worse: silent NuRec355artefacts). Cross-reference the spec before relaxing any of them.356357### Time358359- **Microseconds, uint64, everywhere.** `np.uint64`, not `np.int64`.360- The sequence interval is half-closed `[start, stop)`. Build it with361 `HalfClosedInterval.from_start_end(start, end_inclusive)` — do **not**362 pre-add 1 to `end`.363- Dynamic poses **must exactly span** the interval: `timestamps[0] == start` and364 `timestamps[-1] == stop - 1`. Sensors with timestamps slightly outside this365 range are clamped at write time.366- Per-sensor frame timestamps are `[exposure_start, exposure_end]` (cameras) or367 `[sweep_start, sweep_end]` (LiDAR/radar). They must lie within the sequence368 interval and the **end** must be unique within that sensor's writer.369- Rolling-shutter cameras: `start = trigger + half_shutter`,370 `end = readout_done - half_shutter`. Global shutter: `start == end` is OK.371- LiDAR `frame_timestamps_us[0]` is the **sweep start** (not midpoint). Per-ray372 `timestamp_us` must lie in `[sweep_start, sweep_end]`. Treating the dataset's373 frame timestamp as the midpoint and subtracting half a sweep duration374 introduces a ~50 ms shift on a 10 Hz LiDAR and produces motion-comp blur.375376### Pose graph377378- All intermediate pose math is **float64**. Cast to float32 only as the very379 last step before `store_dynamic_pose` / `store_static_pose`.380 Exception: `world → world_global` stays float64 (NuRec's381 `RigTrajectories.T_world_base` is float64).382- Re-reference dynamic poses to the first ego pose:383 `poses_f64 = inv(poses_f64[0]) @ poses_f64`. Without this, GPS/UTM/ENU at384 10+ km loses sub-cm precision once cast to float32.385- The pose trajectory must be **dense** (waypoint spacing < LiDAR sweep386 duration, e.g. <50 ms for 10 Hz). Combine every available source: per-camera387 per-frame poses, per-LiDAR per-sweep poses, IMU/GPS/odometry. Concatenate,388 `np.unique` by timestamp, sort.389- `T_camera_rig` and `T_lidar_rig` are float32 (NuRec's `transform_poses` leaks390 input dtype through an internal matmul; float64 here collides with the391 float32 trajectory and crashes "Get Lidar Point Clouds" with392 `RuntimeError: double != float`).393394### LiDAR395396- Direction vectors are **unit-norm**, in **sensor coordinates**, at each ray's397 **measurement time** (not the sweep-start time). Three input shapes:398 1. **Polar range image** with beam geometry → derive directly from399 `column_azimuths_rad[col]`, `row_elevations_rad[row]`,400 `row_azimuth_offsets_rad[row]`. Direction:401 `(cos(elev)*cos(azi), cos(elev)*sin(azi), sin(elev))`.402 2. **Sensor-frame XYZ** → `direction = xyz / |xyz|`.403 3. **World-frame or ego-compensated XYZ** → **decompensate** with404 `MotionCompensator.motion_decompensate_points`, then normalise.405- Spinning LiDAR parameter traps (silent — wrong values pass write but fail406 NuRec):407 - `spinning_direction`: nearly all automotive spinning LiDARs (Velodyne,408 Hesai, Ouster, Robosense) are **`"cw"`**. The template defaults to409 `"ccw"` to force the question. Wrong value mirrors Y → Z-flip in NuRec.410 - `row_elevations_rad`: **non-uniform on every common sensor**. Read real411 per-beam angles from the sensor calibration. **Strictly descending**412 (highest beam first); reverse if your source is ascending. NCore asserts413 `np.diff > 0` after projecting through `relative_angle`.414 - `column_azimuths_rad`: must reflect the **actual** per-column azimuth at415 frame start (not a synthetic `linspace`). NCore validates strict ordering416 via `relative_angle(azim[0], azim, spinning_direction)`. If raw417 per-column heading is available, use it; otherwise `linspace(0, ±2π, N,418 endpoint=False)` is acceptable for `"ccw"` / `"cw"` respectively.419 - `row_azimuth_offsets_rad`: zero for most sensors; **non-zero on Ouster**420 (`beam_azimuth_angles` from the HTTP API). Skipping this on Ouster421 misaligns LiDAR-to-camera projection.422- Per-ray `timestamp_us` is **required** for motion compensation; supply real423 per-column firing times (typically column-linear across the sweep).424425### Cuboids426427- `BBox3.centroid` is the **geometric center**. Many AV datasets use428 bottom-center → add `dim_z / 2` to z. Verify empirically: `mean(z) - mean(h)/2`429 near 0 = geometric center; near `mean(h)/2` above ground = bottom-center.430- `reference_frame_id` ∈ `{"rig", "world", <sensor_id>}`.431 `reference_frame_timestamp_us` and `timestamp_us` must lie within the432 sequence interval.433- **`LabelSource`**: tag the **origin pipeline**, not quality. Third-party434 dataset labels (Waymo, NuScenes, PandaSet) → `EXTERNAL`, even if the upstream435 is human GT. Use `GT_ANNOTATION` only when this converter's team owns the436 annotation.437438### Camera439440- Image bytes stored verbatim (JPEG/PNG); NCore does not re-encode.441- Use the model that matches your sensor: `OpenCVPinholeCameraModelParameters`442 (most AV cameras), `OpenCVFisheyeCameraModelParameters` (Kannala-Brandt443 fisheye, e.g. ScanNet++ / GoPro), `FThetaCameraModelParameters` (NVIDIA444 Hyperion / equidistant-radial). Resolution must match the actual image bytes445 bit-for-bit.446- Map shutter direction by **enum name**, not integer cast. The five447 `ShutterType` values are `ROLLING_TOP_TO_BOTTOM=1`, `ROLLING_LEFT_TO_RIGHT=2`,448 `ROLLING_BOTTOM_TO_TOP=3`, `ROLLING_RIGHT_TO_LEFT=4`, `GLOBAL=5`. Build a449 source-enum-to-name dict.450- Provide ego masks. NuRec's data-quality guide **requires** binary451 ego-vehicle masks per camera. Without them the hood/roof rack leaks into the452 reconstruction.453454### NuRec data-quality minimums455456Per the NuRec "Ensure Data Quality" guide, hitting these is the difference457between a clean reconstruction and visible artefacts:458459- Camera extrinsics: < 0.5° rotation, < 2 cm translation (relative to rig).460- Camera intrinsics: < 1 px reprojection error.461- Egomotion: < 0.5° / < 2.5 cm consecutive-frame error; trajectory must cover462 every sensor frame's start and end timestamp.463- Cuboids: < 1° / < 5 cm position / < 5 cm dimension; per-track stable464 `track_id`s.465- Original sensor resolution at full FPS, **no undistortion / rectification**.466467---468469## Format recipes (AV)470471### PAI (NVIDIA Physical AI Autonomous Vehicles, HuggingFace)472473**Built-in.** Streaming or local. Uses Hyperion 8 / 8.1 sensor IDs.474475```bash476# Streaming (no download — recommended for cloud / OSMO)477bazel run //tools/data_converter/pai:convert -- \478 --output-dir <OUT> \479 --camera-id camera_front_wide_120fov \480 --camera-id camera_cross_left_120fov \481 --camera-id camera_cross_right_120fov \482 --camera-id camera_front_tele_30fov \483 pai-stream-v4 \484 --clip-id <CLIP_ID> \485 --hf-token "$HF_TOKEN" \486 --store-type itar \487 --profile separate-sensors \488 --sequence-meta489```490491Notes:492493- HF dataset: `nvidia/PhysicalAI-Autonomous-Vehicles`. License must be accepted494 on HF before `HF_TOKEN` works.495- 7 cameras (FTheta + Fisheye + Pinhole depending on FOV), 1 top LiDAR496 (`lidar_top_360fov`, spinning, see datasheet for elevations).497- Camera intrinsics include `shutter_delay_us` for per-row rolling-shutter498 timestamping, plus optional `BivariateWindshieldModelParameters` for499 windshield refraction.500- Output path: `<output-dir>/pai_<clip-id>/pai_<clip-id>.ncore4.zarr.itar`.501- Discover clip IDs via the `clip_index.parquet` blob in the HF dataset502 (`nvidia/PhysicalAI-Autonomous-Vehicles`) — the503 [`physical-ai-datasets`](../physical-ai-datasets/SKILL.md) skill has the504 download recipe and toolkit pointers.505- Subset by time with `--seek-sec` / `--duration-sec`; subset sensors with506 `--no-lidars` / `--camera-id` (repeatable).507- **car2sim_6cam** sim configs target CARLA sensor names; PAI exports use real508 Hyperion IDs — when feeding NRE training, override `dataset.camera_ids` /509 `dataset.lidar_ids` on the NRE Hydra command line (see510 [`nre`](../nre/SKILL.md) Workflow A and `references/configuration.md`).511512### Waymo Open513514**Built-in.** Reads `.tfrecord` segment files.515516```bash517bazel run //tools/data_converter/waymo:convert -- \518 --root-dir <DIR_OF_TFRECORDS> \519 --output-dir <OUT> \520 waymo-v4 \521 --store-type itar \522 --profile separate-sensors \523 --world-global-mode localized # or 'identity' / 'none'524```525526Notes:527528- 5 cameras (`camera_front_50fov`, `camera_front_left_50fov`,529 `camera_front_right_50fov`, `camera_side_left_50fov`,530 `camera_side_right_50fov`) + 1 top LiDAR (`lidar_top`).531- Camera intrinsics → `OpenCVPinholeCameraModelParameters`. Waymo's local532 camera frame is **`+X` principal axis**; the converter rotates to NCore's533 `+Z` principal axis. If you re-derive extrinsics manually, apply the same534 rotation to `T_camera_rig`.535- Cuboid classes: `unknown, vehicle, pedestrian, sign, cyclist`. Map → NCore536 `class_id` strings; tag `LabelSource.EXTERNAL`.537- Multi-return LiDAR (primary + secondary). Stack into the `[R, N]` distance /538 intensity arrays.539540### PandaSet541542**No upstream converter — author with Path B.** PandaSet (Hesai +543Scale-AI-labelled) ships JSON metadata with each sequence.544545Sensor inventory:546547- 6 cameras (`front_camera`, `front_left_camera`, `front_right_camera`,548 `back_camera`, `left_camera`, `right_camera`) — pinhole, global shutter.549 Intrinsics in `meta/intrinsics/<id>.json`.550- 2 LiDARs:551 - `front_lidar`: Hesai PandarGT (mechanical, 60° HFOV, 150° VFOV552 forward-facing). Treat as a partial spinning sensor (`column_azimuths_rad`553 spans the 60° wedge).554 - `top_lidar` (key sensor for AV reconstruction): Hesai Pandar64,555 **`spinning_direction="cw"`**, 64 beams with **non-uniform** elevations556 (Pandar64 datasheet table — copy the 64 angles, sort descending),557 `row_azimuth_offsets_rad = zeros(64)`, 1800 columns.558559Data layout per sequence (extract):560561```text562<seq_id>/563├── meta/564│ ├── intrinsics/<camera_id>.json # focal, principal point, distortion565│ ├── timestamps.json # per-camera-frame µs566│ └── poses.json # per-frame ego pose (UTM)567├── camera/<camera_id>/<frame>.jpg568├── camera/<camera_id>/poses.json # per-frame camera pose in world569├── lidar/<frame>.pkl.gz # XYZ + intensity + timestamp + ring570└── cuboids/<frame>.pkl.gz # bottom-center xyz + dim + yaw571```572573Conversion checklist:574575- Pose trajectory: union of **per-camera per-frame** poses (6 × ~80 = ~480576 waypoints) and per-LiDAR-frame poses, deduped + sorted, re-referenced to577 the first ego pose.578- LiDAR points are **world-frame XYZ already ego-compensated** → run579 `MotionCompensator.motion_decompensate_points` to recover sensor-frame580 per-ray XYZ, then normalise to direction. Per-ray µs from the `timestamp`581 column.582- Map raw Pandar64 `ring` → NCore `model_element[:, 0]` via a583 `ring_id → row_index` permutation that sorts elevations **descending** (raw584 ring IDs are firing order, not beam index).585- Cuboids: **bottom-center → geometric center** (`centroid_z += dim_z/2`).586 Yaw-only rotation → `rot=(0, 0, yaw_rad)`. `LabelSource.EXTERNAL`.587- Cameras: global shutter (`ShutterType.GLOBAL`). `frame_timestamps_us = [t, t]`.588589### NuScenes590591**No upstream converter — author with Path B.** NuScenes ships nested JSON592tables (`sample`, `sample_data`, `ego_pose`, `calibrated_sensor`, `sensor`,593`sample_annotation`, `instance`).594595Sensor inventory:596597- 6 cameras (`CAM_FRONT`, `CAM_FRONT_LEFT`, `CAM_FRONT_RIGHT`, `CAM_BACK`,598 `CAM_BACK_LEFT`, `CAM_BACK_RIGHT`) — pinhole. Rolling shutter; consult the599 device datasheet for shutter direction (rename to lowercase NCore IDs).600- 1 LiDAR (`LIDAR_TOP`): Velodyne HDL-32E, **`spinning_direction="cw"`**,601 32 beams, 1800 columns at 0.2°. Use VLP/HDL-32 datasheet elevations602 (descending). `row_azimuth_offsets_rad = zeros(32)`.603- 5 radars (`RADAR_FRONT`, `RADAR_FRONT_LEFT`, `RADAR_FRONT_RIGHT`,604 `RADAR_BACK_LEFT`, `RADAR_BACK_RIGHT`). Continental ARS 408. Optional —605 NuRec ignores radar.606607Conversion checklist:608609- A NuScenes "scene" → one V4 sequence. Iterate `sample`s in the scene to610 enumerate `sample_data` per sensor.611- Timestamps in `sample_data.timestamp` are **µs** already.612- `ego_pose` is rig→world (translation + quaternion). Build the dense613 trajectory from the union of ego_pose entries across all sample_data614 (cameras at 12 Hz, LiDAR at 20 Hz, radars at 13 Hz → ~50 ms spacing).615- `calibrated_sensor.translation` + `.rotation` is `T_sensor_rig` (named616 `T_calib` in their docs — verify direction by transforming a sensor-frame617 test point and checking it lands in the expected rig position).618- LiDAR `.pcd.bin` files are already-ego-compensated sensor-frame XYZ + ring619 index + intensity. Per-ray timestamps are not stored — synthesise from620 `column_index` (azimuth bin) and the sweep duration (50 ms at 20 Hz):621 `t_ray = sweep_start + (azim_bin / 1800) * sweep_duration`.622- `sample_annotation`: bottom-of-box origin → **add `size[2]/2` to z**.623 `rotation` quaternion → XYZ-Euler radians. `instance_token` → `track_id`,624 `category_name` → `class_id`. `LabelSource.EXTERNAL`.625- Ego masks: NuScenes does not ship them. Generate via SAM2 / off-the-shelf626 ego segmentation per camera — strongly recommended for NuRec quality.627628---629630## Format recipes (non-AV / sensor-only)631632These do not have a vehicle rig; pick a body-fixed origin and apply the same633rig conventions (`+X` forward, `+Y` left, `+Z` up).634635### Mono camera (no depth, no LiDAR) → COLMAP track636637You only have RGB images. Run COLMAP first to produce poses + sparse points,638then feed COLMAP into the upstream converter:639640```bash641colmap automatic_reconstructor \642 --workspace_path <SCENE> --image_path <SCENE>/images \643 --camera_model OPENCV --single_camera 1644bazel run //tools/data_converter/colmap:convert -- \645 --root-dir <SCENE> --output-dir <OUT> \646 colmap-v4 --include-3d-points --start-time-sec 0647```648649Notes:650651- COLMAP timestamps are synthetic (1 FPS by default; tune with652 `--start-time-sec` and the FPS embedded in your image filenames).653- The COLMAP camera frame already matches NCore (`+Z` optical) — no rotation654 required.655- SfM points become a `PointCloudsComponent` named `sfm_points`. Use this when656 no LiDAR is available so NuRec has a sparse geometric prior.657- Ego masks (`<image_basename>_mask.png` next to images, or `--masks-dir`).658659### Stereo cameras660661Two synchronised cameras at known baseline. No LiDAR.662663- **Rig origin**: midpoint between the two camera optical centres (or the left664 camera — pick one and stay consistent).665- **Static extrinsics**: `T_left_rig`, `T_right_rig` from your stereo calibration666 (OpenCV `stereoCalibrate` outputs `R, T` from right-to-left → invert/compose667 to get rig-relative).668- **Trajectory**: stereo-VIO (ORB-SLAM3, OpenVSLAM) or COLMAP run on left669 images → propagate to rig with `T_left_rig`.670- **No LiDAR component.** Two options:671 1. Skip LiDAR entirely (set `lidar_ids = []`). NuRec falls back to672 image-only reconstruction (lower quality, more views needed).673 2. Compute disparity per stereo pair → 3-D point cloud per frame in the674 left-camera frame → write as a `PointCloudsComponent` per frame675 (analogous to COLMAP `sfm_points` but dense). This gives NuRec a676 geometric prior without faking a spinning LiDAR.677- Ego masks: render or hand-paint a static mask of the rig body if visible in678 the FOV.679680### Multi-stereo rig (surround stereo)681682Multiple synchronised stereo pairs on one rig (e.g. NVIDIA Hyperion 8.1683surround stereo, AV1, custom inspection robots). Each pair feeds Foundation684Stereo independently to produce dense depth around the platform.685686- Encode **each camera as its own** `CameraSensorComponent` with a standard687 `T_camera_rig` extrinsic — do **not** pre-rectify or fuse a pair into a688 single virtual sensor. NCore does not have a "stereo pair" component689 type; pairing is metadata, not structure.690- Declare the pairings on the **sequence-level** `generic_meta_data` so691 downstream tools (Foundation Stereo, NuRec aux-data) can discover them:692693 ```json694 {"stereo_pairs": [695 {"left": "camera_front_left", "right": "camera_front_right"},696 {"left": "camera_rear_left", "right": "camera_rear_right"},697 {"left": "camera_side_left_a", "right": "camera_side_left_b"}698 ]}699 ```700701- Pose trajectory must include every **(camera × frame timestamp)** sample —702 surround stereo at 8 cameras × 100 frames yields 800 trajectory waypoints,703 more than dense enough for per-ray motion compensation. Do not subsample.704- Calibration: stereo intrinsics + extrinsics from `cv2.stereoCalibrate` give705 you `R, T` from right-to-left. Compose with your chosen rig origin to get706 `T_left_rig` and `T_right_rig` separately; do **not** store only the707 baseline.708- Scale is metric by construction (calibrated baseline). Skip scale refinement709 (or run it as a sanity check only).710711### Mono + depth (RGB-D / learned depth)712713Single RGB camera + per-frame depth (sensor: RealSense, Kinect, ZED depth, or714learned mono-depth like Marigold / DepthAnythingV2 / MoGe-2).715716**RGB-D specifics:**717718- **RealSense D4xx / L515**: active IR stereo (D4xx) or LiDAR-class TOF (L515).719 Depth and RGB are co-triggered but **not** pixel-aligned out of the box —720 use the `rs2_align` filter (or pre-aligned topics) before treating depth721 as RGB-frame metric. Intrinsics: read the colour stream's intrinsics for722 the RGB component; ignore the depth-stream intrinsics (depth is aligned723 into the colour frame).724- **Microsoft Kinect Azure / Kinect v2**: TOF depth with non-trivial725 invalidation near object edges. Mask invalidated pixels (depth == 0) before726 encoding as a point cloud.727- **Stereolabs ZED 2 / X**: stereo with a built-in disparity engine. Either728 store both raw left/right images and treat as a stereo rig (preferred —729 Foundation Stereo can re-derive depth at higher quality), or store the730 ZED-native depth as a per-frame `PointCloudsComponent`.731- **Learned mono-depth** (Marigold, DepthAnythingV2, MoGe-2) is **scale732 ambiguous**. Either anchor with one absolute reference (a known object733 size, ground-plane height, IMU + visual-inertial scale) before storing,734 or accept that scale refinement (r2s module 6) will run downstream.735736- **Rig origin**: camera optical centre (or device body if you have a static737 IMU offset).738- **Trajectory**: ARKit/ARCore pose stream, IMU+camera VIO, or depth-aided739 RGB-D SLAM (Open3D, Spectacular AI).740- **No LiDAR** — depth is *not* a LiDAR. Two valid encodings:741 1. **Per-frame `PointCloudsComponent`** (preferred for learned/stereo-quality742 depth, where reliability is uneven). Convert depth + intrinsics →743 camera-frame XYZ, transform to world via `T_camera_world(t)`, store as744 a point cloud per frame. Carry the dense depth into `generic_data` if745 downstream consumers want it.746 2. **Synthetic spinning LiDAR** (only if your depth is dense and reliable).747 Sample a fixed grid of azimuths/elevations, ray-cast against the depth748 map at each frame, and write a `LidarSensorComponent`. This is more749 work and less honest than option 1 — prefer point clouds unless NuRec750 specifically needs a LiDAR component.751- Camera intrinsics: pinhole or fisheye depending on the lens. Depth-camera752 manufacturers ship calibration JSON — copy `fx, fy, cx, cy` and distortion753 coefficients verbatim.754- Frame timestamps: depth and RGB are usually co-triggered. Use the RGB755 exposure timestamp; depth has no separate component.756757### Mono + LiDAR (handheld / robot)758759Single camera + spinning or solid-state LiDAR, e.g. handheld scanner, ground760robot, drone.761762- **Rig origin**: choice driven by mechanical mounting. If the LiDAR is the763 reference for ego-motion (LiDAR-inertial SLAM), set rig = LiDAR (so764 `T_lidar_rig = I`). Otherwise pick the IMU body frame or camera centre.765- **Trajectory**: LIO-SAM, FAST-LIO, or any LiDAR-inertial pipeline.766 Re-reference to the first frame; cast last (float64 → float32).767- **Static extrinsics**: `T_camera_rig` from camera-LiDAR calibration768 (kalibr, lidar_align). Apply the NCore camera-frame rotation if your769 calibration target uses a different convention.770- **LiDAR**:771 - Spinning (Velodyne / Ouster / Hesai / Robosense / Livox Mid-360 in772 repetitive mode): use `RowOffsetStructuredSpinningLidarModelParameters`773 with the sensor's real elevations (datasheet) and `"cw"`. For Ouster774 populate `row_azimuth_offsets_rad` from `beam_azimuth_angles`.775 - Solid-state non-repetitive (Livox Avia, Mid-40, Mid-70 in776 non-repetitive mode): the spinning model does not fit. Either replay777 each scan as a fake "spinning" sweep (stash directions in778 `column_azimuths_rad` per-scan — fragile) or, **preferred**, write the779 points as a `PointCloudsComponent` per frame in the sensor frame and780 skip `LidarSensorComponent`. NuRec can consume point clouds.781- Cuboids: usually unavailable for non-AV — leave the component empty or782 unregistered.783- Ego masks: render the rig body if it intrudes on the FOV (drone arm,784 robot chassis); empty `{}` if not.785786### Solid-state / non-repetitive LiDAR (Livox)787788Livox Avia / Mid-40 / Mid-70 — and Mid-360 in non-repetitive mode — produce789a point cloud per scan that does **not** lay out on a row-major spinning790grid. The `RowOffsetStructuredSpinningLidarModel` does not fit; forcing it791(synthetic columns, fake row bins) breaks NCore validation and792motion-compensation alignment.793794Encoding rules:795796- **Preferred** — write each scan as a per-frame `PointCloudsComponent`797 instance in the **sensor frame**, with per-point µs timestamps. Skip798 `LidarSensorComponent` entirely. NuRec consumes point clouds and r2s depth799 refinement (module 7) treats them as it would LiDAR sweeps.800- **Fallback for repetitive Mid-360** — the Mid-360 in repetitive mode does801 produce a structured grid; treat it as a spinning LiDAR with the802 datasheet's beam table and `spinning_direction="cw"`. This is the only803 Livox variant the spinning model fits.804805Per-point timestamps: Livox custom messages carry `offset_time` (ns from806sweep start). Convert to absolute µs:807`timestamp_us = sweep_start_us + offset_time_ns // 1000`.808809### IMU + camera (visual-inertial)810811IMU is **not** stored as its own NCore component — it densifies the pose812trajectory (and, optionally, anchors metric scale on monocular setups). Two813paths:8148151. **VIO trajectory (preferred)**: run a visual-inertial pipeline (cuVSLAM816 stereo-inertial, ORB-SLAM3, OpenVSLAM, OpenVINS, Spectacular AI) →817 IMU-rate (100–200 Hz) `T_world_rig` poses → store as the dynamic pose.818 The trajectory is already dense enough that per-ray motion compensation819 works without further densification.8202. **IMU integration only (bootstrap / fallback)**: pre-integrate IMU821 (linear accel + angular vel) over short windows, anchor each window with822 the next available camera- or LiDAR-rate pose. Useful for filling gaps or823 extrapolating to per-LiDAR sweep timestamps when the SLAM stack only824 emits poses at frame rate.825826Either way, IMU sample timestamps go into the pose trajectory, **not** into827a separate component. If a downstream consumer wants raw IMU, stash the828samples in the sequence-level `generic_meta_data["imu_samples"]` (compact)829or in a sidecar file referenced from there.830831Calibration: IMU-to-camera and IMU-to-LiDAR extrinsics live in the static832pose graph as `T_imu_rig = I` if you take the IMU body frame as the rig833(common for legged robots and drones), with `T_camera_rig` /834`T_lidar_rig` from kalibr / `lidar_align` outputs. Pick **rig = IMU** when835the IMU is the trajectory reference; otherwise pick the mechanical body836frame and store `T_imu_rig` for downstream tools that want IMU-frame data.837838### ROS2 bag (MCAP / SQLite3)839840Most robotics datasets ship as ROS2 bags. The standard path is the841**rosbags** Python library — no `rclpy` and no native ROS install required:842843```bash844pip install rosbags pyav # pyav decodes H.264 video chunks; rosbags handles .mcap and .db3845```846847Convert in **two passes**: first enumerate sensors and collect calibration /848the static TF tree, then stream frames into the V4 writer. Common topic →849component mapping:850851| ROS2 message type | NCore mapping |852|-------------------|---------------|853| `sensor_msgs/Image` (raw) | `CameraSensorComponent.store_frame` — r854855…(truncated)