Nvidia Isaac Ros Knowledge Patch

NVIDIA Isaac ROS

Use this skill when selecting an Isaac ROS platform, upgrading packages, building accelerated perception or manipulation graphs, integrating NITROS, or diagnosing a quickstart that behaves differently across JetPack, desktop, DGX Spark, and Isaac Sim.

Start with the project's pinned Isaac ROS package set and hardware target. Package availability, tested simulators, camera support, and workarounds are release-dependent. Prefer the installed package index, launch files, and observed runtime behavior when they differ from older examples.

Reference index

Breaking changes and compatibility boundaries

Treat NITROS as GXF-free

NITROS sunset its GXF implementation in 4.5. Do not carry assumptions about the old GXF build or runtime foundation into new NITROS integrations. CUDA streaming is available for NITROS messaging; see the NITROS reference for the related point-cloud and compression changes.

Update coupled hand ordering

The cuMotion controller and Isaac ROS Teleop revised the hand order in their PoseArray interfaces together. When replaying data or connecting custom publishers, update both ends of the interface instead of compensating on only one side.

Resolve package names before reusing launch files

Mapping and localization package membership is release-dependent, and 4.4 included package renames. Check old launch files against the installed package index before debugging missing executables or packages.

Keep simulator and platform support scoped to the package set

JetPack, Jetson model, DGX Spark, Isaac Sim, camera SDK, and architecture support changed across releases. Do not infer compatibility from a nearby release. Use the platform reference to select the exact combination and note historical exclusions that later updates superseded.

Preserve fork assumptions in control integrations

The bundled topic_based_ros2_control and Universal Robots dependencies in the 4.0 manipulation stack came from forks rather than their upstream package lines. Account for those forks when comparing APIs, applying upstream fixes, or replacing dependencies.

Environment quick reference

The runtime matrix snapshot in the platform reference distinguishes three tested families:

• Jetson Thor on its specified JetPack release and NVMe capacity.
• x86_64 with an Ampere-or-newer GPU, Ubuntu, CUDA, driver, GPU-memory, and storage minimums.
• DGX Spark on its specified DGX OS and storage minimum.

Virtual Environment and Bare Metal modes remove the Docker requirement, but they do not remove the dependency or hardware matrix. Use the dedicated JetPack RealSense setup where required instead of treating a Docker-free mode as a camera-stack workaround.

High-impact troubleshooting

Confirm the failure layer first

Classify the problem before changing launch parameters:

1. Verify the hardware, operating system, JetPack, driver, CUDA, simulator, and storage combination.
2. Check whether the package or backend is supported on that platform.
3. Confirm image encodings, CameraInfo, frames, and timestamps at graph boundaries.
4. Check model conversion, TensorRT engine generation, and Triton repository contents.
5. Observe GPU memory, component-container exits, action-server occupancy, and controller acknowledgement state.

Model conversion and inference

• MobileSAM conversion under PyTorch 2.6 may require TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1.
• SAM2 Virtual Environment dependency trouble has a version-specific NumPy workaround in older packages and is fixed in 4.5.
• FoundationStereo FP16 conversion, DOPE conversion on Thor, PeopleNet engine generation, and Triton backend or repository failures have distinct causes. Do not substitute one workaround for another.

Missing image, disparity, or point-cloud output

Trace synchronization and metadata before blaming the accelerator. Known failure modes include RealSense SDK instability, a decoder that does not synchronize CameraInfo with disparity tensors, PointCloud2 or frame metadata problems in simulation, a mono8/rgb8 mismatch, and a Jetson VPI backend exit.

Long-running and intermittent failures

Watch for live-camera SAM2 GPU growth, intermittent H.264 decoder starvation, Unitree hand motor-temperature limits, and manipulation goals that retain the action server after cancellation. These failures need resource or lifecycle diagnosis rather than repeated relaunches alone.

Perception and mapping routing

Use the perception reference for:

• FoundationStereo, Fast-FoundationStereo, ESS, and the stereo-decoder package.
• GroundingDINO, RT-DETR, YOLOv8, DetectNet, DOPE, CenterPose, and FoundationPose.
• Segment Anything 2 and MobileSAM conversion or live-camera constraints.
• Nvblox lidar dynamics and motion compensation.
• RGB-D Visual SLAM, cuVSLAM build behavior, visual mapping, global localization, occupancy-grid localization, and point-cloud utilities.
• RealSense, ZED, Isaac Sim, SIPL, and Leopard Imaging stereo workflows.

Fast-FoundationStereo is research-only; use FoundationStereo for commercial work. Stereo resizing for RealSense, ZED, and Isaac Sim no longer preserves aspect ratio in the moved decoder workflows.

NITROS, data, and cloud routing

Use the NITROS reference when changing transport or dataflow. It covers CUDA point-cloud support, CUDA streaming, native V4L2 H.264, dynamic image sizes, QoS revisions, and the MCAP-to-LeRobot converter's session, resampling, and effort-export behavior.

Cloud-control integrations use separate scene-recording, VDA5050 client, and action-handler packages. Unitree G1 mission support is split between robot hardware and simulation; consult the manipulation reference before assuming the same mission types are available in both.

Manipulation and teleoperation routing

Use the manipulation reference to align:

• The Isaac for Manipulation name with the isaac_manipulator package and reference workflows.
• cuMotion 1.1 planning behavior, ESDF consistency, AABB clearing, and hand ordering.
• CloudXR operation without Docker, Quest 3 input, raw controller data, configurable XR transforms, and RViz visualization.
• Flexiv Rizon, Universal Robots, Bring Your Own Robot, static-scene, and cloud pick-and-place integrations.
• Unitree G1 recording, GR00T deployment, teleoperation, bridge defaults, firmware acknowledgements, cloud missions, and AGILE policy deployment.

Upgrade checklist

Before accepting an upgrade:

• Re-resolve platform and simulator support for the target package set.
• Compare camera SDK and backend boundaries for the exact Jetson platform.
• Replace old NITROS/GXF assumptions and test CUDA stream propagation.
• Audit renamed packages and moved stereo workflows in launch files.
• Validate PoseArray hand ordering across teleop and cuMotion consumers.
• Rebuild or relocate model artifacts using the workflow-specific guidance.
• Exercise cancellation, shutdown, and recovery paths for manipulation goals.
• Run perception with real timestamps, encodings, CameraInfo, and frames.
• Monitor GPU memory and thermals during sustained live-camera or robot tests.

Working rule

Treat the references as compatibility constraints, not as a substitute for the package set in the workspace. Apply advice only when its release, hardware, and workflow scope match the project, then verify the resulting graph with its real messages and lifecycle behavior.