Paper Rob Idp3

iDP3 — Improved 3D Diffusion Policy

Generalizable Humanoid Manipulation with 3D Diffusion Policies

Problem & Contribution

Autonomous humanoid manipulation typically requires per-scene data and precise camera calibration. iDP3 is a calibration-free, segmentation-free 3D visuomotor policy that:

1. Uses egocentric 3D point clouds from a single depth camera (RealSense L515) — no extrinsic calibration or object segmentation needed
2. Learns from noisy human teleoperation demos collected in a single scene
3. Generalizes to novel objects, viewpoints, and unseen scenes
4. Validated with 2000+ real-world policy rollouts on a 25-DoF Fourier GR1 humanoid

Method Overview

RealSense L515 ──► Raw Point Cloud (4096 pts) ──► MultiStagePointNet ──► 128-d features ─┐
                                                                                           ├──► ConditionalUnet1D ──► Actions [T, 25]
Robot Joints (32-d) ──────────────────────────► State MLP ──────────────► 64-d features ──┘
                                                                     (FiLM conditioning)
                                                                           ▲
                                                                    DDIM diffusion
                                                                    (50 train / 10 infer steps)

Key improvements over DP3 (RSS 2024):

Architecture Details

MultiStagePointNetEncoder

• 4 Conv1d stages, each with global max-pool
• Multi-scale features concatenated before final projection
• Input: [B, N, 3] → Output: [B, 128]
• Config: h_dim=128, out_channels=128, num_layers=4

ConditionalUnet1D (Diffusion Backbone)

• 1D U-Net over action trajectory dimension
• FiLM conditioning from encoder output at down/mid/up paths
• Down dims: [256, 512, 1024], kernel size 5
• Timestep embedding: sinusoidal → MLP

Diffusion Schedule

• Scheduler: DDIM (diffusers.DDIMScheduler)
• Train timesteps: 50, inference steps: 10
• Beta schedule: squaredcos_cap_v2
• Prediction type: sample (predicts clean trajectory)

Paper-Code Mapping

Key Hyperparameters

Data Format

Zarr dataset with arrays:

Normalization: actions are linearly normalized; point clouds and agent_pos use identity (no normalization).

Input/Output Shapes

# Observation dict
obs_dict = {
    'point_cloud': torch.Tensor,  # [B, n_obs_steps, 4096, 3]
    'agent_pos':   torch.Tensor,  # [B, n_obs_steps, 32]
}

# Action output
action = policy.predict_action(obs_dict)['action']  # [B, n_action_steps, 25]

Dependencies

python==3.8
torch==2.1.0 (CUDA 12.1)
diffusers==0.11.1
hydra-core==1.2.0
einops==0.4.1
open3d
zarr==2.12.0
numba==0.56.4
wandb

Hardware: Training on RTX 4090 (24GB). Deployment on CPU (onboard compute). Sensor: Intel RealSense L515 (LiDAR). Do NOT use RealSense D435 — imprecise depth.

Gotchas & Tips

• The repo uses Hydra for config. Override via CLI: python train.py task.dataset.zarr_path=/path/to/data
• Point clouds are not normalized — the encoder handles raw egocentric coordinates
• n_action_steps = horizon - n_obs_steps + 1 = 16 - 2 + 1 = 15
• Deployment uses CPU inference on the robot's onboard computer
• The deploy.py is specific to Fourier GR1 — adapt env.step() for your robot
• communication.py (robot comms) lives in the Humanoid-Teleoperation repo
• Data collection uses Apple Vision Pro for whole-upper-body teleoperation
• Community extension: PointVLA (arXiv 2503.07511) uses the iDP3 encoder in a 3D VLA