Paper Rob Any3d Vla

Any3D-VLA -- Enhancing VLA Robustness via Diverse Point Clouds

Problem & Contribution

VLA models using only 2D images have limited spatial understanding. Any3D-VLA incorporates 3D point clouds to enhance robustness by:

1. Unifying diverse point cloud sources -- simulator-generated, real sensor, and model-estimated (monocular depth) point clouds in one training pipeline
2. Domain-agnostic 3D feature learning -- gated residual fusion of Concerto (3D encoder) features with DINOv2+SigLIP (2D) features to mitigate depth-scale and cross-environment domain gaps
3. CoT + flow matching -- chain-of-thought autoregressive generation (bbox/goal tokens) followed by flow-matching-based action generation

Architecture Overview

                        Language Instruction
                              |
                              v
Input Images (front+wrist) --> DINOv2 + SigLIP ViT --> 2D features (L patches, dim=2304)
                                                              |
Depth Maps --> Monocular Depth Estimator --> Point Cloud      |
   (or simulator/sensor point cloud)        |                 |
                                            v                 |
                                    Concerto (3D encoder)     |
                                    (PTv3-based, dim=1728)    |
                                            |                 |
                                            v                 v
                                    Gated Residual Fusion ---------> Fused Features (dim=2304)
                                                                         |
                                                                         v
                                                               FusedMLPProjector --> LLM dim
                                                                         |
                                                                         v
                                                               InternLM2 1.8B (LLM backbone)
                                                                    |          |
                                                                    v          v
                                                          CoT tokens     Action Expert (smaller LLM)
                                                         (bbox, goal)        |
                                                                             v
                                                                    Flow Matching Module
                                                                             |
                                                                             v
                                                                    Action (7-DoF x chunk_len)

Key Components

1. 2D Backbone: DINOv2 + SigLIP (ConcertoDinoSigLIPViTMonoBackbone)

2. 3D Backbone: Concerto (PCEncoder32)

3. Gated Residual Fusion

# Learnable gate initialized at sigmoid(-2.1972) ~ 0.1
fuse_gate = nn.Parameter(torch.full((1,), -2.1972))
# Fusion: image_feat + sigmoid(gate) * LayerNorm(MLP(cat(pc_feat, image_feat)))
fused = image_feats + sigmoid(fuse_gate) * fuse_ln(fuse_mlp(cat(pc_feats, image_feats)))

When no point cloud is available, empty_pc_token (learned parameter) is used as fallback.

4. Projector (FusedMLPProjector)

5. LLM Backbone (LLMBackbone)

6. Action Expert

A smaller LLM (same architecture, scaled down hidden/intermediate sizes) that processes action tokens with cross-attention (via KV cache) to the main LLM's prefix output.

7. Flow Matching Module (VLAFlowMatchingModule)

8. Prediction Modes

9. Point Cloud Preprocessing (in DataPreprocessor)

Paper-Code Mapping

Key Hyperparameters

Dependencies

Installation

conda create -n any3dvla_env python=3.12 -y
conda activate any3dvla_env

pip install -r requirements.txt --index-url https://download.pytorch.org/whl/cu118

# Install core package
pip install -e src/vla_network

# Install Concerto (3D encoder)
git clone https://github.com/Pointcept/Concerto.git
cd Concerto && pip install -e . && cd ..

Inference

Server mode (ZMQ)

# Download checkpoint from HuggingFace: XianzheFan/Any3D-VLA
# Expected path: storage/ckpt/exp/grit-Concerto-mono-dinosiglip-16-128-40000/checkpoint-340000/model.safetensors

bash serve_mono.sh          # default port 6666
bash serve_mono.sh --compile  # ~50% faster inference, ~3min warmup

Python API

from vla_network.model.vla import VLAAgent
import numpy as np

# Load model
agent = VLAAgent(path="path/to/model.safetensors", compile=False)
agent.preprocessor.config.robot_rep = "identity"

# Prepare input
sample = {
    'text': 'pick up elephant',
    'image_array': [np.zeros((256, 256, 3), dtype=np.uint8)],      # front camera
    'image_wrist_array': [np.zeros((256, 256, 3), dtype=np.uint8)], # wrist camera
    'depth_array': [np.zeros((256, 256, 1), dtype=np.float32)],
    'depth_wrist_array': [np.zeros((256, 256, 1), dtype=np.float32)],
    'proprio_array': [np.zeros((7,), dtype=np.float32)] * 4,       # 4 history steps
    'traj_metadata': None,
    'env_id': 1,
}

# Get action
results = agent([sample])
# results[0]['action']: np.ndarray of shape (action_len * dt_steps, 7)
#   columns: [dx, dy, dz, droll, dpitch, dyaw, gripper]
# results[0].get('goal'): (xyz, rpy) tuple if CoT mode
# results[0].get('bbox'): bounding boxes if CoT mode

Supported instructions

• pick up {object}
• pick up {color} {object}
• stack {color} bowl onto {color} bowl
• stack {color} cube onto {color} cube
• move {object} to {container}
• move {object} to {color} {container}

Gotchas & Tips

• flex_attention required for action expert -- the code enforces this; if your LLM doesn't support it, you must use a non-action-expert config
• Concerto runs in float32 -- wrapped in PCEncoder32 to prevent dtype casting issues with mixed precision training
• Point cloud preprocessing is heavy -- the DataPreprocessor runs monocular depth estimation (DepthAnything3) at inference time if real depth is not available
• serve_mono.sh sets HF_ENDPOINT to hf-mirror.com -- change this for non-China networks
• gx_utils -- the code depends on a private gx_utils package for logging, file management, robot configs, and data types; this is bundled with the checkpoint/config but not in the public repo
• Action interpolation -- at inference, predicted delta actions are interpolated by dt_steps using axis-angle decomposition (transforms3d)
• Gripper discretization -- predicted gripper values are discretized to {-1, 0, 1} at inference