Simart Articulated Asset Decomposition

What This Skill Does

Swap dense voxel-based 3D tokenization with sparse 3D Vector Quantized Variational Autoencoder (VQ-VAE) to achieve 70% reduction in token count while preserving mesh articulation quality for multimodal 3D understanding models.

Problem with Dense Voxel Tokenization

Standard 3D tokenization (voxel grids, point clouds) generates long token sequences that consume excessive memory. Dense representations encode every spatial position uniformly, creating redundant sequences where:

• Empty/sparse regions waste tokens
• Long sequences make complex articulated objects intractable for MLLMs
• High token count forces shorter context windows or model size reductions

The paper's insight: 3D shapes are inherently sparse (most of space is empty). Sparse tokenization that only encodes occupied regions reduces sequence length dramatically while preserving articulation structure.

The Swap: Dense Voxels → Sparse 3D VQ-VAE

Replace uniform voxel grids with learned sparse quantization:

# Dense voxel tokenization (baseline)
def dense_voxel_tokenize(mesh, grid_size=64):
    """Encode mesh on uniform voxel grid"""
    voxel_grid = mesh_to_voxels(mesh, resolution=grid_size)  # 64³ = 262,144 positions
    # All positions encoded, even empty ones
    tokens = voxel_grid.flatten()  # ~260K tokens for modest resolution
    return tokens

# Sparse 3D VQ-VAE (proposed)
def sparse_vq_vae_tokenize(mesh, latent_dim=256, num_codes=1024):
    """
    Encode only occupied regions via learned sparse representation.
    Key: VQ-VAE learns to compress sparse geometry into discrete codes.
    """
    # 1. Extract sparse geometry (octree or list of occupied voxels)
    occupied_voxels = mesh_to_sparse_voxels(mesh)  # Only ~5-10% of grid

    # 2. Encode occupied regions through learned VAE encoder
    z = vae_encoder(occupied_voxels)  # Continuous latent

    # 3. Quantize to nearest codebook vector (discrete)
    z_quantized, indices = vq_layer.encode(z)  # indices reference codebook

    # 4. Output: sparse token indices (70% fewer tokens than dense)
    return indices  # ~6-8K tokens vs ~260K for dense

Critical component: VQ layer learns k-means-like codebook of geometry patterns, enabling compression beyond just spatial sparsity.

Performance Impact

Baseline (Dense Voxel Tokenization):

• Token count for articulated mesh: ~260K (64³ grid)
• Memory per example: high (limits batch size, context window)
• Sequence length: prohibitive for standard MLLMs

With Sparse 3D VQ-VAE:

• Token count: ~8K (70% reduction)
• Memory footprint: 3.3× lower
• Enables real-time physics simulation within MLLM context

Ablation (implicit from paper):

• Without sparsity (dense encoding): 260K tokens (baseline)
• With spatial sparsity only: ~26K tokens (10% of space occupied)
• With VQ-VAE compression on sparse: ~8K tokens (additional 70% reduction via learned codes)

Performance on PartNet-Mobility: Maintains articulation prediction accuracy while achieving dense tokenization's representational quality with 70% fewer tokens.

When to Use

• 3D mesh decomposition and articulation prediction (PartNet-Mobility, real-world CAD models)
• Multimodal models (vision + 3D) where token budget is tight
• Physics-based simulation of articulated objects (robotics, embodied AI)
• Large-scale 3D generation where memory is constrained (batch size, context window)
• When downstream task requires sim-ready assets (not just static meshes)

When NOT to Use

• If you need extremely high geometric fidelity (VQ-VAE introduces quantization artifacts)
• Simple geometric tasks where dense encoding is already efficient (e.g., single closed shapes)
• Models with unlimited token budget or no memory constraints
• Tasks requiring continuous 3D coordinates (discrete codes may lose sub-voxel precision)

Implementation Checklist

To integrate sparse 3D VQ-VAE:

1. Build sparse representation: Convert mesh to sparse voxel list (octree or coordinate-based) instead of dense grid.

2. Train VQ-VAE on sparse geometry:

   • Input: Sparse voxel features (occupancy + normal + color)
   • Codebook size: 1024 codes (tunable)
   • Latent dimension: 256 (balance expressiveness vs compression)
   • Loss: Reconstruction + codebook commitment loss (standard VQ-VAE)

3. Integrate with MLLM:

   • Replace voxel embedding with VQ code embedding
   • Token count now ~8K instead of 260K
   • Decoder: VQ codes → latent → sparse voxels → mesh (for sim-ready output)

4. Verify articulation prediction:

   • Benchmark: PartNet-Mobility (part segmentation + kinematic chain prediction)
   • Metrics: Part IOU, hinge axis accuracy
   • Target: Match dense baseline accuracy with 70% fewer tokens

5. Optional tuning:

   • Codebook size: 512-2048 (larger = more expressiveness, more memory)
   • Grid resolution: 32-64 (finer = higher detail, more sparse regions)
   • Occupancy threshold: 0.5 (adjust for sparse vs dense tradeoff)

Known Issues

• Quantization artifacts: Discrete codes may lose fine geometric detail. Use higher codebook size if accuracy drops > 2%.
• Codebook collapse: If codebook size too large, some codes unused. Monitor codebook utilization during training.
• Sparse handling: Different mesh sparsity across dataset can cause training instability. Normalize sparse density or pad to consistent size.
• Sim-readiness: Sparse representation may have gaps; post-processing (closing small holes) sometimes needed before physics simulation.

Related Work

Builds on VQ-VAE (van den Oord et al.) and sparse 3D representations (octrees, point clouds). Relates to neural sparse voxel grids (SNeRG, PlenVoxels) but adds quantization for discrete MLLM tokens. Distinct from dense 3D tokenizers (DiT, Point-E) by focusing on sparsity + discrete codes.