Lattice 3d Generation

Summary

LATTICE introduces VoxSet, a semi-structured latent representation that combines the efficiency of VecSet methods with spatial structure guidance from coarse voxel grids. The approach uses voxel-anchored queries with rotary positional embeddings in diffusion transformers, enabling high-fidelity 3D generation with strong test-time scaling and pure transformer architecture.

Core Technique

VoxSet Representation: Hybrid latent space balancing efficiency and structure:

• Query Positions: Voxel grid centers (coarse structure)
• Learned Features: Per-voxel latent codes (detail)
• Positional Encoding: Rotary embeddings anchoring queries to spatial positions

Two-Stage Pipeline:

1. Coarse Generation: Voxelize existing 3D model to initialize voxel structure
2. Fine Generation: Produce detailed VoxSet latents within sparse voxel structure

Test-Time Scaling: Increase token count during inference from 6K to 30K, leveraging learned spatial structure to maintain coherence.

Implementation

Voxel grid initialization:

def create_voxel_grid(coarse_geometry, voxel_size=0.05):
    # Convert mesh to voxels
    voxels = voxelize(coarse_geometry, voxel_size)

    # Extract voxel centers (query positions)
    voxel_centers = voxel_positions(voxels)

    return voxel_centers  # [num_voxels, 3]

VoxSet representation:

class VoxSetLatent:
    def __init__(self, num_voxels, latent_dim=64):
        # Query positions (fixed at voxel centers)
        self.positions = nn.Parameter(voxel_centers)  # [num_voxels, 3]

        # Learnable feature codes per voxel
        self.codes = nn.Parameter(torch.randn(num_voxels, latent_dim))

    def forward(self):
        # Apply rotary position embeddings
        positions_embedded = rotary_embed(self.positions)
        features = self.codes * positions_embedded
        return features

Rotary positional embeddings for 3D:

def rotary_embed_3d(positions):
    # positions: [num_positions, 3]
    dim = 64  # Feature dimension

    # Compute rotation angles per dimension pair
    freq = 1.0 / (10000 ** (torch.arange(0, dim, 2).float() / dim))

    angles = positions.unsqueeze(-1) @ freq.unsqueeze(0)
    embed = torch.cat([torch.cos(angles), torch.sin(angles)], dim=-1)

    return embed

Two-stage generation:

def generate_3d(coarse_mesh, num_steps=100):
    # Stage 1: Voxelize coarse geometry
    voxels = voxelize(coarse_mesh)
    voxset_init = VoxSetLatent(num_voxels=voxels.sum())

    # Stage 2: Diffusion to refine details
    for step in range(num_steps):
        noise = randn_like(voxset_init.codes)
        # Denoising step preserving voxel structure
        voxset_init.codes = diffusion_step(voxset_init.codes, noise, step)

    # Decode to mesh
    mesh = voxset_to_mesh(voxset_init)
    return mesh

Test-time scaling (token extension):

def generate_with_scaling(voxset, target_tokens=30000):
    initial_tokens = voxset.codes.shape[0]

    if target_tokens > initial_tokens:
        # Interpolate new voxel positions
        new_positions = interpolate_positions(voxset.positions, target_tokens)
        new_codes = interpolate_features(voxset.codes, target_tokens)

        voxset_scaled = VoxSetLatent(target_tokens)
        voxset_scaled.positions = new_positions
        voxset_scaled.codes = new_codes

    # Continue diffusion with scaled latents
    mesh = voxset_to_mesh(voxset_scaled)
    return mesh

When to Use

• High-fidelity 3D model generation from text descriptions
• Applications where test-time scaling can improve quality
• Scenarios requiring pure transformer architecture without sparse ops
• Tasks balancing generation quality and computational efficiency

When NOT to Use

• Real-time 3D generation where voxel structure is unnecessary
• Applications with simple or small 3D objects
• Scenarios requiring exact mesh topology
• Tasks where point cloud or implicit representations work better

Key References

• 3D generation and mesh synthesis
• Voxel representations and spatial structure
• Diffusion transformers for generation
• Test-time compute scaling for improved quality