2xplat Two Expert 3dgs

What This Skill Does

Swap a monolithic 3D Gaussian Splatting architecture with a modular two-expert framework that explicitly separates geometry estimation (camera pose prediction) from appearance synthesis (Gaussian generation). Achieves 30× faster convergence while maintaining quality comparable to state-of-the-art pose-dependent methods.

Problem with Monolithic 3D Gaussian Splatting

Monolithic architectures jointly optimize all parameters (camera poses, Gaussian positions, colors, covariances) in a single network. This creates inherent conflicts:

• Strict geometric accuracy (tight pose constraints) can degrade visual appearance (fewer degrees of freedom for colors)
• Unified representations cannot leverage specialized architectures proven effective for each sub-task
• Large parameter space (poses + gaussians) requires extensive training (150K+ iterations on 16 GH200 GPUs)
• Dense architecture cannot reuse pretrained geometric or appearance priors

The paper's insight: Decompose into specialized experts. Geometry expert predicts precise camera poses; appearance expert (already trained on multi-view synthesis) generates Gaussians conditioned on known poses. This enables reuse of pretrained models and faster convergence.

The Swap: Monolithic → Two-Expert Decomposition

Replace unified architecture with sequential expert pipeline:

# Monolithic approach (baseline)
class MonolithicGaussianSplatter(nn.Module):
    """Single network jointly predicts poses + Gaussians"""
    def __init__(self):
        self.image_encoder = Encoder()
        self.pose_head = nn.Linear(feat_dim, 6)      # Predict 6-DoF poses
        self.gaussian_head = nn.Linear(feat_dim, 14*N_gaussians)  # Predict positions, colors, covariance

    def forward(self, images):
        features = self.image_encoder(images)
        poses = self.pose_head(features)       # Joint optimization
        gaussians = self.gaussian_head(features)
        return poses, gaussians

# Two-expert decomposition (proposed)
class TwoExpertGaussianSplatter(nn.Module):
    """Separate experts for geometry and appearance"""
    def __init__(self):
        # Geometry expert: specialized for pose estimation
        self.geometry_expert = DepthAnything3()  # Pretrained depth/pose model

        # Appearance expert: specialized for multi-view Gaussian synthesis
        self.appearance_expert = MultiViewPyramidTransformer()  # Pretrained multi-view synthesizer

    def forward(self, images):
        # Stage 1: Geometry expert predicts camera poses
        depth_maps = self.geometry_expert(images)
        poses = depth_to_poses(depth_maps)  # Extract poses from depth

        # Stage 2: Appearance expert generates Gaussians conditioned on estimated poses
        gaussians = self.appearance_expert(images, poses=poses)  # Pose-conditioned synthesis

        # Stage 3: End-to-end fine-tuning (optional, joint loss)
        return poses, gaussians

# Fine-tuning: joint loss allows appearance expert to adapt to noisy poses
def joint_loss(rendered, target, poses, poses_target):
    render_loss = l2_loss(rendered, target)
    pose_supervision = mse_loss(poses, poses_target)  # Regularize pose estimates
    return render_loss + lambda * pose_supervision

Key differences:

• Monolithic: One network, shared features, conflicting gradients
• Two-expert: Reuse pretrained Depth Anything 3 for geometry, pretrained multi-view model for appearance; fine-tune jointly

Performance Impact

Baseline (Monolithic, feed-forward):

• Training iterations: 150,000 on 16 GH200 GPUs
• Convergence time: ~48 hours
• Quality: Baseline for comparison

With Two-Expert Architecture:

• Training iterations: < 5,000 on 8 H200 GPUs
• Speedup: 30× faster (150K → 5K iterations, 16×8 → 8×8 GPUs)
• Quality: Substantially outperforms prior pose-free methods; matches pose-dependent methods
• Cross-dataset generalization: Strong (DL3DV → ScanNet++) with minimal fine-tuning
• Robustness to view count: Scales from 6 to 128 input views

Ablation (implicit):

• Monolithic baseline: slow convergence due to conflicting gradients
• Geometry expert alone: accurate poses but no appearance
• Two-expert (frozen): decent performance, no adaptation
• Two-expert + fine-tuning: best results (poses + appearance jointly optimized)

When to Use

• Feed-forward 3D Gaussian Splatting when pose is unknown (casual capture, internet images)
• Multi-view reconstruction on diverse datasets (10-100 views)
• When you have access to pretrained geometry (Depth Anything) and appearance experts (multi-view models)
• Large-scale 3D reconstruction where convergence speed matters (avoid training-from-scratch)
• Cross-dataset scenarios (pretrained experts generalize better than monolithic models)

When NOT to Use

• Single-image 3D reconstruction (two-expert assumes multiple views)
• When you have ground truth poses (pose-dependent methods will be simpler)
• Highly constrained domains where monolithic end-to-end training is standard (e.g., synthetic data)
• If pretrained experts (Depth Anything, multi-view synthesizer) are unavailable or domain-specific
• Real-time applications needing absolute minimal latency (two stages add minor overhead)

Implementation Checklist

To integrate two-expert architecture:

1. Prepare geometry expert:

   • Use Depth Anything 3 or comparable depth estimator
   • Input: multi-view images
   • Output: depth maps → camera poses (via SfM-like recovery or direct pose regression)

2. Prepare appearance expert:

   • Use Multi-View Pyramid Transformer or pretrained multi-view synthesis model
   • Input: images + camera intrinsics/extrinsics
   • Output: 3D Gaussian parameters (position, color, covariance)

3. Implement sequential pipeline:

   Multi-view images → Geometry expert → Pose estimates
                                            ↓
                                    Appearance expert + pose-conditioning
                                            ↓
                                      3D Gaussians
   

4. Add joint fine-tuning (optional but recommended):

   • Forward: render Gaussians, compare to input images
   • Loss: render loss + pose supervision (small weight ~0.01)
   • Fine-tune both experts jointly for 5K iterations

5. Verify convergence and generalization:

   • Baseline: train monolithic from scratch, measure time to convergence
   • New: train two-expert, target 30× faster
   • Cross-dataset: evaluate on different domain (e.g., test ScanNet++ after training on DL3DV)
   • Benchmark: PSNR/SSIM on standard multi-view datasets

6. Optional tuning:

   • Pose supervision weight: 0.001-0.1 (balance rendering vs pose accuracy)
   • Freezing strategies: which layers to fine-tune (try all, appearance-only, geometry-only)
   • View count: test on 6, 12, 24, 48, 128 views for scaling study

Known Issues

• Noisy pose estimates degrade appearance: If geometry expert produces poor poses, fine-tuning may not fully recover. Monitor pose estimation accuracy early.
• Incompatibility of pretrained models: If geometry expert and appearance expert were trained differently (e.g., different camera models), fine-tuning reconciles but takes longer.
• Mode collapse on few views: With < 5 views, geometry expert may be ambiguous. Add pose uncertainty or regularization.
• Generalization to narrow baselines: If training data has wide baselines but test data is narrow (or vice versa), performance drops. Ensure training set matches test distribution.

Related Work

Builds on 3D Gaussian Splatting (Kerbl et al.), Depth Anything (Yang et al.), and multi-view synthesis (Transformer-based approaches). Relates to prior modular 3D reconstruction methods (NeRF with pose networks) but applies to Gaussian representations. Distinct from monolithic end-to-end 3D learning by leveraging pretrained, specialized experts.