Parallel Rendering Architecture

Parallel Rendering Architecture

StickerNest uses a parallel rendering architecture where two rendering systems coexist:

• DOM Renderer: Traditional React/CSS for desktop mode (fast, accessible, proven)
• WebGL Renderer: Three.js/R3F for VR/AR modes (immersive, 3D-capable)

Why Parallel, Not Unified?

We chose parallel because:

1. Desktop rendering already works and is performant
2. VR/AR requires fundamentally different rendering (WebGL, cameras, depth)
3. Incremental migration is safer than full rewrite
4. Fallback to DOM if WebGL fails

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                    useSpatialModeStore                  │
│              activeMode: 'desktop' | 'vr' | 'ar'        │
└─────────────────────────┬───────────────────────────────┘
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
┌─────────────────────┐   ┌─────────────────────────────┐
│   DOM Renderer      │   │      WebGL Renderer         │
│   (CanvasRenderer)  │   │   (SpatialCanvas + R3F)     │
│                     │   │                             │
│  - React components │   │  - Three.js scene           │
│  - CSS transforms   │   │  - XR session               │
│  - 2D hit testing   │   │  - 3D raycasting            │
└─────────────────────┘   └─────────────────────────────┘
              │                       │
              └───────────┬───────────┘
                          ▼
              ┌───────────────────────┐
              │    Shared State       │
              │  (Zustand stores)     │
              │                       │
              │  - Widget positions   │
              │  - Entity data        │
              │  - Selection state    │
              └───────────────────────┘

Key Components

1. Mode Store (useSpatialModeStore)

Controls which renderer is active:

import { useSpatialModeStore, useActiveSpatialMode } from '../state/useSpatialModeStore';

const spatialMode = useActiveSpatialMode(); // 'desktop' | 'vr' | 'ar'

// Mode determines which renderer shows
if (spatialMode === 'desktop') {
  // Show DOM renderer
} else {
  // Show WebGL renderer
}

2. DOM Renderer (CanvasRenderer)

The existing 2D renderer at src/components/CanvasRenderer.tsx:

// Existing component - no changes needed for parallel architecture
<CanvasRenderer
  widgets={widgets}
  entities={entities}
  // ... props
/>

3. WebGL Renderer (SpatialCanvas)

New Three.js/R3F renderer at src/components/spatial/SpatialCanvas.tsx:

import { Canvas } from '@react-three/fiber';
import { XR, createXRStore } from '@react-three/xr';

const xrStore = createXRStore({ /* config */ });

function SpatialCanvas({ widgets, entities }) {
  return (
    <Canvas>
      <XR store={xrStore}>
        {/* 3D scene that mirrors widget/entity state */}
        <WidgetLayer3D widgets={widgets} />
        <EntityLayer3D entities={entities} />
      </XR>
    </Canvas>
  );
}

4. Renderer Switcher

Coordinates which renderer is visible:

function EditorCanvas() {
  const spatialMode = useActiveSpatialMode();
  const widgets = useWidgets();
  const entities = useEntities();

  return (
    <div className="editor-canvas">
      {/* DOM renderer - visible only in desktop mode */}
      <div style={{ display: spatialMode === 'desktop' ? 'block' : 'none' }}>
        <CanvasRenderer widgets={widgets} entities={entities} />
      </div>

      {/* WebGL renderer - visible only in VR/AR modes */}
      {spatialMode !== 'desktop' && (
        <SpatialCanvas widgets={widgets} entities={entities} />
      )}
    </div>
  );
}

Data Flow

Both renderers consume the same state:

┌──────────────────┐
│  User Interaction │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐     ┌──────────────────┐
│  DOM Events      │ OR  │  XR Input        │
│  (mouse, touch)  │     │  (controller)    │
└────────┬─────────┘     └────────┬─────────┘
         │                        │
         └──────────┬─────────────┘
                    ▼
         ┌──────────────────┐
         │  Intent System   │
         │  (select, grab)  │
         └────────┬─────────┘
                  │
                  ▼
         ┌──────────────────┐
         │  Zustand Store   │
         │  (single source) │
         └────────┬─────────┘
                  │
         ┌────────┴────────┐
         ▼                 ▼
┌──────────────┐   ┌──────────────┐
│ DOM Renderer │   │ WebGL Render │
│   updates    │   │   updates    │
└──────────────┘   └──────────────┘

Position Mapping

DOM uses pixels, WebGL uses meters. Convert between them:

// Constants
const PIXELS_PER_METER = 100; // 1 meter = 100 pixels

// 2D → 3D position
function toSpatialPosition(pos2D: { x: number; y: number }): [number, number, number] {
  return [
    pos2D.x / PIXELS_PER_METER,
    -pos2D.y / PIXELS_PER_METER, // Y is inverted in 3D
    0 // Z = 0 for flat widgets
  ];
}

// 3D → 2D position
function toDOMPosition(pos3D: [number, number, number]): { x: number; y: number } {
  return {
    x: pos3D[0] * PIXELS_PER_METER,
    y: -pos3D[1] * PIXELS_PER_METER
  };
}

Widget Rendering in 3D

Widgets are HTML - they need special handling in WebGL:

Option A: HTML to Texture (Simple, Limited)

import { Html } from '@react-three/drei';

function Widget3D({ widget }) {
  const position = toSpatialPosition(widget.position);

  return (
    <Html
      position={position}
      transform
      occlude
      style={{ width: widget.width, height: widget.height }}
    >
      <WidgetContent widget={widget} />
    </Html>
  );
}

Option B: CSS3DRenderer (Better Interactivity)

// Use CSS3DRenderer alongside WebGLRenderer
// Widgets stay as DOM but are positioned in 3D space

Option C: Render to Texture (Best Performance)

// Render widget HTML to canvas, use as texture
// Best for many widgets, worst for interactivity

Recommendation: Start with Option A (<Html>), upgrade as needed.

Switching Modes

When user clicks VR toggle:

// 1. Store updates
useSpatialModeStore.getState().toggleVR();

// 2. This triggers re-render
const spatialMode = useActiveSpatialMode(); // now 'vr'

// 3. Renderer switcher responds
if (spatialMode === 'desktop') {
  // Hide WebGL, show DOM
} else {
  // Hide DOM, show WebGL
  // Start XR session if VR/AR
}

XR Session Lifecycle

function SpatialCanvas() {
  const setActiveMode = useSpatialModeStore((s) => s.setActiveMode);
  const setSessionState = useSpatialModeStore((s) => s.setSessionState);

  return (
    <Canvas>
      <XR
        store={xrStore}
        => {
          setSessionState('active');
          // Detect VR vs AR from session
          const mode = xrStore.getState().session?.mode;
          setActiveMode(mode?.includes('ar') ? 'ar' : 'vr');
        }}
        => {
          setSessionState('none');
          setActiveMode('desktop'); // Return to DOM renderer
        }}
      >
        <Scene />
      </XR>
    </Canvas>
  );
}

File Structure

src/
├── components/
│   ├── CanvasRenderer.tsx      # Existing DOM renderer (unchanged)
│   └── spatial/
│       ├── SpatialCanvas.tsx   # Main WebGL canvas wrapper
│       ├── SpatialScene.tsx    # 3D scene content
│       ├── WidgetLayer3D.tsx   # Widgets in 3D
│       ├── EntityLayer3D.tsx   # Entities in 3D
│       └── XRControls.tsx      # VR/AR specific controls
├── state/
│   └── useSpatialModeStore.ts  # Mode switching state
└── utils/
    └── spatialCoordinates.ts   # 2D ↔ 3D conversion

Testing Strategy

Desktop Mode

• All existing tests pass
• DOM renderer unaffected

VR Mode

• Use WebXR Emulator extension
• Test widget visibility in 3D
• Test interaction raycasting

Mode Switching

• Toggle VR → scene appears
• Toggle back → DOM restored
• State preserved across switches

Performance Considerations

1. Don't render both simultaneously - one visible at a time
2. Lazy load WebGL - only when VR/AR mode requested
3. Dispose Three.js objects - prevent memory leaks on mode switch
4. Throttle state sync - don't update 3D scene on every frame

Common Issues

Issue: Widgets not visible in VR

Cause: Z-position too far or wrong scale Fix: Check toSpatialPosition conversion, verify meters not pixels

Issue: Performance drops when switching modes

Cause: Both renderers running Fix: Ensure only active renderer is mounted/visible

Issue: XR session fails

Cause: HTTPS required, or device unsupported Fix: Check useSpatialModeStore.capabilities, show fallback message

Reference Files

• Mode Store: src/state/useSpatialModeStore.ts
• DOM Renderer: src/components/CanvasRenderer.tsx
• Spatial Canvas: src/components/spatial/SpatialCanvas.tsx (to be created)
• XR Skill: .claude/skills/implementing-spatial-xr/