Virtual Avatar Engineering

Virtual Avatar Engineering & Real-Time Computer Vision

This skill outlines the process of building high-performance, real-time virtual avatars and computer vision pipelines using Svelte 5 and SvelteKit. It focuses on isolating heavy Machine Learning (ML) inference from the main UI thread, applying mathematical biomechanical calculations, and ensuring stable 60+ FPS broadcast-quality rendering.

Activation Criteria

Use this skill when:

• Developing web-based VTuber avatars or Face/Hand/Holistic tracking pipelines using MediaPipe.
• Implementing low-latency device streams (getUserMedia) and processing raw video frames.
• Integrating 2D (Canvas), 2.5D (Live2D), or 3D (VRM/Three.js) avatars driven by real-time sensor data.
• Isolating heavy computational workloads using Web Workers and WebAssembly.

1. Computer Vision & WebWorker Isolation

Computer vision inference (like MediaPipe) blocks the main thread, causing severe UI jank and dropped frames in avatar rendering.

• Worker Isolation: Always execute ML models and computer vision pipelines inside a dedicated Web Worker. The main thread should only handle the Svelte UI and the WebGL/Canvas rendering loop.
• Zero-Copy Transfers: Use ImageBitmap and postMessage with Transferable Objects to send video frames to the Web Worker without duplicating memory.
• Execution Delegates: Explicitly configure execution delegates (CPU vs. GPU) depending on the environment. For broadcast setups (e.g., OBS Studio browser sources), CPU delegates often perform better because OBS heavily saturates the GPU.

2. Mathematical Animation & Biomechanics

Raw tracking data from computer vision models is inherently noisy and jittery. It must be smoothed before driving avatar rigs.

• Linear Interpolation (LERP): Apply LERP to all positional data (x, y, z coordinates) and rotational data (quaternions/Eulers) to smooth out camera micro-jitters.

  // LERP Function
  const lerp = (start, end, amt) => (1 - amt) * start + amt * end;
  

• Biomechanical Metrics: Compute relational metrics rather than absolute coordinates to handle varying face distances.
  • Eye Aspect Ratio (EAR): Calculate EAR to determine if an eye is open or closed, triggering blink animations.
  • Mouth Open Factor (MOF): Calculate the vertical distance between inner lip landmarks divided by face height to drive the avatar's jaw bone.
• Geometric Clamping: Clamp tracking metrics to strict upper and lower bounds to prevent avatars from contorting unnaturally when landmarks fail.

3. Svelte 5 Rendering Architecture

Separate the reactive UI state from the 60+ FPS rendering context.

• Runes Protocol: Use Svelte 5 $state and $derived strictly for UI configurations (e.g., selecting a microphone, toggling debug overlays, changing avatars).
• Decoupled Rendering: The Canvas or WebGL context (Three.js/PixiJS) must be updated inside a pure requestAnimationFrame loop. Do not bind high-frequency avatar bone transforms directly to Svelte reactive state variables.
• Immutable Config Sync: When UI configuration changes (e.g., user selects a new accessory), sync the data to the render loop using immutable references to prevent deep-diffing overhead.

4. Device Streams & Defensive Programming

Hardware access is highly volatile. The application must gracefully handle stream failures and permission denials.

• Permission Handling: Wrap navigator.mediaDevices.getUserMedia in try/catch blocks. Provide clear Svelte UI fallback states if permissions are denied or hardware is missing.
• Stream Cleanup: If a component unmounts or a user switches cameras, you must explicitly call track.stop() on all MediaStream tracks.
• WASM Teardown: MediaPipe WASM instances consume significant memory. Ensure the Web Worker receives a teardown message to run .close() on tasks when the avatar session ends.

5. Self-Validation Protocol

Before deploying or committing a virtual avatar component, verify:

• Thread Safety: Is the MediaPipe inference fully isolated in a Web Worker?
• Memory Leaks: Does the $effect teardown stop the video stream, cancel the requestAnimationFrame, and terminate the Web Worker?
• Smoothing: Is raw tracking data being smoothed via LERP or moving averages before being applied to the skeleton?
• Fallback UI: Is there a reactive Svelte UI state ($state(error)) indicating when a camera is inaccessible or the ML model failed to load?

Reference Example

Below is a conceptual Svelte 5 component demonstrating stream initialization, worker isolation, and decoupled rendering.

<script lang="ts">
	import { onDestroy } from 'svelte';

	let videoElement: HTMLVideoElement;
	let canvasElement: HTMLCanvasElement;
	
	let streamStatus = $state<'loading' | 'active' | 'error'>('loading');
	let errorMessage = $state('');

	let worker: Worker;
	let rAFId: number;
	let mediaStream: MediaStream | null = null;

	// Avatar State (Non-reactive for performance)
	const avatarState = {
		headYaw: 0,
		mouthOpen: 0,
		lerpFactor: 0.2
	};

	$effect(() => {
		// 1. Initialize Web Worker for ML isolation
		worker = new Worker(new URL('./vision-worker.ts', import.meta.url), { type: 'module' });
		
		worker.onmessage = (e) => {
			// Receive raw data, apply LERP smoothing
			const { rawYaw, rawMouth } = e.data;
			avatarState.headYaw = lerp(avatarState.headYaw, rawYaw, avatarState.lerpFactor);
			avatarState.mouthOpen = lerp(avatarState.mouthOpen, rawMouth, avatarState.lerpFactor);
		};

		// 2. Request Camera
		navigator.mediaDevices.getUserMedia({ video: { width: 640, height: 480 }, audio: false })
			.then((stream) => {
				mediaStream = stream;
				videoElement.srcObject = stream;
				videoElement.play();
				streamStatus = 'active';
				
				// 3. Start render loop
				startRenderLoop();
			})
			.catch((err) => {
				streamStatus = 'error';
				errorMessage = err.message;
			});

		// 4. Defensive Cleanup
		return () => {
			if (rAFId) cancelAnimationFrame(rAFId);
			if (worker) worker.terminate();
			if (mediaStream) {
				mediaStream.getTracks().forEach(track => track.stop());
			}
		};
	});

	function startRenderLoop() {
		const ctx = canvasElement.getContext('2d');
		
		const loop = () => {
			if (videoElement.readyState >= 2 && streamStatus === 'active') {
				// Send frame to worker (simplified concept, usually draw to OffscreenCanvas)
				worker.postMessage({ type: 'PROCESS_FRAME' });
				
				// Render Avatar to Canvas based on smoothed avatarState
				ctx.clearRect(0, 0, canvasElement.width, canvasElement.height);
				ctx.fillStyle = 'blue'; // Example mouth
				ctx.fillRect(150, 200, 100, avatarState.mouthOpen * 50);
			}
			rAFId = requestAnimationFrame(loop);
		};
		loop();
	}

	const lerp = (start: number, end: number, amt: number) => (1 - amt) * start + amt * end;
</script>

<div class="avatar-container">
	{#if streamStatus === 'loading'}
		<p>Initializing Camera and ML Models...</p>
	{:else if streamStatus === 'error'}
		<p class="error">Camera Error: {errorMessage}</p>
	{/if}

	<!-- Hidden video element for stream source -->
	<video bind:this={videoElement} style="display: none;" playsinline muted></video>
	
	<!-- Avatar Render Target -->
	<canvas bind:this={canvasElement} width="640" height="480"></canvas>
</div>

<style>
	.avatar-container { position: relative; }
	.error { color: red; }
</style>