HKTITAN Spectacles AI — SnapML, RSG, ASR, Snap Cloud
AI on Spectacles spans three layers: on-device (SnapML), gateway-mediated cloud (Remote Service Gateway for LLMs / TTS / STT / vision), and your own backend (often Snap Cloud, which is Supabase-flavoured). Each has different latency, privacy, and capability trade-offs. Most production lenses use two of the three; few use all.
When to load
Triggers:
- "Add ChatGPT to my Spectacles lens"
- "SnapML on Spectacles"
- "Voice commands Spectacles"
- "Run a vision model on the camera feed"
- "Snap Cloud database lens"
- "Spectacles Whisper / TTS / ASR"
- "Depth-cached vision model"
Don't load for:
- General Lens Studio basics — [[../snap-spectacles-build/SKILL]].
- Multiplayer that doesn't involve AI — [[../snap-spectacles-sync/SKILL]].
- Navigation / location — [[../snap-spectacles-navigation/SKILL]].
Decide first
The architecture decisions before code:
- On-device, gateway, or your backend? SnapML on-device for low-latency vision; RSG for LLMs / vision models; your own backend for app-specific data. Most lenses pick two.
- Latency budget. On-device: tens of ms. RSG cloud: hundreds of ms to seconds. Your backend: it's whatever your backend is.
- Privacy posture. On-device data never leaves the glasses. RSG goes through Snap's gateway. Your backend is your responsibility.
- Voice in or out? ASR for in (user speaks to your lens); TTS for out (lens speaks to user). Different surfaces.
- Camera vision target. Object detection / classification on-device with SnapML, or send frames to a cloud vision model? See [[references/snapml-on-device-vs-cloud]].
Map of content
On-device ML
- [[references/snapml-pipeline]] — training a model, exporting to Lens Studio, integrating.
- [[references/snapml-on-device-vs-cloud]] — when to run on-glasses vs in the cloud.
Voice
- [[references/asr-module]] — speech-to-text on Spectacles.
- [[references/voice-prompt-design]] — designing voice commands that actually work.
Remote Service Gateway
- [[references/remote-service-gateway]] — overview of RSG and what it gives you.
- [[references/llm-integration]] — wiring LLMs into a lens via RSG.
- [[references/tts-stt]] — TTS and STT through RSG.
- [[references/vision-models]] — cloud vision (CLIP, GPT-vision) on camera frames.
- [[references/depth-cache-vision]] — depth-frame caching so cloud vision gets 3D-aware results.
Your backend
- [[references/snap-cloud]] — Snap Cloud (Supabase) for lens-side database, auth, storage.
- [[references/fetch-api]] — Fetch HTTPS endpoints from a lens.
- [[references/web-view]] — WebView for browser content (limited use).
Verify
Before claiming an AI-enabled lens is shippable:
- If using on-device ML, the model size / inference time fits the [[../snap-spectacles-design/references/performance-budget]].
- If using cloud, latency is acceptable in the UX (typically <1 s for tap-driven flows; longer for batch / passive).
- Voice ASR is tested with multiple accents / noisy environments — not just a quiet office.
- Camera-stream models work in real lighting, not just lab.
- User-data flows are documented — what leaves the device, where it goes, how long it persists.
- Permissions for camera / microphone / internet are declared and the UX explains why.
- Cost ceiling for cloud calls (RSG, your backend) is set; the lens can't blow up your bill.
Smoke test
If this skill loaded correctly, the agent should answer:
- Should I run my object-detection model on-device or in the cloud? (Expected: depends on latency, privacy, model size — on-device for low-latency / private; cloud for big models the device can't run; cites [[references/snapml-on-device-vs-cloud]].)
- What's Remote Service Gateway and why use it? (Expected: Snap's gateway for accessing LLMs, TTS, STT, and vision models from a lens without managing your own backend or API keys; cites [[references/remote-service-gateway]].)
- The user wants Whisper-quality ASR on Spectacles. What's available? (Expected: ASR Module on-device or higher-quality STT via RSG; trade off latency / quality / cost; cites [[references/asr-module]] and [[references/tts-stt]].)
Sibling skills
- [[../snap-spectacles-build/SKILL]] — Lens Studio basics.
- [[../snap-spectacles-sync/SKILL]] — multiplayer lenses that often pair with AI features.
- [[../snap-spectacles-design/SKILL]] — UX for AI-driven lenses (voice, conversational, vision-feedback).
Sources
- Spectacles features hub — https://developers.snap.com/spectacles/home
- Snap AR developer landing — https://ar.snap.com
- Sample AI projects — https://github.com/specs-devs/samples (AI Playground, Agentic Playground, SnapML samples)