snap-spectacles-ai

# Spectacles AI — SnapML, RSG, ASR, Snap Cloud AI on Spectacles spans three layers: on-device (SnapML), gateway-mediated cloud (Remote Service Gateway for LLMs / TTS / STT / vision), and your own backend (often Snap Cloud, which is Supabase-flavoured). Each has different latency, privacy, and capability trade-offs. Most production lenses use two of the three; few use all. ## When to load Triggers: - "Add ChatGPT to my Spectacles lens" - "SnapML on Spectacles" - "Voice commands Spectacles" - "Run a vision model on the camera feed" - "Snap Cloud database lens" - "Spectacles Whisper / TTS / ASR" - "Depth-cached vision model" Don't load for: - General Lens Studio basics — [[../snap-spectacles-build/SKILL]]. - Multiplayer that doesn't involve AI — [[../snap-spectacles-sync/SKILL]]. - Navigation / location — [[../snap-spectacles-navigation/SKILL]]. ## Decide first The architecture decisions before code: 1. **On-device, gateway, or your backend?** SnapML on-device for low-latency vision; RSG for LLMs / vision models; your own backend for app-specific data. Most lenses pick two. 2. **Latency budget.** On-device: tens of ms. RSG cloud: hundreds of ms to seconds. Your backend: it's whatever your backend is. 3. **Privacy posture.** On-device data never leaves the glasses. RSG goes through Snap's gateway. Your backend is your responsibility. 4. **Voice in or out?** ASR for in (user speaks to your lens); TTS for out (lens speaks to user). Different surfaces. 5. **Camera vision target.** Object detection / classification on-device with SnapML, or send frames to a cloud vision model? See [[references/snapml-on-device-vs-cloud]]. ## Map of content ### On-device ML - [[references/snapml-pipeline]] — training a model, exporting to Lens Studio, integrating. - [[references/snapml-on-device-vs-cloud]] — when to run on-glasses vs in the cloud. ### Voice - [[references/asr-module]] — speech-to-text on Spectacles. - [[references/voice-prompt-design]] — designing voice commands that actually work. ### Remote Service Gateway - [[references/remote-service-gateway]] — overview of RSG and what it gives you. - [[references/llm-integration]] — wiring LLMs into a lens via RSG. - [[references/tts-stt]] — TTS and STT through RSG. - [[references/vision-models]] — cloud vision (CLIP, GPT-vision) on camera frames. - [[references/depth-cache-vision]] — depth-frame caching so cloud vision gets 3D-aware results. ### Your backend - [[references/snap-cloud]] — Snap Cloud (Supabase) for lens-side database, auth, storage. - [[references/fetch-api]] — Fetch HTTPS endpoints from a lens. - [[references/web-view]] — WebView for browser content (limited use). ## Verify Before claiming an AI-enabled lens is shippable: - [ ] If using on-device ML, the model size / inference time fits the [[../snap-spectacles-design/references/performance-budget]]. - [ ] If using cloud, latency is acceptable in the UX (typically <1 s for tap-driven flows; longer for batch / passive). - [ ] Voice ASR is tested with multiple accents / noisy environments — not just a quiet office. - [ ] Camera-stream models work in real lighting, not just lab. - [ ] User-data flows are documented — what leaves the device, where it goes, how long it persists. - [ ] Permissions for camera / microphone / internet are declared and the UX explains why. - [ ] Cost ceiling for cloud calls (RSG, your backend) is set; the lens can't blow up your bill. ## Smoke test If this skill loaded correctly, the agent should answer: 1. Should I run my object-detection model on-device or in the cloud? (Expected: depends on latency, privacy, model size — on-device for low-latency / private; cloud for big models the device can't run; cites [[references/snapml-on-device-vs-cloud]].) 2. What's Remote Service Gateway and why use it? (Expected: Snap's gateway for accessing LLMs, TTS, STT, and vision models from a lens without managing your own backend or API keys; cites [[references/remote-service-gateway]].) 3. The user wants Whisper-quality ASR on Spectacles. What's available? (Expected: ASR Module on-device or higher-quality STT via RSG; trade off latency / quality / cost; cites [[references/asr-module]] and [[references/tts-stt]].) ## Sibling skills - [[../snap-spectacles-build/SKILL]] — Lens Studio basics. - [[../snap-spectacles-sync/SKILL]] — multiplayer lenses that often pair with AI features. - [[../snap-spectacles-design/SKILL]] — UX for AI-driven lenses (voice, conversational, vision-feedback). ## Sources - Spectacles features hub — <https://developers.snap.com/spectacles/home> - Snap AR developer landing — <https://ar.snap.com> - Sample AI projects — <https://github.com/specs-devs/samples> (AI Playground, Agentic Playground, SnapML samples)