snap-spectacles-ai

name	snap-spectacles-ai
description	Load when the user is adding AI to a Spectacles lens — SnapML on-device models, voice ASR, Remote Service Gateway for LLMs and TTS, depth-cached vision, Snap Cloud as backend, or asks "how do I run a vision model on Spectacles." Don't load for general Lens Studio basics — use snap-spectacles-build.
license	MIT
metadata	{"author":"HKTITAN","version":"1.0.0","graph":true,"depends":["snap-spectacles-build"]}

Spectacles AI — SnapML, RSG, ASR, Snap Cloud

AI on Spectacles spans three layers: on-device (SnapML), gateway-mediated cloud (Remote Service Gateway for LLMs / TTS / STT / vision), and your own backend (often Snap Cloud, which is Supabase-flavoured). Each has different latency, privacy, and capability trade-offs. Most production lenses use two of the three; few use all.

When to load

Triggers:

• "Add ChatGPT to my Spectacles lens"
• "SnapML on Spectacles"
• "Voice commands Spectacles"
• "Run a vision model on the camera feed"
• "Snap Cloud database lens"
• "Spectacles Whisper / TTS / ASR"
• "Depth-cached vision model"

Don't load for:

• General Lens Studio basics — [[../snap-spectacles-build/SKILL]].
• Multiplayer that doesn't involve AI — [[../snap-spectacles-sync/SKILL]].
• Navigation / location — [[../snap-spectacles-navigation/SKILL]].

Decide first

The architecture decisions before code:

1. On-device, gateway, or your backend? SnapML on-device for low-latency vision; RSG for LLMs / vision models; your own backend for app-specific data. Most lenses pick two.
2. Latency budget. On-device: tens of ms. RSG cloud: hundreds of ms to seconds. Your backend: it's whatever your backend is.
3. Privacy posture. On-device data never leaves the glasses. RSG goes through Snap's gateway. Your backend is your responsibility.
4. Voice in or out? ASR for in (user speaks to your lens); TTS for out (lens speaks to user). Different surfaces.
5. Camera vision target. Object detection / classification on-device with SnapML, or send frames to a cloud vision model? See [[references/snapml-on-device-vs-cloud]].

Map of content

On-device ML

• [[references/snapml-pipeline]] — training a model, exporting to Lens Studio, integrating.
• [[references/snapml-on-device-vs-cloud]] — when to run on-glasses vs in the cloud.

Voice

• [[references/asr-module]] — speech-to-text on Spectacles.
• [[references/voice-prompt-design]] — designing voice commands that actually work.

Remote Service Gateway

• [[references/remote-service-gateway]] — overview of RSG and what it gives you.
• [[references/llm-integration]] — wiring LLMs into a lens via RSG.
• [[references/tts-stt]] — TTS and STT through RSG.
• [[references/vision-models]] — cloud vision (CLIP, GPT-vision) on camera frames.
• [[references/depth-cache-vision]] — depth-frame caching so cloud vision gets 3D-aware results.

Your backend

• [[references/snap-cloud]] — Snap Cloud (Supabase) for lens-side database, auth, storage.
• [[references/fetch-api]] — Fetch HTTPS endpoints from a lens.
• [[references/web-view]] — WebView for browser content (limited use).

Verify

Before claiming an AI-enabled lens is shippable:

• If using on-device ML, the model size / inference time fits the [[../snap-spectacles-design/references/performance-budget]].
• If using cloud, latency is acceptable in the UX (typically <1 s for tap-driven flows; longer for batch / passive).
• Voice ASR is tested with multiple accents / noisy environments — not just a quiet office.
• Camera-stream models work in real lighting, not just lab.
• User-data flows are documented — what leaves the device, where it goes, how long it persists.
• Permissions for camera / microphone / internet are declared and the UX explains why.
• Cost ceiling for cloud calls (RSG, your backend) is set; the lens can't blow up your bill.

Smoke test

If this skill loaded correctly, the agent should answer:

1. Should I run my object-detection model on-device or in the cloud? (Expected: depends on latency, privacy, model size — on-device for low-latency / private; cloud for big models the device can't run; cites [[references/snapml-on-device-vs-cloud]].)
2. What's Remote Service Gateway and why use it? (Expected: Snap's gateway for accessing LLMs, TTS, STT, and vision models from a lens without managing your own backend or API keys; cites [[references/remote-service-gateway]].)
3. The user wants Whisper-quality ASR on Spectacles. What's available? (Expected: ASR Module on-device or higher-quality STT via RSG; trade off latency / quality / cost; cites [[references/asr-module]] and [[references/tts-stt]].)

Sibling skills

• [[../snap-spectacles-build/SKILL]] — Lens Studio basics.
• [[../snap-spectacles-sync/SKILL]] — multiplayer lenses that often pair with AI features.
• [[../snap-spectacles-design/SKILL]] — UX for AI-driven lenses (voice, conversational, vision-feedback).

Sources

• Spectacles features hub — https://developers.snap.com/spectacles/home
• Snap AR developer landing — https://ar.snap.com
• Sample AI projects — https://github.com/specs-devs/samples (AI Playground, Agentic Playground, SnapML samples)