xrblocks

Guide to building AI + XR applications with the XR Blocks SDK (the `xrblocks` package, source in `src/`). Use when writing, editing, or debugging WebXR / Android XR / VR / AR / mixed-reality experiences on this framework — authoring `xb.Script` classes, configuring `xb.Options`, and wiring spatial UI, hand tracking, gestures, depth & occlusion, plane/object detection, physics, spatial audio, or Gemini/OpenAI integration — and when running them in the desktop simulator or on-device. Covers the canonical app skeleton, the real `enable*` option methods, the `Script` lifecycle hooks, the global aliases (`xb.core`, `xb.user`, `xb.world`, `xb.ai`...), and the most common hallucinated-API mistakes to avoid. Read before generating XR Blocks code.

Develop and test XR Blocks apps on the desktop without a headset using the built-in simulator — a simulated user, hands, depth, and planes rendered in a normal browser, with control modes for moving the user, posing hands, or driving controllers. Use when running/iterating locally, reproducing XR interactions on desktop, posing hands for gesture work, or adding the optional 2D simulator settings UI. Covers the `?formFactor=desktop` autostart, `options.simulator.*`, `xb.SimulatorMode`, the `SimulatorAddons` 2D UI import, and the `onSimulatorStarted()` hook.

Write sequential asynchronous functional, integration, or simulator tests for xrblocks apps using the testing addon. Use this when you need to mock WebGL/WebAudio in headless environments (like JSDOM / Vitest), simulate user locomotion, trigger controller pointing, raycasts, and select/squeeze hand inputs. Covers `TestRunner` and `TestRunnerConfig`.

Build rich spatial user interfaces in XR Blocks apps with the uiblocks addon — flexbox-laid-out 3D cards and panels with gradients, strokes, rounded corners, drop/inner shadows, MSDF text, material icons, images, and spatial behaviors (head-leash, billboard, grab/manipulate, object-anchor, show/hide animations). Use when authoring or debugging `UICard` / `UIPanel` / `UIText` / `UIImage` / `UIIcon` UI imported from `xrblocks/addons/uiblocks/src` (wrapping `@pmndrs/uikit` + yoga-layout) — for panels, menus, HUDs, dialogs, or any styled, web-like UI in WebXR / Android XR. Includes the bootstrap (`options.uikit.enable(uikit)` + `raycastSortFunction`), styling/layout rules, behavior config, and a troubleshooting playbook for clicks/styling/sizing failures. (For lightweight panels with no extra deps, prefer core `xb.SpatialPanel` instead.) Includes a design guide (§6) for UX designers composing complex, elegant, multi-section spatial UI — tokens, spatial comfort, elevation/shadows, passthrough legibility, motion, a

Add real-time multiplayer to an XR Blocks app with the netblocks addon — presence avatars (remote heads + hands), replicated `NetObject` transforms with cooperative ownership, typed pub/sub RPC events, and opt-in spatial WebRTC voice, over pluggable transports (BroadcastChannel for local dev, WebRTC/PeerJS for serverless P2P, WebSocket relay for scalable rooms). Use when authoring or debugging shared/co-located XR, `enableNet()`, `joinRoom()`, `NetObject`, `session.events`, or `session.voice` imported from `xrblocks/addons/netblocks/src`. For the wire protocol, transport details, and threat model, read the full reference in this folder's README.md.

Add audio-driven avatar mouths to an XR Blocks app with the lipsync addon — heuristic vowel-formant viseme mapping that turns any `MediaStream` (mic or remote peer's voice) into mouth shapes on a `StylizedFace` decal attached to an avatar's head. Zero ML runtime, no model download. Use when you want shared rooms to stop being silent spheres and become faces that visibly speak, or to lip-sync a TTS playback to an NPC. Covers `LipsyncMouth`, `xb.StylizedFace`, the `target`/`audioContext`/`fftSize` constructor options, and the `session.voice.onTrack` netblocks pairing. Lower-level pieces (`FormantVisemeMapper`, `MfccExtractor`, `computeAudioFeatures`) and types (`VisemeWeights`, `VisemeTarget`) are exported for swapping in a model-based mapper. Full reference at src/addons/lipsync/.

Add audio-driven avatar mouths to an XR Blocks app with the lipsync addon — heuristic vowel-formant viseme mapping that turns any `MediaStream` (mic or remote peer's voice) into mouth shapes on a `StylizedFace` canvas decal attached to an avatar's head. No ML runtime, no model download. Use when authoring or debugging avatar mouths in single-user demos or in multiplayer scenes paired with `xb-netblocks` (every remote peer's voice stream drives their own face). Public surface: `LipsyncMouth` (the driver), `xb.StylizedFace` (the decal, in xrblocks core), `session.voice.onTrack` for the netblocks hook, plus the lower-level `FormantVisemeMapper`, `MfccExtractor`, `computeAudioFeatures` and types (`VisemeWeights`, `VisemeTarget`, `AudioFeatures`) for plugging in a model-based mapper later. For the DSP pipeline, caveats, and samples read this folder's README.md.

Add real-time multiplayer to an XR Blocks app with the netblocks addon — presence avatars (remote heads + hands), replicated `NetObject` transforms with cooperative ownership, typed pub/sub RPC events, and opt-in spatial WebRTC voice, over pluggable transports (BroadcastChannel for local dev, WebRTC/PeerJS for serverless P2P, WebSocket relay for scalable rooms). Use when building shared or co-located XR. Covers `enableNet()`, `joinRoom()`, `NetObject`, and `session.events`; the full reference (wire protocol, transports, threat model) is at src/addons/netblocks/.

Add WebXR depth sensing to an XR Blocks app — a live depth map and depth mesh of the room — so virtual content is occluded by real-world geometry and (optionally) generates physics colliders that match the environment. Use for realistic occlusion (a person walks in front of a virtual object), depth-aware reticles, or depth-mesh collision for physics. Covers `enableDepth()`, the `DepthOptions` presets (`xrDepthMeshOptions`, `xrDepthMeshPhysicsOptions`), `colliderUpdateFps`, and `showReticleOnDepthMesh`. Pair with xb-physics for geometry-aware collisions.

Integrate generative AI into an XR Blocks app via `xb.ai` — text and multimodal (image) queries to Gemini or OpenAI, real-time Gemini Live audio/video sessions, and image generation. Use to answer questions, describe what the camera sees, drive conversational agents, or generate content from an XR scene. Covers `enableAI()`, `xb.ai.isAvailable()`, `xb.ai.query()`, `startLiveSession()`, `generate()`, and local API-key handling (`?key=` / `keys.json`). Always guard calls with `isAvailable()`; never ship keys in production client code.

Bootstrap and structure an XR Blocks app: the `xb.Script` lifecycle, `xb.Options` configuration with chainable `enable*()` methods, the global aliases (`xb.core`, `xb.user`, `xb.scene`, `xb.world`, `xb.ai`…), and how to add objects and run the app in the desktop simulator or on an XR device. Use this first whenever creating a new XR Blocks experience or wiring up the main entry point, the frame loop (`update`), or select/pinch/click and key input. Other skills (xb-ui, xb-hands, xb-depth, xb-ai, …) build on this foundation.

Detect named hand gestures in an XR Blocks app — pinch, open-palm, fist, thumbs-up, point, and spread — and subscribe to gesturestart / gestureupdate / gestureend events with per-hand name and confidence. Use when triggering actions from poses rather than raw pinch/touch (e.g. point-to-aim, open-palm menu, thumbs-up confirm). Covers `enableGestures()`, `options.gestures.setGestureEnabled`, and the `xb.core.gestureRecognition` event API. For raw hand joints/touch/grab, use xb-hands; for stroke/shape recognition use the strokes API.

Add WebXR hand tracking to an XR Blocks app — enable hand joints, optional joint and mesh visualizations, pinch-to-select, and direct touch/grab of meshes. Use when you need the user's hands (index-finger touch, grab, wrist/joint positions) rather than just controller rays, or want to visualize hands in the simulator's pose mode. Covers `enableHands()`, the `options.hands.*` flags, `xb.user.hands` (`getIndexTip`, `getWrist`), and the object touch/grab lifecycle hooks. For named gestures (fist, point, spread…) use xb-gestures.

Load and display 3D models in an XR Blocks app with `xb.ModelViewer` — GLTF/GLB models (static or animated), Gaussian-splat (.spz) models, or your own `THREE.Object3D` — wrapped with an optional draggable platform, bounding box, and raycast cylinder so users can grab and reposition them in XR. Use when spawning, scaling, or placing 3D objects/assets in the scene, putting a model inside a UI panel, or auto-placing a model on a real-world horizontal surface. Covers `loadGLTFModel`, `loadSplatModel`, `setupPlatform`, and `world.placeOnHorizontalSurface`.

Add rigid-body physics to an XR Blocks app using the Rapier3D engine — gravity, collisions, and (with xb-depth) colliders that match real-world geometry so virtual objects bounce off the floor and furniture. Use for ball pits, throwing/ dropping objects, projectiles, or any depth-aware physical interaction. Covers enabling physics via `options.physics.RAPIER`, the `initPhysics(physics)` / `physicsStep()` Script hooks, and the depth-mesh collider recipe. There is no `enablePhysics()` method — assigning `options.physics.RAPIER` is what turns it on.

Play and capture audio in an XR Blocks app via `xb.sound` — spatial (positional) audio anchored to scene objects, master volume control, microphone recording and playback, plus speech recognition and synthesis. Use to add sound effects, 3D-positioned audio, voice recording, or text-to-speech / speech-to-text. Covers `xb.core.sound` (`startRecording`/`stopRecording`/`playRecordedAudio`, `getAudioListener`, `setMasterVolume`) and the exported `SpeechRecognizer` / `SpeechSynthesizer` / `SoundSynthesizer` classes. See samples/sound.

Build spatial UI in an XR Blocks app with the core `xb.SpatialPanel` system — a grid of rows and columns holding text, text buttons, icon buttons, and images, with a unified `onTriggered` select handler. Use for HUDs, menus, dialogs, and control panels when you want a lightweight, no-extra-dependency UI. For rich flexbox layouts with gradients, strokes, and shadows, use xb-uiblocks instead. Covers `addGrid`/`addRow`/`addCol`/`addText`/`addTextButton`/`addIconButton`, panel positioning, and the `enableUI()` setup.

Build rich spatial UI with the uiblocks addon — flexbox-laid-out 3D cards/panels (`UICard`, `UIPanel`, `UIText`, `UIImage`, `UIIcon`) with gradients, strokes, rounded corners, drop/inner shadows, and spatial behaviors (head-leash, billboard, grab/manipulate, object-anchor). Use when you need web-like styling fidelity or real flexbox layout beyond the lightweight core `xb.SpatialPanel`. Requires `options.uikit.enable(uikit)` and the `@pmndrs/uikit` peer deps. The complete reference (styling rules, behaviors, gotchas, troubleshooting) lives at src/addons/uiblocks/SKILL.md.

Understand the real world in an XR Blocks app via `xb.world` — detect horizontal/ vertical planes, reconstruct scene meshes, and recognize physical objects (with a Gemini or MediaPipe backend) as `THREE.Object3D`s positioned in 3D. Use to place virtual content on real surfaces, react to room geometry, or attach affordances to recognized objects. Covers `enablePlaneDetection()`, `enableObjectDetection()`, `options.world.*`, `xb.world.objects.runDetection()`, and `xb.world.placeOnHorizontalSurface()`. For raw depth/occlusion use xb-depth.