visionos-3d-development

name

visionos-3d-development

description

Comprehensive latest-first guidance for Apple Vision Pro 3D development with RealityKit, RealityFoundation, ARKit, SwiftUI spatial APIs, Compositor Services, Spatial, simd, PHASE, Model I/O, and directly related rendering, physics, interaction, asset-pipeline, and performance workflows. Use when Codex needs to design, build, debug, review, migrate, or optimize visionOS 27 / Xcode 27 / Swift 6.4 3D apps, immersive spaces, volumetric experiences, games, simulations, tracking systems, or low-level immersive renderers for Apple Vision Pro.

visionOS 3D Development

Overview

Build Apple Vision Pro 3D software against the current local stack on this machine: XROS27.0.sdk with Apple Swift 6.4. Use the skill to answer both workflow questions and API-surface questions without falling back to stale pre-visionOS patterns. Confirm the active SDK with scripts/locate_xros_sdk.sh before version-sensitive work.

Source Order

1. Trust the installed XROS27.0 SDK and generated API indexes first for symbol truth, availability, and naming.
2. Use official Apple DocC pages, sample-code pages, and WWDC sessions for guidance, best practices, and architecture.
3. Use older 2023-2024 material only when it still describes the current model or can be translated cleanly to the current APIs.
4. Treat SceneKit and older iOS ARKit app architecture as legacy unless the task is explicitly migration or interop.

Quick Start

1. Confirm the local SDK with scripts/locate_xros_sdk.sh.
2. Read references/00-navigation.md for the map.
3. Read references/01-authority-and-versioning.md before using older Apple material.
4. Read references/02-framework-selection.md to choose the right stack.
5. Read the smallest relevant framework or pattern file in references/frameworks/ or references/patterns/.
6. Query broad API coverage with python3 scripts/query_api_index.py ....
7. Query official sources with python3 scripts/query_source_catalog.py ....

Decision Rules

• For scene content, entities, components, systems, animation, physics, particles, portals, and immersive interaction: start with RealityKit and RealityFoundation.
• For sensing, tracking, anchors, room understanding, hand tracking, object tracking, world tracking, accessory tracking, and permissions: add ARKit.
• For app shell, windows, volumes, immersive spaces, ornaments, and 2D/3D composition: add SwiftUI spatial APIs.
• For fully custom immersive renderers or direct stereo Metal control: add Compositor Services and Metal.
• For transforms, poses, bounds, and coordinate math: use Spatial and simd rather than ad hoc math types.
• For spatial audio: use PHASE and the visionOS audio references.
• For asset ingestion and preprocessing: use Model I/O plus Reality Composer Pro and USD assets.

Reference Map

• references/00-navigation.md: entry map and suggested reading paths.
• references/01-authority-and-versioning.md: current-version policy, normalization rules, and migration caveats.
• references/02-framework-selection.md: choose between RealityKit, ARKit, SwiftUI, Compositor Services, PHASE, Model I/O, Metal, and supporting APIs.
• references/frameworks/realitykit.md: RealityKit and RealityFoundation workflow.
• references/frameworks/arkit.md: tracking, anchors, scene understanding, and ARKit integration.
• references/frameworks/swiftui-spatial.md: windows, volumes, immersive spaces, RealityView, spatial layout, and presentation.
• references/frameworks/compositor-services.md: low-level immersive rendering and Metal integration.
• references/frameworks/metal-immersive.md: curated Metal and MetalKit guidance for custom immersive renderers.
• references/frameworks/spatial-audio-assets.md: Spatial, simd, PHASE, Model I/O, Reality Composer Pro, USD, and templates.
• references/frameworks/games-and-simulation.md: game-specific architecture and simulation guidance.
• references/patterns/ecs-and-systems.md: entity/component/system design.
• references/patterns/transforms-input-physics.md: coordinate conversion, gestures, manipulation, collision, and physics.
• references/patterns/anchors-tracking-and-scene-understanding.md: current ARKit provider model and environment integration.
• references/patterns/rendering-animation-and-audio.md: rendering, animation, and audio rules.
• references/patterns/performance-and-debugging.md: performance, profiling, testing, and review rules.
• references/examples/latest-recipes.md: short current-style code patterns.
• references/examples/template-and-sample-map.md: Apple templates, sample projects, WWDC sessions, and where each one helps.
• references/indexes/README.md: generated local API/source indexes.
• references/advanced-hand-physics/00-navigation.md: advanced custom hand-object physics, solver design, and simulation architecture.
• references/implementation-hand-simulation/00-navigation.md: implementation-facing solver, runtime, stabilization, and release mechanics for advanced hand simulation.

Search Patterns

• Search a symbol locally: python3 scripts/query_api_index.py --module RealityKit PortalComponent
• Search a concept locally: python3 scripts/query_api_index.py collision
• Search curated SwiftUI spatial APIs: python3 scripts/query_api_index.py --module SwiftUI-spatial ImmersiveSpace
• Search curated Metal immersive APIs: python3 scripts/query_api_index.py --module Metal-immersive MTLRenderPassDescriptor
• Search curated hand-physics APIs: python3 scripts/query_api_index.py --module Hand-physics-apis HandTrackingProvider
• Search curated hand-simulation runtime APIs: python3 scripts/query_api_index.py --module Hand-physics-simulation-apis PhysicsSimulationComponent
• Search official Apple sources: python3 scripts/query_source_catalog.py room-tracking
• Search advanced hand-physics sources: python3 scripts/query_source_catalog.py --catalog references/advanced-hand-physics/hand_physics_source_catalog.json grasp
• Search implementation hand-simulation sources: python3 scripts/query_source_catalog.py --catalog references/implementation-hand-simulation/simulation_source_catalog.json release
• Open index overviews first: references/indexes/local-source-map.md and references/indexes/official-source-map.md

Review And Implementation Guardrails

• Prefer visionOS 27, XROS27.0, and Swift 6.4 patterns unless the user explicitly targets older deployment baselines.
• State when a recommendation is inferred from local SDK availability rather than directly stated in a guide.
• When Apple material spans 2023-2025, keep the mental model but update names, providers, and SwiftUI patterns to current APIs.
• For code review, flag outdated ARSession-centric patterns, ad hoc transform math, heavy scene-graph mutation on the wrong actor, and Compositor Services used where RealityKit would be simpler.