3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol
Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.
Design inspiration: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) spec-first — define quality criteria and component hierarchy before any rendering; (2) stage-gated sculpting — progressive refinement with acceptance checks at each stage.
Why Spec-First for MCP Rendering?
The original MCP pipeline was reactive: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:
No upfront quality criteria → agent cannot self-assess before rendering
No stage gates → errors compound across steps (bad camera → bad selection → bad edit)
No component hierarchy → edits are flat, no part-level control
The fix: Introduce a define_scene_spec tool that runs before any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.
Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's blockout → structural → form → material → surface → lighting flow.
Gate Evaluation Protocol
For each stage gate, the agent follows this protocol:
1. Execute stage tool (e.g., sculpt_form with parameters)
2. Call render_frame() to get current visual state
3. Call query_scene(query_type="stats") to get quantitative metrics
4. Compare metrics against spec gate criteria
5. If pass → advance to next stage
6. If fail → adjust parameters and retry (max 3 attempts)
7. If 3 failures → report to user with diagnostic info
Voice-Driven Sculpting Example
Loaded on demand — See mcp-tools-spec.md for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).
Code-First Rendering Philosophy (v0.9.0)
Design inspiration: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.
Traditional 3DGS Export vs Code-First Export
Aspect
Traditional (.ply/.splat)
Code-First (.js + .splat)
Editability
Binary blob, hard to edit
Source code, any field adjustable
Version control
Binary diff, no merge
Text diff, git-friendly
File size
Full Gaussian set (MB-GB)
Code skeleton (KB) + compressed splat data
Scene composition
Single flat Gaussian cloud
Hierarchical code with part-level control
Interaction logic
Must be added externally
Embedded in code
3DGS data
All in one file
Separate .splat file loaded by code
Procedural elements
Not supported
Parametric geometry in code (e.g., desk surface = PlaneGeometry)
Hybrid: Procedural Code + 3DGS Splatting
The key insight: not everything needs to be Gaussians. For a desk scene:
Desk surface → procedural BoxGeometry in code (simple, editable, lightweight)
Monitor screen texture → procedural MeshStandardMaterial (or 3DGS if view-dependent)
The code-first approach connects to SLAT (see ../../references/slat-unified-representation.md): the structured latent's voxel grid naturally maps to a procedural geometry skeleton, while the per-voxel features decode to 3DGS splatting for complex regions. SLAT encode → hierarchical decode: simple voxels → procedural code, complex voxels → 3DGS splats.
SLAT Latent Editing (v1.0.0)
Theoretical basis: SLAT (Structured Latent Aggregation Transform) — see ../../references/slat-unified-representation.md. A scene is encoded into a compact structured latent (a voxel grid over the scene, each voxel aggregating local Gaussian features), edited in latent space, then re-decoded back to a Gaussian set. This lets the agent manipulate entire semantic regions with a single operation, independent of per-Gaussian IDs.
Encoding: Scene → Structured Latent
encode_scene_slatent voxelizes the active scene into a regular grid (voxel_size, default 1.0), assigning each Gaussian to a voxel by position. Each voxel stores an aggregated feature vector (mean position, mean scale, mean color, mean opacity, size, plus optional weighted semantic/part labels). The result is a slat_id referencing an in-memory snapshot with an encode_loss (reconstruction RMSE), letting the agent judge fidelity before editing.
Editing in Latent Space
edit_scene_latent applies a LatentEditOp to voxels matched by a LatentSelector (by voxel ids, a spatial box, or a part name — substring, case-insensitive). Seven operations are supported:
Op
Fields
Effect
translate
delta: Vec3
Move matched voxels (and their Gaussians) by a vector
scale
factor: number, origin: Vec3
Scale voxel positions relative to an origin
rotate
angleDeg: number, axis: Vec3, origin: Vec3
Rotate voxels around an axis (degrees)
recolor
color: Vec3, mix: number
Blend matched voxels' colors toward a target
opacity
opacity: number, mode
Set or scale opacity (mode set/scale)
smooth
iterations: number, strength: number
Smooth feature positions/colors by averaging neighbors
delete
target: "voxel"
Remove all Gaussians in matched voxels
Schema vs core naming: the MCP JSON schema uses snake_case (angle_deg); the internal LatentEditOp uses camelCase (angleDeg). Handlers convert at the boundary. Library/test callers use camelCase directly.
Safety gate: edit_scene_latent computes affected_gaussians; if this exceeds 10% of the scene, the edit is rejected unless confirm=true. This reuses the project-wide 10% safety rule.
Apply to scene: with apply_to_scene=true (default) the edit is re-decoded and broadcast to the renderer via modify_gaussians; with false it only updates the in-memory snapshot, so the agent can preview/cancel before committing.
Decoding: Latent → Scene
Decoding rebuilds the Gaussian set: matched voxels are re-instantiated from edited features, untouched voxels keep their original Gaussians. delete removes the affected Gaussians entirely.
Voice-Driven SLAT Example
Loaded on demand — See mcp-tools-spec.md for full SLAT voice examples ("encode the scene", "move the cluster left", "scale the group up", etc.).
Cross-Scene Latent Transfer & Interpolation (v1.1.0)
v1.1 extends SLAT beyond a single scene. A latent edit computed on one scene (source) can now be transferred to another scene (target), or the two scenes can be interpolated in latent space. Both operations rely on a spatial correspondence built over the voxel grids.
Correspondence: Voxel Grid Matching
Both operations build a voxel grid over the source scene (cell size = match_radius) via buildVoxelGrid, then for each target voxel find the nearest source voxel within match_radius (nearestVoxel, 3×3×3 neighborhood search). The resulting pairs carry the relative changes across scenes.
Transferring a Latent Edit
transfer_scene_edit re-applies a LatentEditOp from source to target as a relative change:
Op
Transferred As
translate
Same delta applied to matched target voxels, scaled by strength
recolor
Color offset (target − source voxel color) applied to matched target voxels, scaled by strength
opacity
Opacity ratio (edited / original) scaled toward 1 by strength
delete
Matched target voxels removed when source voxels were deleted
match_radius (default 1.0) bounds the spatial correspondence.
strength (0–1) controls how strongly the source change is applied; 0 applies nothing, 1 applies fully.
Safety gate: if the matched fraction exceeds 10% of the target scene, confirm=true is required (same project-wide 10% rule).
apply_to_scene (default true) re-decodes and broadcasts via modify_gaussians; false only updates the in-memory snapshot for preview.
interpolate_scene_latent
interpolate_scene_latent blends the target scene toward the source in latent space:
t (0–1): 0 = target unchanged, 1 = fully source. Position, color, and opacity are all linearly interpolated per matched voxel.
match_radius (default 1.0) governs the correspondence as above.
Same 10% safety gate and apply semantics as transfer.
Design note: transfer carries relative change (style), while interpolation carries absolute blend (morph). Use transfer to reuse an edit, interpolation to morph one scene into another.
Voice Examples for Cross-Scene Transfer
"transfer the recolor to the other scene" → transfer_scene_edit (op="recolor")
"reuse this translate on scene B" → transfer_scene_edit (op="translate", target_slat_id=sceneB)
"blend scene B toward scene A" → interpolate_scene_latent (t=0.5)
"morph the table into the desk" → interpolate_scene_latent (t=1.0)
MCP Tools Specification
21 core MCP tools (fully implemented) + 13 experimental tools (schema-only stubs) enable agent-controlled 3DGS rendering, editing, sculpting, latent editing, cross-scene transfer, and export. Full JSON schemas are loaded on demand.
#
Tool Name
Description
1
import_scene
Load a 3DGS scene from PLY/SPLAT file or URL
2
set_camera
Set camera position, target, and field of view
3
modify_gaussians
Modify Gaussian properties by selection criteria (IDs, region, label)
4
render_frame
Render current scene from current camera as image
5
query_scene
Query scene stats, bbox, point, segmentation, or materials
6
cast_ray
Cast ray for distance/normal via DDF-GS neural field
7
simulate_physics
Invoke external physics engine (MPM/SPH/PBD) on 3DGS scene
8
query_4d_scene
Query dynamic 3D scene at arbitrary (x,y,t) coordinates
9
deform_elastic
Apply particle-skinned eigenmode deformation to 3DGS object
10
query_spatial_context
Spatial understanding query (grounding, relation, measurement, scene graph)
11
bayesian_density_control
DP-Splat Bayesian nonparametric Gaussian density control
v1.1: SLAT cross-scene latent transfer implemented — 2 new core tools (transfer_scene_edit, interpolate_scene_latent) with voxel-grid spatial correspondence, relative-change transfer, latent interpolation, 2 new voice intent patterns, 10% safety gate, tests passing. Remaining: full voice-driven scene construction (spec → sculpt → export pipeline with real STT).
v1.2: Full voice-driven scene construction (spec → sculpt → export pipeline with real STT) + SLAT cross-scene latent transfer for dynamic/articulated scenes
Rules
Never modify original PLY files: All operations are in-memory only; export requires explicit user command
Validate before render: Always verify camera parameters and Gaussian bounds before rendering
Respect GPU limits: Check available VRAM before loading large scenes; provide downsampling option
Report rendering time: Always include render_time_ms in render_frame output for performance monitoring
Safety gate: Operations affecting >10% of Gaussians require explicit user confirmation
Spec before sculpt (v0.9.0): sculpt_pipeline must not be called without a valid spec_id. The spec defines acceptance criteria; without it, gate evaluation is impossible.
Stage order enforced (v0.9.0): Sculpting stages must execute in order: blockout → structural → form → material → surface → lighting. Skipping stages requires explicit user override.
Code-first default (v0.9.0): When exporting a scene, prefer export_scene_code with format="threejs+splat" over pure .ply export. Pure .ply should only be used when the user explicitly requests a binary blob.
SLAT safety gate (v1.0.0): edit_scene_latent affecting >10% of Gaussians requires confirm=true. Preview with apply_to_scene=false before committing destructive latent edits.
Naming boundary (v1.0.0): MCP tool arguments use snake_case (angle_deg); the core LatentEditOp uses camelCase (angleDeg). Handlers convert at the boundary; never mix cases in the core layer.
Cross-scene safety gate (v1.1.0): transfer_scene_edit and interpolate_scene_latent affecting >10% of target Gaussians require confirm=true. Preview with apply_to_scene=false before committing cross-scene edits.
The following are categorical prohibitions. Violating any of these invalidates the output:
No invented data: Never fabricate MCP tool schemas, API behaviors, or rendering capabilities not in the loaded reference files. If a value is not found, write "data not available" or "N/A".
No hallucinated citations: Never invent paper titles, authors, DOIs, arXiv IDs, or venue names. Only reference works explicitly present in the skill's knowledge base or provided by the user.
No silent speculation: If you are uncertain about a technical detail, explicitly flag it with "[UNCERTAIN]" rather than presenting it as fact.
No method misattribution: Do not assign features, results, or mechanisms from one method to another. Each method's data is specific to that method.
No oversimplified comparisons: Do not reduce multi-dimensional rendering trade-offs to a single judgment without context.
Related Skills
3dgs-engineering-guide — Production deployment (use for end-to-end deployment workflows)
3dgs-spatial-agent — Spatial intelligence agent (use for agent-driven 3D interaction)
3dgs-articulated-reasoner — Articulated object reasoning (use for interactive object manipulation)
3dgs-visualizer — Visualization (use for rendering pipeline output quality assessment)
cad-mesh-3dgs — CAD/Mesh/3DGS conversion (use for code-first export partitioning and SLAT encoding)
nerf-to-3dgs-migrator — NeRF migration (use for SLAT-based component mapping)
SLAT unified representation — See ../../references/slat-unified-representation.md for the shared theoretical framework underlying scene code-first export and latent editing
Guardrail: Do Not Apply From Memory
Do NOT try to apply the logic, method data, bug patterns, or technical details described in this skill from memory. Always read the SKILL.md and referenced files from disk before producing any output. The knowledge base is updated frequently; stale memory may produce outdated, inaccurate, or fabricated results.
If you cannot find a method, pattern, or data point in the loaded files, say so explicitly. Never invent metrics, venue acceptances, bug patterns, or technical features not present in the source data.
18
define_scene_spec
Define Object Spec (hierarchy, materials, quality gates) before sculpting
19
sculpt_pipeline
Execute one stage of spec-first sculpting (6 stages, gate-evaluated)
20
export_scene_code
Export scene as Three.js code + 3DGS splat (code-first philosophy)
21
encode_scene_slatent
Encode current scene into a SLAT structured latent snapshot (voxel grid + per-voxel features)
22
edit_scene_latent
Apply a latent edit (translate/scale/rotate/recolor/opacity/smooth/delete) to a SLAT snapshot, optionally re-decode to scene
23
list_slatents
List in-memory SLAT snapshots (id, voxel count, source Gaussian count)
24
transfer_scene_edit
Transfer a latent edit computed on a source scene to a target scene (relative change, spatial correspondence)
25
interpolate_scene_latent
Interpolate the target scene toward the source in latent space (position/color/opacity blend)
