3dgs-mcp-renderer

name	3dgs-mcp-renderer
description	MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering, voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering, agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting, interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering.
license	Apache-2.0
metadata	{"version":"0.7.0","author":"jaccen","tags":["mcp","3dgs","gaussian-splatting","rendering","three.js","webgpu","voice","agent","interactive"],"disable-model-invocation":true,"user-invocable":true}

3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

Architecture

┌─────────────┐     ┌─────────────┐     ┌──────────────────┐     ┌──────────────────┐
│ Voice/Text  │────▶│   Agent     │────▶│  MCP Server      │────▶│  3DGS Renderer   │
│ (Whisper/   │     │ (Claude/    │     │  (Node.js/       │     │  (Three.js/      │
│  Prompt)    │     │  TeleClaw)  │     │   Python)        │     │   WebGPU/HiGS/   │
│             │◀────│             │◀────│                  │◀────│   DDF-GS)        │
└─────────────┘     └─────────────┘     └──────────────────┘     └──────────────────┘
                        │                      │                       │
                        │  Tool calls          │  WebSocket/HTTP       │  WebGL/WebGPU/
                        │  (MCP protocol)       │  transport            │  HiGS/DDF-GS

MCP Tools Specification

Tool 1: import_scene

{
  "name": "import_scene",
  "description": "Load a 3DGS scene from PLY/SPLAT file or URL into the renderer",
  "inputSchema": {
    "type": "object",
    "properties": {
      "source": { "type": "string", "description": "File path or URL to .ply/.splat file" },
      "format": { "enum": ["ply", "splat", "spz", "ksplat"], "description": "File format" }
    },
    "required": ["source"]
  },
  "output": { "type": "object", "properties": { "scene_id": "string", "gaussian_count": "number", "bbox": "object" } }
}

Tool 2: set_camera

{
  "name": "set_camera",
  "description": "Set camera position, target, and field of view",
  "inputSchema": {
    "type": "object",
    "properties": {
      "position": { "type": "array", "items": {"type": "number"}, "description": "[x, y, z]" },
      "target": { "type": "array", "items": {"type": "number"}, "description": "[x, y, z] look-at point" },
      "fov": { "type": "number", "description": "Field of view in degrees" },
      "up": { "type": "array", "items": {"type": "number"}, "description": "[x, y, z] up vector" }
    },
    "required": ["position", "target"]
  }
}

Tool 3: modify_gaussians

{
  "name": "modify_gaussians",
  "description": "Modify properties of Gaussians by selection criteria",
  "inputSchema": {
    "type": "object",
    "properties": {
      "select": {
        "type": "object",
        "properties": {
          "ids": { "type": "array", "items": {"type": "integer"}, "description": "Specific Gaussian IDs" },
          "region": { "type": "object", "properties": {"center": "array", "radius": "number"}, "description": "Sphere selection" },
          "label": { "type": "string", "description": "Semantic label from segmentation" }
        }
      },
      "operations": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "property": { "enum": ["opacity", "color", "position", "scale", "rotation"] },
            "action": { "enum": ["set", "add", "multiply"] },
            "value": {}
          }
        }
      }
    },
    "required": ["select", "operations"]
  }
}

Tool 4: render_frame

{
  "name": "render_frame",
  "description": "Render current scene from current camera and return as image",
  "inputSchema": {
    "type": "object",
    "properties": {
      "width": { "type": "integer", "default": 1920 },
      "height": { "type": "integer", "default": 1080 },
      "format": { "enum": ["png", "jpeg", "webp"], "default": "png" },
      "background": { "type": "string", "default": "#000000" }
    }
  },
  "output": { "type": "object", "properties": { "image": "string (base64)", "render_time_ms": "number" } }
}

Tool 5: query_scene

{
  "name": "query_scene",
  "description": "Query scene information: statistics, geometry, semantics",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query_type": { "enum": ["stats", "bbox", "gaussian_at_point", "segmentation", "materials"] },
      "point": { "type": "array", "items": {"type": "number"}, "description": "[x, y, z] for point queries" }
    },
    "required": ["query_type"]
  }
}

Tool 6: cast_ray

{
  "name": "cast_ray",
  "description": "Cast a ray from origin in direction and return distance to first surface hit. Leverages DDF-GS (arXiv:2606.00817) neural field distilled from trained 3DGS.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "origin": { "type": "array", "items": {"type": "number"}, "description": "[x, y, z] ray origin" },
      "direction": { "type": "array", "items": {"type": "number"}, "description": "[x, y, z] ray direction (normalized)" }
    },
    "required": ["origin", "direction"]
  },
  "output": { "type": "object", "properties": { "distance": "number", "hit": "boolean", "normal": "array [x,y,z]" } }
}

Use cases: Shadow rendering, ambient occlusion, reflection rays, global illumination

Limitation: Requires DDF distillation step after 3DGS training (adds ~10 min for 52MB model)

Tool 7: simulate_physics

MCP Tool: simulate_physics — Invoke external physics engine (MPM/SPH/PBD) on 3DGS scene via RAF-style representation abstraction; parameters: object_ids, force, solver_type; returns: updated Gaussian positions/covariances

{
  "name": "simulate_physics",
  "description": "Invoke external physics engine (MPM/SPH/PBD) on 3DGS scene via RAF-style representation abstraction",
  "inputSchema": {
    "type": "object",
    "properties": {
      "object_ids": { "type": "array", "items": {"type": "integer"}, "description": "IDs of objects to simulate" },
      "force": { "type": "object", "properties": {"linear": "array", "angular": "array"}, "description": "Applied force/torque" },
      "solver_type": { "enum": ["mpm", "sph", "pbd", "rigid_body"], "description": "Physics solver backend" },
      "dt": { "type": "number", "description": "Time step in seconds", "default": 0.016 },
      "steps": { "type": "integer", "description": "Number of simulation steps", "default": 1 }
    },
    "required": ["object_ids", "solver_type"]
  },
  "output": { "type": "object", "properties": { "updated_positions": "array", "updated_covariances": "array", "energy": "number" } }
}

Use cases: Physics-driven scene editing, collapse/fall simulation, fluid interaction with Gaussian objects

Tool 8: query_4d_scene

MCP Tool: query_4d_scene — Query dynamic 3D scene at arbitrary (x,y,t) coordinates; returns: 3D position, flow vector, segmentation label; enables voice-driven temporal navigation

{
  "name": "query_4d_scene",
  "description": "Query dynamic 3D scene at arbitrary (x,y,t) coordinates; enables voice-driven temporal navigation via D4RT unified query mechanism",
  "inputSchema": {
    "type": "object",
    "properties": {
      "x": { "type": "number", "description": "X coordinate in scene space" },
      "y": { "type": "number", "description": "Y coordinate in scene space" },
      "t": { "type": "number", "description": "Time index in dynamic sequence" },
      "query_fields": { "type": "array", "items": {"enum": ["position_3d", "flow_vector", "segmentation_label", "depth"]}, "description": "Fields to return" }
    },
    "required": ["x", "y", "t"]
  },
  "output": { "type": "object", "properties": { "position_3d": "array [x,y,z]", "flow_vector": "array [dx,dy,dz]", "segmentation_label": "string", "depth": "number" } }
}

Use cases: "What was here at time t=5?", temporal object tracking, voice-driven time scrubbing

Tool 9: deform_elastic

MCP Tool: deform_elastic — Apply particle-skinned eigenmode deformation to 3DGS object; parameters: object_id, mode_indices, amplitudes; returns: deformed Gaussian positions

{
  "name": "deform_elastic",
  "description": "Apply particle-skinned eigenmode deformation to 3DGS object (FreeForm-style elastic deformation)",
  "inputSchema": {
    "type": "object",
    "properties": {
      "object_id": { "type": "integer", "description": "ID of object to deform" },
      "mode_indices": { "type": "array", "items": {"type": "integer"}, "description": "Eigenmode indices to activate" },
      "amplitudes": { "type": "array", "items": {"type": "number"}, "description": "Amplitude per eigenmode" },
      "interpolation": { "enum": ["linear", "smoothstep"], "description": "Interpolation method for deformation", "default": "smoothstep" }
    },
    "required": ["object_id", "mode_indices", "amplitudes"]
  },
  "output": { "type": "object", "properties": { "deformed_positions": "array", "eigenmode_energies": "array" } }
}

Use cases: Elastic soft-body deformation, eigenmode-based shape editing, physically plausible object bending

Tool 10: query_spatial_context

{
  "name": "query_spatial_context",
  "description": "Query spatial understanding of the current 3DGS scene using spatial intelligence models (Spatial-TTT/Holi-Spatial pipeline). Returns spatial relations, grounding, and scene graph.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "scene_id": { "type": "string", "description": "Scene identifier from import_scene" },
      "query": { "type": "string", "description": "Natural language spatial query about the scene" },
      "mode": { "enum": ["grounding", "relation", "measurement", "scene_graph"], "description": "Type of spatial query" }
    },
    "required": ["scene_id", "query", "mode"]
  },
  "output": { "type": "object", "properties": { "answer": "string", "spatial_data": "object", "confidence": "number" } }
}

Integrates Holi-Spatial (ICML 2026 Oral) data pipeline for automated spatial annotation and Spatial-TTT (ECCV 2026) for streaming spatial memory updates.

Tool 11: bayesian_density_control

{
  "name": "bayesian_density_control",
  "description": "Agent-controlled Bayesian nonparametric Gaussian density control. Uses DP-Splat (arXiv:2607.10912) Dirichlet-process prior to automatically determine optimal Gaussian count per region, eliminating manual density hyperparameter tuning.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "scene_id": { "type": "string" },
      "concentration": { "type": "number", "minimum": 0.1, "maximum": 10.0, "description": "DP concentration parameter α (higher = more Gaussians)" },
      "base_measure": { "enum": ["uniform", "saliency-weighted", "gradient-weighted"], "description": "Base measure for DP prior" },
      "max_iterations": { "type": "integer", "default": 50, "description": "Maximum MCMC iterations for posterior inference" }
    },
    "required": ["scene_id", "concentration"]
  },
  "output": { "type": "object", "properties": { "gaussian_count": "number", "regions_adjusted": "array", "elpd": "number" } }
}

Use cases: Auto-tune density for unknown scenes, eliminate manual clone/split threshold tuning, adapt density to scene complexity

Tool 12: moe_deform

{
  "name": "moe_deform",
  "description": "Apply Mixture-of-Experts dynamic deformation to selected Gaussians. Uses MoE-GS/MoDE (arXiv:2607.08250, TPAMI 2026) expert routing per motion pattern for physically plausible dynamic deformation.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "scene_id": { "type": "string" },
      "object_ids": { "type": "array", "items": {"type": "integer"}, "description": "IDs of objects to deform" },
      "target_motion": { "type": "string", "description": "Description of desired motion (e.g., 'wave left hand', 'open door')" },
      "num_experts": { "type": "integer", "default": 4, "description": "Number of deformation experts" },
      "temporal_range": { "type": "array", "items": {"type": "number"}, "description": "[start_time, end_time] for deformation" }
    },
    "required": ["scene_id", "object_ids", "target_motion"]
  },
  "output": { "type": "object", "properties": { "deformed_positions": "array", "expert_weights": "array", "motion_coherence": "number" } }
}

Use cases: Voice-driven character animation, dynamic scene editing with motion-specific expert routing, 4D content creation

Tool 13: surgical_tracking

{
  "name": "surgical_tracking",
  "description": "Track surgical instruments and reconstruct tissue map in real-time using Track2Map (arXiv:2607.08408, MICCAI 2026) surgical GS SLAM. Enables agent-assisted minimally invasive surgery guidance.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "scene_id": { "type": "string" },
      "tracking_mode": { "enum": ["instrument", "tissue", "both"], "description": "What to track" },
      "update_rate_hz": { "type": "number", "default": 30, "description": "Target tracking update rate" },
      "safety_margin_mm": { "type": "number", "default": 2.0, "description": "Safety margin for collision warning (mm)" }
    },
    "required": ["scene_id", "tracking_mode"]
  },
  "output": { "type": "object", "properties": { "instrument_poses": "array", "tissue_map_update": "boolean", "collision_warnings": "array", "tracking_accuracy_mm": "number" } }
}

Use cases: Surgical navigation, instrument tracking, tissue deformation monitoring, collision avoidance in surgery

Voice Intent Mapping

Voice Intent Example	Intent Type	MCP Tool Call
"What is to the left of the chair?"	Spatial grounding query	query_spatial_context (mode="grounding")
"How far is the table from the door?"	Spatial measurement	query_spatial_context (mode="measurement")

Voice-Driven Reconstruction Flow

User: "Show me the scene from above"
  │
  ▼
Whisper STT ──▶ Text: "Show me the scene from above"
  │
  ▼
Agent (Claude/TeleClaw) interprets:
  - Intent: Change camera to bird's-eye view
  - Parameters: position=[0, 10, 0], target=[0, 0, 0], up=[0, 0, -1]
  │
  ▼
MCP tool call: set_camera(position=[0, 10, 0], target=[0, 0, 0])
  │
  ▼
MCP tool call: render_frame(width=1920, height=1080)
  │
  ▼
Agent receives base64 image, verifies, reports to user

User: "Make the left wall transparent"
  │
  ▼
Agent:
  1. query_scene(query_type="segmentation") → find "left wall" label
  2. modify_gaussians(select={label: "left wall"}, operations=[{property: "opacity", action: "multiply", value: 0.2}])
  3. render_frame() → verify visual result

Implementation Stack

Component	Technology	Status
MCP Server	Node.js + @modelcontextprotocol/sdk	Prototype
3DGS Renderer	Three.js + gaussian-splat-3d / gsplat.js	Available
WebGPU backend	WebGPU + WGSL compute shaders	Experimental
HiGS backend	Dual-scale tile rasterization (arXiv:2606.00352)	Planned
DDF-GS backend	Neural distance field for ray queries (arXiv:2606.00817)	Planned
Transport	WebSocket (localhost)	Working
Voice STT	Whisper API / Web Speech API	Available
Agent integration	Claude Code / TeleClaw MCP client	Pending

Current Renderer Compatibility

Renderer	Format	WebGPU	MCP-Ready	Stars
gsplat.js	.ply/.splat	Yes	Needs adapter	—
GaussianSplats3D	.ply	WebGL	Needs adapter	—
viser/nerfstudio	.ply	WebGL	Partial	—
PlayCanvas	.ply	Yes	Needs adapter	—
brush (Rust/WebGPU)	.ply	Yes	Closest	4.3k
HiGS	.ply	Yes	Planned	—
DDF-GS	.ply + .ddf	Yes	Planned	—

DDF-GS Distillation Pipeline

1. Train 3DGS scene normally
2. Distill into Directed Distance Function (DDF) neural field
   • Input: trained 3DGS model (.ply)
   • Output: DDF model (~52MB, size independent of Gaussian count)
   • Training time: ~10 minutes
   • Quality: shadow at 30.3 dB PSNR, AO at 21.3 dB PSNR
3. DDF enables: shadow maps, AO, reflections, global illumination

HiGS Hierarchical Rendering Integration

• HiGS (arXiv:2606.00352) achieves 15.8x rendering speedup via dual-scale tile architecture
• MCP integration: render_frame() can leverage HiGS backend for real-time rendering
• Architecture: Agent → MCP → HiGS Renderer (macro-tile partitioning + micro-tile rasterization)
• Performance target: 950+ FPS on NVIDIA GPU for interactive scene exploration

Known Limitations

1. Latency: Large scenes (>1M Gaussians) require progressive loading; MCP render_frame may take 100-500ms
2. Selection precision: Sphere/label-based Gaussian selection may miss thin structures; need ray-picking
3. State management: MCP server must maintain scene state across tool calls; no built-in undo
4. GPU memory: WebGL/WebGPU shares GPU memory with browser; cannot load >2GB scenes on most devices

Roadmap

• v0.1: MCP tool specification (this document)
• v0.2: Node.js MCP server + gsplat.js adapter + DDF-GS cast_ray tool + HiGS backend
• v0.3: Voice-to-MCP pipeline (Whisper → Agent → MCP → render) + simulate_physics (RAF) + query_4d_scene (D4RT) + deform_elastic (FreeForm)
• v0.4: Semantic querying (integrate OP2GS/Gaga for label-based selection)
• v0.5: Real-time streaming (WebSocket-based progressive rendering)
• v0.6: DDF-GS distillation integration (shadow/AO/reflection rendering)
• v0.7: HiGS hierarchical rendering backend (950+ FPS target)
• v0.8: Bayesian density control (DP-Splat) + MoE deformation (MoE-GS/MoDE) + Surgical tracking (Track2Map)

Rules

1. Never modify original PLY files: All operations are in-memory only; export requires explicit user command
2. Validate before render: Always verify camera parameters and Gaussian bounds before rendering
3. Respect GPU limits: Check available VRAM before loading large scenes; provide downsampling option
4. Report rendering time: Always include render_time_ms in render_frame output for performance monitoring
5. Safety gate: Operations affecting >10% of Gaussians require explicit user confirmation

Part of Awesome-Gaussian-Skills

Related Skills

• 3dgs-engineering-guide — Production deployment (use for end-to-end deployment workflows)
• 3dgs-spatial-agent — Spatial intelligence agent (use for agent-driven 3D interaction)
• 3dgs-articulated-reasoner — Articulated object reasoning (use for interactive object manipulation)
• 3dgs-visualizer — Visualization (use for rendering pipeline output quality assessment)

Guardrail: Do Not Apply From Memory

Do NOT try to apply the logic, method data, bug patterns, or technical details described in this skill from memory. Always read the SKILL.md and referenced files from disk before producing any output. The knowledge base is updated frequently; stale memory may produce outdated, inaccurate, or fabricated results.

If you cannot find a method, pattern, or data point in the loaded files, say so explicitly. Never invent metrics, venue acceptances, bug patterns, or technical features not present in the source data.