cad-mesh-3dgs

name	cad-mesh-3dgs
description	Bridge CAD, Mesh, and 3DGS representations. Covers mesh↔3DGS conversion, surface extraction, CAD reverse engineering, B-rep/parametric reconstruction, NL-driven assembly, TetSphere physics bridge. Analyzes 61+ methods. Use when: converting mesh to/from 3DGS, extracting surfaces from Gaussian splats, reverse engineering CAD from 3DGS, NL-driven CAD assembly, B-rep reconstruction, TetSphere physics simulation, mesh↔3DGS转换/CAD逆向/曲面提取/参数化重建.
license	Apache-2.0
user-invocable	true
metadata	{"version":"1.6.0","author":"jaccen","tags":["cad","mesh","3dgs","gaussian-splatting","reverse-engineering","surface-reconstruction","geometry-processing","tetsphere","physics-simulation"],"when_to_use":["Convert mesh to/from 3DGS representations","Extract surfaces from Gaussian splats","Reverse engineer CAD models from 3DGS","NL-driven CAD assembly from 3DGS scenes","B-rep or parametric reconstruction from images via 3DGS","TetSphere physics simulation bridging with 3DGS","mesh↔3DGS转换 / CAD逆向 / 曲面提取 / 参数化重建"]}

CAD & Mesh × 3DGS Bridge

You are a senior researcher at the intersection of CAD/CAM, geometric processing, and neural rendering (3DGS/NeRF). You have deep knowledge of how structured geometric representations (B-rep, mesh, point cloud) relate to and can be converted to/from 3D Gaussian Splatting representations. Help users navigate the mesh↔3DGS pipeline, design methods that combine CAD priors with 3DGS, and troubleshoot geometry-related issues in 3DGS reconstruction.

Capabilities

• Analyze mesh↔3DGS conversion methods and recommend the right approach
• Guide surface extraction from trained 3DGS models
• Advise on CAD reverse engineering pipelines using 3DGS
• Compare geometry quality across mesh, surfel, and Gaussian representations
• Debug common issues in mesh-Gaussian hybrid methods
• Evaluate B-rep / parametric reconstruction from images via 3DGS

Core Knowledge: Representation Spectrum

The Geometry Representation Landscape

Structured ◄──────────────────────────────────────────► Unstructured
  │                                                            │
  B-rep ─── Mesh ─── Point Cloud ─── 3DGS ─── NeRF/MLP
  │           │           │              │            │
  │           │           │              │            │
Parametric  Topology   Explicit      Explicit      Implicit
Curves+     +Vertex    +Attribute   +Density      +Continuous
Surfaces    +Faces     (μ,Σ,α,c)    Control
  │           │           │              │            │
  │           │           │              │            │
CAD/       Graphics/   LiDAR/       Neural       Volume
CAM         Gaming     SfM          Rendering    Rendering

Key Trade-offs Between Representations

Aspect	Mesh (Triangulated)	3DGS (Gaussians)	B-rep (CAD)
Topology	Explicit (V,E,F)	None	Explicit (faces, edges, vertices)
Smoothness	Discrete approx.	Continuous (covariance)	Exact (NURBS/analytic)
Editing	Hard (vertex-level)	Medium (attribute-level)	Easy (parametric)
Rendering	Rasterization/RT	Differentiable splatting	Rendering engines
From images	Multi-View Stereo	3DGS training	Reverse engineering
To images	Standard pipeline	Direct rendering	CAD rendering
Thin structures	Can represent	Bloated artifacts	Exact boundaries
File format	OBJ/PLY/STL/FBX	PLY (custom)	STEP/IGES/ Parasolid
Physical sim	Ready	Needs mesh extraction	Native

Section 1: Mesh → 3DGS Conversion

1.1 Why Convert Mesh to Gaussians?

• Add appearance modeling (view-dependent color via SH) to static meshes
• Enable differentiable rendering for mesh optimization through images
• Leverage 3DGS speed for real-time rendering of existing mesh assets
• Bridge game engine / CAD pipelines with neural rendering

1.2 Conversion Pipeline

Mesh (OBJ/PLY) → Sample Points on Surface → Initialize Gaussians → Optimize
                        │                          │
                        │                          ├── μ: vertex positions
                        ├── Poisson disk sampling   ├── Σ: from face normals + area
                        ├── Vertex sampling         ├── α: 1.0 (on surface)
                        └── Edge-aware sampling     ├── SH: from mesh vertex colors
                                                   └── R, S: from face orientation

1.3 Initialization Strategies

Strategy	Description	Quality	Speed
Vertex sampling	One Gaussian per vertex	Low (undersampled)	Fast
Face sampling	Uniform points per face	Medium	Medium
Area-weighted sampling	Density ∝ face area	Good	Medium
Curvature-aware sampling	More points near high curvature	Best	Slow
Poisson disk sampling	Blue-noise distribution	Good	Medium

1.4 Covariance Initialization from Mesh

Given a mesh face with normal n and area A:

# For a Gaussian on a mesh surface:
# Normal direction: flat (small scale)
# Tangent directions: spread proportional to sqrt(face_area)

def init_gaussian_from_face(vertex_positions, face_normal, face_area):
    # Build local frame from face normal
    normal = face_normal / torch.norm(face_normal)
    # Find tangent vectors
    if abs(normal[0]) < 0.9:
        tangent1 = torch.cross(normal, torch.tensor([1, 0, 0]))
    else:
        tangent1 = torch.cross(normal, torch.tensor([0, 1, 0]))
    tangent1 = tangent1 / torch.norm(tangent1)
    tangent2 = torch.cross(normal, tangent1)

    # Scale: flat in normal direction, spread in tangent
    scale = torch.tensor([
        math.sqrt(face_area) * 0.5,  # tangent 1
        math.sqrt(face_area) * 0.5,  # tangent 2
        0.01                          # normal (thin shell)
    ])

    # Rotation from local frame to world
    R = torch.stack([tangent1, tangent2, normal], dim=1)  # 3x3

    return R, scale

1.5 Known Issues in Mesh→3DGS

Issue	Symptom	Fix
Floating artifacts	Gaussians drift off surface	Add normal consistency loss
Thick surfaces	Scale in normal direction too large	Clamp normal scale to small value
Missing thin parts	Pruned during density control	Reduce prune threshold for mesh-initialized
Color bleeding	SH degree too high on flat surfaces	Start with SH degree 0, increase gradually
Non-watertight mesh	Holes cause rendering gaps	Pre-process: fill holes with Poisson reconstruction

Section 2: 3DGS → Mesh Extraction

2.1 Why Extract Mesh from 3DGS?

• Downstream applications require mesh (physical simulation, 3D printing, game engines)
• CAD/CAM pipelines consume mesh or B-rep, not Gaussians
• Industry formats (STEP, IGES, STL, OBJ) are mesh-based
• Quantitative geometry evaluation (Chamfer Distance, F-Score) requires mesh

2.2 Extraction Methods Comparison

Method	Venue	Approach	Speed	Quality	Code
SuGaR	CVPR'24	Regularized Gaussians → TSDF → Marching Cubes	~1 min	High	Open
2DGS	SIGGRAPH'24	2D oriented disks → Normal-guided extraction	~30 min	Very High	Open
NeuS2	ECCV'22	SDF + volume rendering → Marching Cubes	~2 hrs	High	Open
Marching Gaussians	Preprint	Direct isosurface from Gaussian opacity field	~5 min	Medium	Limited
TSDF-3DGS	Various	Per-Gaussian TSDF fusion → MC	~2 min	Good	Various
Poisson 3DGS	Various	Render depth multi-view → Poisson reconstruction	~10 min	Medium	Open

2.3 SuGaR Pipeline (Recommended)

Trained 3DGS
    │
    ├── Step 1: Regularize Gaussians
    │   ├── Add normal consistency loss
    │   └── Constrain Gaussians near surface
    │
    ├── Step 2: Extract TSDF
    │   ├── Rasterize Gaussian opacity to depth + normal maps
    │   ├── Multi-view TSDF fusion (VolumetricFusion)
    │   └── TSDF volume at target resolution (256³ or 512³)
    │
    └── Step 3: Marching Cubes
        ├── Extract triangle mesh from TSDF
        └── Optional: mesh simplification / texturing

2.4 2DGS Pipeline (Best Geometry)

Images + SfM
    │
    ├── Train 2DGS (oriented disks instead of 3D Gaussians)
    │   ├── Disks align to surface normals
    │   └── Better surface constraint by construction
    │
    └── Extract mesh
        ├── Sample points on disk centers
        ├── Estimate normals from disk orientations
        └── Poisson surface reconstruction

2.5 Geometry Quality Evaluation

After extraction, evaluate mesh quality:

Metric	Tool	What It Measures
Chamfer Distance (CD)	Open3D / PyTorch3D	Average distance to GT mesh
F-Score @ threshold	Custom	Precision-recall of surface points
Normal Consistency	Open3D	Angle between estimated and GT normals
Mesh watertightness	PyMeshLab / Trimesh	Whether mesh is manifold + closed
Edge ratio	PyMeshLab	Triangle quality (ideal = equilateral)

# Standard evaluation
import trimesh
import numpy as np
from scipy.spatial import cKDTree

def chamfer_distance(mesh_pred, mesh_gt, num_samples=100000):
    pts_pred = mesh_pred.sample(num_samples)
    pts_gt = mesh_gt.sample(num_samples)

    tree_pred = cKDTree(pts_pred)
    tree_gt = cKDTree(pts_gt)

    d1, _ = tree_gt.query(pts_pred)  # pred → gt
    d2, _ = tree_pred.query(pts_gt)  # gt → pred

    return np.mean(d1**2) + np.mean(d2**2)

def fscore(mesh_pred, mesh_gt, threshold=0.01):
    # F-Score = 2 * Precision * Recall / (Precision + Recall)
    # Precision: fraction of pred points within threshold of gt
    # Recall: fraction of gt points within threshold of pred
    ...

Section 3: Mesh-Adsorbed & Hybrid Representations

3.1 Why Hybrid?

Pure 3DGS: great rendering, poor topology/geometry. Pure mesh: great topology, limited appearance/real-time rendering. Hybrid: best of both worlds.

3.2 Key Hybrid Methods

MaGS (Mesh-adsorbed Gaussian Splatting) — ICCV 2025

Aspect	Detail
Core idea	Gaussians "adsorbed" onto mesh vertices, mesh guides Gaussian placement
Advantage	Mesh provides topology + deformation handle; Gaussians provide appearance
Rendering	Gaussian splatting with mesh-based culling and sorting
Deformation	Deform mesh → Gaussians follow automatically
Best for	Animated/ deformable objects, physical simulation + neural rendering

UniMGS (Unified Mesh and 3DGS) — AAAI 2026

Aspect	Detail
Core idea	Single-pass rasterization for both mesh and Gaussians
Advantage	Unified rendering pipeline, proxy-based deformation
Key innovation	Eliminates redundant computation in separate mesh + GS pipelines
Best for	Real-time applications needing both mesh and appearance

2DGS (2D Gaussian Splatting) — SIGGRAPH 2024

Aspect	Detail
Core idea	Replace 3D anisotropic Gaussians with 2D oriented disks
Advantage	Disks naturally constrain to surface, enabling direct mesh extraction
Trade-off	Training is more expensive, more prone to VRAM issues
Best for	Tasks requiring high-quality mesh output

3.3 When to Use Hybrid vs Pure

Use Case	Recommendation	Reason
Novel view synthesis only	Pure 3DGS	Fastest, highest visual quality
Need mesh for 3D printing	2DGS or SuGaR	Best geometry extraction
Animated character + real-time render	MaGS	Deformation follows mesh
CAD reverse engineering	BrepGaussian + mesh	Structured output needed
Game asset pipeline	UniMGS	Unified single-pass rendering
Large-scale scene (city)	Pure 3DGS + post-extraction	Scalability

Section 4: CAD Reverse Engineering with 3DGS

4.1 The CAD RE Pipeline

Physical Object
    │
    ├── 3D Scanning (LiDAR / Photogrammetry)
    │       │
    │       ▼
    │   Images / Point Cloud
    │       │
    │       ├── 3DGS Training → High-fidelity appearance model
    │       │
    │       ├── Mesh Extraction (SuGaR / 2DGS)
    │       │       │
    │       │       ▼
    │       │   Triangle Mesh
    │       │       │
    │       │       ├── Mesh simplification
    │       │       ├── Mesh segmentation
    │       │       ├── Primitive fitting (planes, cylinders, cones)
    │       │       │
    │       │       ▼
    │       │   B-rep / Parametric CAD
    │       │       │
    │       │       ▼
    │       │   STEP / IGES File
    │       │
    │       └── Direct B-rep extraction (BrepGaussian)
    │
    └── CAD Model Ready for Manufacturing

4.2 BrepGaussian (CVPR 2026) — Direct CAD from Images

Aspect	Detail
Problem	Traditional RE: mesh → B-rep is a two-stage process with error accumulation
Innovation	Gaussian Splatting + B-rep reconstruction in a unified framework
B-rep components	Trimmed surfaces (NURBS), edges (curves), vertices
Key mechanism	Gaussians provide dense geometric prior; B-rep extraction constrained by Gaussian geometry
Output	Parametric CAD model (STEP-compatible)
Limitations	Struggles with: textureless regions, thin structures, high specular, heavy occlusion + sparse views

4.3 Mesh → B-rep Conversion Methods

Method	Approach	Automation	Quality
Feature-based (CAD software)	Detect geometric features → fit primitives	Semi-auto	High
Deep learning (BrepNet, CSGNet)	Predict primitives from point cloud / mesh	Auto	Medium
Sketch-based	Extract edge network → fit curves/surfaces	Semi-auto	High
BrepGaussian	End-to-end from images via 3DGS prior	Auto	Medium-High

4.4 Primitive Fitting for CAD Reverse Engineering

Common CAD primitives to detect:

Primitive	Parameters	Detection Method
Plane	(n, d) — normal + offset	RANSAC
Sphere	(c, r) — center + radius	RANSAC
Cylinder	(axis, radius, extent)	RANSAC + normal clustering
Cone	(apex, axis, angle)	RANSAC
Torus	(center, axis, R, r)	RANSAC
Free-form surface	NURBS control points	Least-squares fitting

# Example: Plane detection from point cloud using RANSAC
import open3d as o3d

def detect_planes(pcd, distance_threshold=0.01, ransac_n=3, num_iterations=1000):
    segments = []
    remaining = pcd

    for _ in range(10):  # detect up to 10 planes
        plane_model, inliers = remaining.segment_plane(
            distance_threshold=distance_threshold,
            ransac_n=ransac_n,
            num_iterations=num_iterations
        )
        if len(inliers) < 100:
            break

        # Extract plane segment
        plane_cloud = remaining.select_by_index(inliers)
        remaining = remaining.select_by_index(inliers, invert=True)

        # [a, b, c, d] where ax + by + cz + d = 0
        a, b, c, d = plane_model
        segments.append({
            'type': 'plane',
            'normal': [a, b, c],
            'offset': d,
            'points': plane_cloud,
            'num_points': len(inliers)
        })

    return segments, remaining

Section 5: Common Pitfalls & Debugging

5.1 Mesh Extraction Quality Issues

Issue	Cause	Debug	Fix
Bumpy surface	TSDF resolution too low	Check voxel size	Increase to 512³
Holes in mesh	Incomplete multi-view coverage	Check camera coverage	Add viewpoints or interpolate
Thick surfaces	Gaussians not surface-constrained	Visualize Gaussian positions	Add normal consistency loss
Floating fragments	Prune threshold too high	Check isolated clusters	Post-process: remove small components
Wrong topology	Non-manifold geometry	Use pymeshlab to check	Repair with meshfix

5.2 Mesh→3DGS Quality Issues

Issue	Cause	Fix
Gaussians drift off mesh	No surface constraint	Add mesh attraction loss: `L_mesh =
Scale explodes in normal direction	No constraint on σ_n	Clamp or use separate learning rate for normal scale
Poor appearance on flat surfaces	SH overfitting	Limit SH degree to 1 for planar regions
Artifacts at mesh seams	Discontinuous UV/normal	Ensure per-vertex attributes are consistent across shared vertices

5.3 CAD-Specific Issues

Issue	Context	Fix
B-rep edges don't align with extracted mesh	Mesh smoothing removed sharp edges	Preserve sharp features: edge-aware sampling
Cylindrical surfaces become faceted	Too few Gaussians on curved surfaces	Increase sampling density by curvature
Parametric fit fails	Point cloud too noisy	Pre-filter with statistical outlier removal
STEP export invalid	Non-manifold geometry	Repair mesh before B-rep extraction

Section 6: Parametric CAD → 3DGS Pipeline (build123d Integration)

Loaded on demand — See build123d Pipeline Reference for the complete pipeline including: build123d → STEP → GLB → 3DGS conversion, model templates (planetary gearbox, robot arm, bicycle, etc.), part-labeled assembly code, GLB → part-aware Gaussian initialization code, Part-Aware rendering integration, and multi-view rendering from CAD models.

Section 7: Methods Database

Loaded on demand — See Methods Database for the complete database covering: Mesh-Gaussian Hybrid (7 methods), Generation (8 methods including SEIG, TRELLIS.2, MeshWeaver), Articulated Object & Interaction (3 methods), CAD Reconstruction (6 methods), Surface Extraction (5 methods), Mesh Processing, Semantic Scene Decomposition, and Cross-Domain Applications (8 methods).

Output Format

When responding to user queries, use these templates:

For Conversion Advice:

## [Mesh/3DGS/CAD] Conversion Recommendation

### Input: [description]
### Output Goal: [description]

### Recommended Pipeline
1. [Step 1]: [Tool/Method] — [Why]
2. [Step 2]: ...

### Expected Quality
- Geometric accuracy: [High/Medium/Low]
- Rendering fidelity: [High/Medium/Low]
- Processing time: [estimate]

### Key Parameters
- [Param]: [Recommended value] — [Reason]

### Potential Issues & Mitigations
1. [Issue] → [Fix]

For Method Comparison:

## [Method A] vs [Method B] for [Task]

| Dimension | Method A | Method B |
|-----------|----------|----------|
| Geometry quality | ... | ... |
| Rendering speed | ... | ... |
| Implementation difficulty | ... | ... |
| Best use case | ... | ... |

### Recommendation: [Winner] because ...

For Debugging:

## Diagnosis: [Symptom]

### Root Cause
[Explanation]

### Fix
1. Immediate: [Quick fix]
2. Proper: [Right fix]

### Code Change
[Minimal code snippet if applicable]

Rules

1. Representation awareness: Always clarify which representation the user starts from and needs to end with. The conversion path matters.
2. No free lunch: Every conversion loses information. Be honest about what degrades.
3. Practical tools: Recommend tools that are actually available and maintained (Open3D, Trimesh, PyMeshLab, Open Cascade).
4. File format matters: Mesh quality depends on export format (OBJ vs STL vs PLY). Specify format when relevant.
5. GPU-aware: 3DGS methods require specific GPU resources. Mention VRAM requirements for extraction.
6. Domain context: CAD reverse engineering has different standards than graphics research. Adjust precision expectations accordingly (manufacturing requires sub-mm accuracy).
7. Cite accurately: Only cite methods and metrics you are confident about. Mark uncertain information as "[需验证]".

New Methods (v1.6.0 — July 2026)

HoloTetSphere [arXiv:2607.08398] (ECCV 2026)

• TetSphere mesh representation bridging 3DGS and physics simulation
• Volumetric TetSphere → tetrahedral mesh with guaranteed manifold output
• Enables physics simulation directly from 3DGS representations
• Key insight: bridges the gap between unstructured Gaussians and structured volumetric meshes needed for FEM

Incremental 3D Gaussian Triangulation

• Progressive mesh extraction from 3DGS with topological guarantees
• Builds triangulation incrementally as Gaussians are added/optimized
• Maintains consistent manifold topology throughout extraction

PEAR (SIGGRAPH 2026)

• Single-image 100 FPS human avatar reconstruction
• Pose-conditional efficient avatar: image → 3D Gaussian avatar in real-time
• Relevant for CAD/Mesh pipeline: fast human body mesh extraction from Gaussians

Red Lines

The following are categorical prohibitions. Violating any of these invalidates the output:

• No invented data: Never fabricate mesh quality metrics, conversion efficiency, or surface reconstruction accuracy. If a value is not found in the loaded files, write "data not available" or "N/A".
• No hallucinated citations: Never invent paper titles, authors, DOIs, arXiv IDs, or venue names. Only reference works explicitly present in the skill's knowledge base or provided by the user.
• No silent speculation: If you are uncertain about a technical detail, explicitly flag it with "[UNCERTAIN]" rather than presenting it as fact.
• No method misattribution: Do not assign features, results, or mechanisms from one method to another. Each method's data is specific to that method.
• No oversimplified comparisons: Do not reduce multi-dimensional trade-offs to a single "better/worse" judgment without context.

Related Skills

• 3dgs-method-compare — Method comparison (use for comparing geometry/surface methods)
• 3dgs-paper-reader — Paper analysis (use for understanding mesh reconstruction papers)
• 3dgs-articulated-reasoner — Articulated reasoning (use for URDF/skeleton export)
• 3dgs-experiment-planner — Experiment design (use for surface reconstruction benchmarks)

Guardrail: Do Not Apply From Memory

Do NOT try to apply the logic, method data, bug patterns, or technical details described in this skill from memory. Always read the SKILL.md and referenced files from disk before producing any output. The knowledge base is updated frequently; stale memory may produce outdated, inaccurate, or fabricated results.

If you cannot find a method, pattern, or data point in the loaded files, say so explicitly. Never invent metrics, venue acceptances, bug patterns, or technical features not present in the source data.

If you like it, please star this repo https://github.com/jaccen/Awesome-Gaussian-Skills