partcrafter-3d-mesh-generation

name	partcrafter-3d-mesh-generation
title	PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.05573
keywords	["3d-generation","diffusion-models","part-aware","mesh-synthesis","generative-models"]
description	Generates semantically-meaningful 3D parts from single images via compositional diffusion transformers with part-level identity and local-global attention.

PartCrafter: Structured 3D Mesh Generation

Core Concept

Generating 3D meshes with explicit semantic structure is challenging because parts must be geometrically distinct yet semantically coherent. PartCrafter departs from traditional two-stage approaches (segment then reconstruct) by using a unified generative model that jointly synthesizes multiple part-aware 3D meshes directly from RGB images. The key innovation is a compositional diffusion transformer with disentangled latent tokens per part, local-global attention for intra-part and inter-part reasoning, and identity-aware permutation-invariant design. This enables end-to-end generation of complex multi-part objects and scenes without pre-segmented inputs.

Architecture Overview

• Unified Architecture: Single end-to-end model for part-aware 3D generation from images
• Disentangled Part Tokens: Each semantic part represented by K learnable identity-embedding tokens
• Compositional Latent Space: Multi-part latents composed before diffusion process
• Local-Global Attention: Dual attention mechanism handling within-part (local) and between-part (global) dependencies
• Identity-Aware Design: Permutation-invariant to part order, generalizing to variable part counts
• Rectified Flow Matching: Improved training stability over standard diffusion
• Occluded Part Generation: Capable of hallucinating non-visible parts

Implementation

The following code demonstrates the compositional architecture and attention mechanism:

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Optional
import math

class PartIdentityEmbedding(nn.Module):
    """
    Learnable identity embeddings for individual parts.
    """
    def __init__(self, part_id: int, latent_dim: int = 768, num_tokens: int = 8):
        super().__init__()
        self.part_id = part_id
        self.num_tokens = num_tokens

        # Identity tokens: learnable embeddings unique to each part
        self.identity_tokens = nn.Parameter(
            torch.randn(num_tokens, latent_dim) / (latent_dim ** 0.5)
        )

    def forward(self, batch_size: int, device: torch.device) -> torch.Tensor:
        """
        Generate identity embeddings for a batch.
        Returns: (batch_size, num_tokens, latent_dim)
        """
        # Repeat identity tokens across batch
        embeddings = self.identity_tokens.unsqueeze(0).expand(batch_size, -1, -1)
        return embeddings.to(device)


class LocalGlobalAttention(nn.Module):
    """
    Dual attention: local (intra-part) and global (inter-part).
    """
     ():
        ().__init__()
        .latent_dim = latent_dim
        .num_heads = num_heads
        .head_dim = latent_dim // num_heads

        
        .local_q = nn.Linear(latent_dim, latent_dim)
        .local_k = nn.Linear(latent_dim, latent_dim)
        .local_v = nn.Linear(latent_dim, latent_dim)

        
        .global_q = nn.Linear(latent_dim, latent_dim)
        .global_k = nn.Linear(latent_dim, latent_dim)
        .global_v = nn.Linear(latent_dim, latent_dim)

        .out_proj = nn.Linear(latent_dim * , latent_dim)

     () -> [torch.Tensor]:
        
        num_parts = (part_latents)
        updated_latents = []

         part_idx  (num_parts):
            part = part_latents[part_idx]  

            
            Q_local = .local_q(part)
            K_local = .local_k(part)
            V_local = .local_v(part)

            
            Q_local = Q_local.view(-, .num_heads, .head_dim)
            K_local = K_local.view(-, .num_heads, .head_dim)
            V_local = V_local.view(-, .num_heads, .head_dim)

            local_scores = torch.matmul(Q_local, K_local.transpose(-, -)) / math.sqrt(.head_dim)
            local_attn = F.softmax(local_scores, dim=-)
            local_out = torch.matmul(local_attn, V_local)
            local_out = local_out.view(-, part.shape[], .latent_dim)

            
            
            other_parts = torch.cat([part_latents[j]  j  (num_parts)  j != part_idx], dim=)

            Q_global = .global_q(part)
            K_global = .global_k(other_parts)
            V_global = .global_v(other_parts)

            Q_global = Q_global.view(-, .num_heads, .head_dim)
            K_global = K_global.view(-, .num_heads, .head_dim)
            V_global = V_global.view(-, .num_heads, .head_dim)

            global_scores = torch.matmul(Q_global, K_global.transpose(-, -)) / math.sqrt(.head_dim)
            global_attn = F.softmax(global_scores, dim=-)
            global_out = torch.matmul(global_attn, V_global)
            global_out = global_out.view(-, part.shape[], .latent_dim)

            
            combined = torch.cat([local_out, global_out], dim=-)
            output = .out_proj(combined)

            updated_latents.append(output)

         updated_latents


 (nn.Module):
    
     ():
        ().__init__()
        .latent_dim = latent_dim
        .max_parts = max_parts
        .tokens_per_part = tokens_per_part
        .num_diffusion_steps = num_diffusion_steps

        
        .part_embeddings = nn.ModuleList([
            PartIdentityEmbedding(i, latent_dim, tokens_per_part)
             i  (max_parts)
        ])

        
        .image_encoder = ._build_image_encoder()

        
        .local_global_attn = nn.ModuleList([
            LocalGlobalAttention(latent_dim)  _  ()  
        ])

        
        .time_embedding = nn.Sequential(
            nn.Linear(, latent_dim),
            nn.SiLU(),
            nn.Linear(latent_dim, latent_dim)
        )

        
        .mesh_decoder = nn.Sequential(
            nn.Linear(latent_dim * tokens_per_part, ),
            nn.ReLU(),
            nn.Linear(, ),
            nn.ReLU(),
            nn.Linear(, ),  
        )

     () -> nn.Module:
        
         nn.Sequential(
            nn.Conv2d(, , kernel_size=, stride=),
            nn.ReLU(),
            nn.Conv2d(, , kernel_size=, stride=),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(),
            nn.Flatten(),
            nn.Linear(, .latent_dim)
        )

     () -> [[torch.Tensor], torch.Tensor]:
        
        batch_size = image.shape[]

        
        image_features = .image_encoder(image)  

        
        part_latents = []
         part_idx  ((num_parts, .max_parts)):
            part_emb = .part_embeddings[part_idx](batch_size, image.device)
            
            part_emb = part_emb + image_features.unsqueeze() * 
            part_latents.append(part_emb)

        
        t_emb = .time_embedding(torch.tensor([[timestep]], dtype=torch.float32,
                                                  device=image.device))
        t_emb = t_emb.unsqueeze().expand(batch_size, -)

        
         attn_block  .local_global_attn:
            
             i  ((part_latents)):
                part_latents[i] = part_latents[i] + t_emb.unsqueeze() * 

            
            part_latents = attn_block(part_latents)

        
        composed_latent = torch.cat(part_latents, dim=)  
        composed_latent = composed_latent.mean(dim=)  

         part_latents, composed_latent

     () -> [torch.Tensor]:
        
        meshes = []

         part_latent  part_latents:
            
            flattened = part_latent.view(part_latent.shape[], -)
            vertices = .mesh_decoder(flattened)
            
            vertices = vertices.view(vertices.shape[], -, )
            meshes.append(vertices)

         meshes


 :
    
     ():
        .num_steps = num_steps

     () -> torch.Tensor:
        
         torch.rand(batch_size, device=device)

     () -> torch.Tensor:
        
        
        latent_t = ( - timestep.unsqueeze()) * latent_start + timestep.unsqueeze() * latent_end

        
        target = latent_end - latent_start

        
        loss = F.mse_loss(model_output, target)

         loss


 :
    
     ():
        .model = model
        .optimizer = torch.optim.Adam(model.parameters(), lr=)
        .flow_matching = RectifiedFlowMatching()

     () -> :
        
        batch_size = image.shape[]

        
        timestep = .flow_matching.sample_noise_schedule(batch_size, image.device)

        
        part_latents, composed = .model(image, num_parts, timestep[].item())

        
        noise_latents = [torch.randn_like(pl)  pl  part_latents]

        
        loss = 
         i, (start, end)  ((noise_latents, part_latents)):
            loss += .flow_matching.compute_flow_matching_loss(
                start, end, end, timestep.unsqueeze()
            )

        loss = loss / (part_latents)

        
        .optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(.model.parameters(), )
        .optimizer.step()

         (loss)

Practical Guidance

Part Count Flexibility: The model handles variable part counts (1-8 in standard setup). For objects with more parts, extend max_parts in initialization or use hierarchical decomposition.

Identity Embedding Dimension: Use num_tokens=8-16 per part. Larger values increase expressivity but require more memory; smaller values constrain geometric diversity.

Local-Global Balance: The 0.1 weighting for global context in local attention (line 153) controls cross-part influence. Increase to 0.2 if parts should be more interdependent; decrease to 0.05 for more independent parts.

Training Data: The approach requires part-level annotations. Use the provided dataset curation pipeline on Objaverse/ShapeNet to mine ~50K annotated objects automatically.

Occluded Part Hallucination: The compositional design naturally predicts unseen parts. Validate on occlusion benchmarks during training to ensure realistic completion.

Inference Time: Generation of 4-part objects takes ~34 seconds with standard setup. Optimize by reducing num_diffusion_steps to 10 for 2-3x speedup with minor quality loss.

Mesh Quality Metrics: Evaluate using Chamfer Distance (geometry) and F-Score (occupancy). The independence measure (IoU between part predictions) is critical for part-aware evaluation.

Reference

PartCrafter achieves strong results on part-aware 3D generation:

• Single-image generation: Multiple semantically-meaningful parts from RGB
• Multi-object scenes: Outperforms two-stage approaches (HoloPart, MIDI)
• Occluded parts: Generates geometrically plausible unseen components
• Efficiency: 34 seconds per 4-part object on single GPU

The end-to-end approach is particularly valuable for interactive 3D modeling, content creation pipelines, and applications requiring structured decomposition of generated geometry.