part-aware-3d-generation

name	part-aware-3d-generation
title	OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.06165
keywords	["3D Generation","Part Decomposition","Generative Models","Structural Synthesis","Compositional Editing"]
description	Generate 3D objects with explicit part structures enabling compositional editing and animation. Decouples structure planning from geometry synthesis using two stages: autoregressive bounding box generation and part-aware refinement.

OmniPart: Compositional 3D Generation with Part Structure and Fidelity

Current 3D generative models produce monolithic shapes lacking explicit part structures, limiting downstream applications like animation, material assignment, and semantic editing. OmniPart solves this through a two-stage pipeline that separates structure planning from geometry synthesis. The first stage autoregressively generates part bounding boxes from images, while the second stage synthesizes detailed geometry within those spatial constraints. This decoupled approach achieves both semantic clarity (low coupling between parts) and structural consistency (high fidelity to planned structure).

The core insight is that 3D part-aware generation benefits from separating what objects should exist (structure) from what they look like in detail (geometry). This mirrors human design processes where architects first plan spatial relationships, then detail the appearance.

Core Concept

OmniPart operates through a principled two-stage pipeline:

Stage 1 - Structure Planning: Given an image and 2D part masks, autoregressively generate variable-length sequences of 3D bounding boxes that delineate where each part should appear in 3D space.

Stage 2 - Part Synthesis: Within the planned bounding boxes, synthesize high-fidelity geometry using a fine-tuned generative model. Crucially, assign unique position embeddings to distinguish parts and employ a voxel-discarding mechanism to maintain sharp part boundaries.

This separation ensures generated parts are semantically distinct yet spatially and visually coherent.

Architecture Overview

• Structure planner: Transformer-based autoregressive model predicting part bounding boxes
• Coverage loss: Ensures generated boxes comprehensively enclose their corresponding 2D parts
• Geometry synthesizer: Fine-tuned voxel latent generative model (building on TRELLIS)
• Position-aware embeddings: Distinguish parts during synthesis via spatial position encoding
• Voxel discarding mechanism: Filters noisy voxels at part boundaries for sharp transitions
• Minimal annotation requirement: Fine-tuning uses only 15K annotated shapes (not per-part supervision)

Implementation

Implement the structure planning stage using autoregressive bounding box prediction:

import torch
import torch.nn as nn
from omnipart.planner import StructurePlanner
 omnipart.losses  PartCoverageLoss

planner = StructurePlanner(
    num_parts_max=,  
    bbox_dim=  
)

coverage_loss_fn = PartCoverageLoss()

 ():
    

    
    image_features = planner.image_encoder(image)
    mask_features = planner.mask_encoder(part_masks_2d)
    context = torch.cat([image_features, mask_features], dim=)

    
    bboxes = []
    hidden_state = planner.init_hidden(context)

     step  (planner.num_parts_max):
        
        bbox_logits = planner.bbox_predictor(hidden_state)
        bbox = planner.decode_bbox(bbox_logits)

        
         bbox[] < :  
            

        bboxes.append(bbox)

        
        hidden_state = planner.update_hidden(hidden_state, bbox)

    
    loss_coverage = coverage_loss_fn(
        predicted_bboxes_3d=bboxes,
        part_masks_2d=part_masks_2d,
        image=image
    )

     bboxes, loss_coverage