canvas-to-image-composition

name	canvas-to-image-composition
title	Canvas-to-Image: Compositional Image Generation with Multimodal Controls
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.21691
keywords	["Compositional Image Generation","Multi-Task Diffusion","Spatial Control","Multi-Person Generation"]
description	Generate images with unified control over identity, spatial position, pose, and layout: encode diverse control modalities (spatial canvas, pose canvas, box canvas) into single RGB image, train diffusion model jointly across all control types, and enable flexible multi-modal composition at inference without task-specific fine-tuning.

Canvas-to-Image: Unified Compositional Image Generation

Current image generation systems handle individual control types separately (identity control, spatial guidance, pose constraints), forcing practitioners to chain multiple models or accept reduced control flexibility. This skill demonstrates Canvas-to-Image, which consolidates diverse guidance signals into a unified visual canvas—a single RGB image encoding all control information—enabling diffusion transformers to learn compositional generation jointly across control modalities.

The core innovation is treating multimodal controls as encodable canvas variants, enabling flexible composition without retraining during inference.

Core Concept

Canvas-to-Image implements unified compositional control through:

1. Multi-Task Canvas Representation: Encodes different control types into RGB canvas variants (spatial, pose, box)
2. Joint Training Architecture: Single diffusion transformer trained across all canvas types simultaneously
3. Emergent Generalization: Model learns to compose multiple controls together despite single-control training samples
4. Constant Computational Cost: Adding more controls doesn't increase inference cost

Architecture Overview

• Canvas Encoder: Converts control canvases to visual tokens
• Vision-Language Model: Encodes text and visual controls jointly
• Diffusion Transformer: Generates images conditioned on canvas + text
• Multi-Task Loss: Single objective covering all control types
• Flexible Canvas Composition: Supports mixing control types at inference

Implementation Steps

The system converts controls to canvas format and trains with unified architecture.

1. Implement Canvas Encoding for Different Control Types

Create methods to encode each control type into RGB canvas.

class CanvasEncoder:
    """
    Converts different control types into unified RGB canvas representation.
    Supports spatial placement, pose guidance, and bounding box constraints.
    """
    def __init__(self, canvas_size=(512, 512)):
        self.canvas_size = canvas_size

     () -> np.ndarray:
        
        canvas = np.ones((image_height, image_width, ), dtype=np.uint8) * 

         subject  subjects:
            subject_img = subject[]
            x_norm = subject[]  
            y_norm = subject[]

            
            x_pixel = (x_norm * image_width)
            y_pixel = (y_norm * image_height)

            
            subject_width = (image_width * )  
            subject_height = (subject_img.height * subject_width / subject_img.width)

            subject_resized = subject_img.resize((subject_width, subject_height))

            
            x_end = (x_pixel + subject_width, image_width)
            y_end = (y_pixel + subject_height, image_height)

            x_start = (, x_pixel)
            y_start = (, y_pixel)

            canvas[y_start:y_end, x_start:x_end] = np.array(subject_resized)

         canvas.astype(np.float32) / 

     () -> np.ndarray:
        
        canvas = np.ones((image_height, image_width, ), dtype=np.uint8) * 

         pose  poses:
            keypoints = pose[]  

            
            skeleton_pairs = [
                (, ), (, ), (, ), (, ),  
                (, ), (, ), (, ),  
                (, ), (, ), (, ),  
                (, ), (, ), (, )  
            ]

             start_idx, end_idx  skeleton_pairs:
                 start_idx < (keypoints)  end_idx < (keypoints):
                    start = keypoints[start_idx]
                    end = keypoints[end_idx]

                    
                    x1 = (start[] * image_width)
                    y1 = (start[] * image_height)
                    x2 = (end[] * image_width)
                    y2 = (end[] * image_height)

                    
                    cv2.line(canvas, (x1, y1), (x2, y2), (, , ), )

            
             kpt  keypoints:
                x = (kpt[] * image_width)
                y = (kpt[] * image_height)
                cv2.circle(canvas, (x, y), , (, , ), -)

         canvas.astype(np.float32) / 

     () -> np.ndarray:
        
        canvas = np.ones((image_height, image_width, ), dtype=np.uint8) * 

        colors = [(, , ), (, , ), (, , ), (, , )]

         idx, box  (boxes):
            bbox = box[]  
            label = box[]

            
            x1 = (bbox[] * image_width)
            y1 = (bbox[] * image_height)
            x2 = (bbox[] * image_width)
            y2 = (bbox[] * image_height)

            color = colors[idx % (colors)]

            
            cv2.rectangle(canvas, (x1, y1), (x2, y2), color, )

            
            font = cv2.FONT_HERSHEY_SIMPLEX
            cv2.putText(canvas, label, (x1, y1 - ), font, , color, )

         canvas.astype(np.float32) / 

     () -> np.ndarray:
        
         weights  :
            weights = [ / (canvases)] * (canvases)

        merged = np.zeros_like(canvases[])

         canvas, weight  (canvases, weights):
            merged += canvas * weight

         np.clip(merged, , )