| name | canvas-to-image-composition |
| title | Canvas-to-Image: Compositional Image Generation with Multimodal Controls |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2511.21691 |
| keywords | ["Compositional Image Generation","Multi-Task Diffusion","Spatial Control","Multi-Person Generation"] |
| description | Generate images with unified control over identity, spatial position, pose, and layout: encode diverse control modalities (spatial canvas, pose canvas, box canvas) into single RGB image, train diffusion model jointly across all control types, and enable flexible multi-modal composition at inference without task-specific fine-tuning. |
Canvas-to-Image: Unified Compositional Image Generation
Current image generation systems handle individual control types separately (identity control, spatial guidance, pose constraints), forcing practitioners to chain multiple models or accept reduced control flexibility. This skill demonstrates Canvas-to-Image, which consolidates diverse guidance signals into a unified visual canvas—a single RGB image encoding all control information—enabling diffusion transformers to learn compositional generation jointly across control modalities.
The core innovation is treating multimodal controls as encodable canvas variants, enabling flexible composition without retraining during inference.
Core Concept
Canvas-to-Image implements unified compositional control through:
- Multi-Task Canvas Representation: Encodes different control types into RGB canvas variants (spatial, pose, box)
- Joint Training Architecture: Single diffusion transformer trained across all canvas types simultaneously
- Emergent Generalization: Model learns to compose multiple controls together despite single-control training samples
- Constant Computational Cost: Adding more controls doesn't increase inference cost
Architecture Overview
- Canvas Encoder: Converts control canvases to visual tokens
- Vision-Language Model: Encodes text and visual controls jointly
- Diffusion Transformer: Generates images conditioned on canvas + text
- Multi-Task Loss: Single objective covering all control types
- Flexible Canvas Composition: Supports mixing control types at inference
Implementation Steps
The system converts controls to canvas format and trains with unified architecture.
1. Implement Canvas Encoding for Different Control Types
Create methods to encode each control type into RGB canvas.
class CanvasEncoder:
"""
Converts different control types into unified RGB canvas representation.
Supports spatial placement, pose guidance, and bounding box constraints.
"""
def __init__(self, canvas_size=(512, 512)):
self.canvas_size = canvas_size
() -> np.ndarray:
canvas = np.ones((image_height, image_width, ), dtype=np.uint8) *
subject subjects:
subject_img = subject[]
x_norm = subject[]
y_norm = subject[]
x_pixel = (x_norm * image_width)
y_pixel = (y_norm * image_height)
subject_width = (image_width * )
subject_height = (subject_img.height * subject_width / subject_img.width)
subject_resized = subject_img.resize((subject_width, subject_height))
x_end = (x_pixel + subject_width, image_width)
y_end = (y_pixel + subject_height, image_height)
x_start = (, x_pixel)
y_start = (, y_pixel)
canvas[y_start:y_end, x_start:x_end] = np.array(subject_resized)
canvas.astype(np.float32) /
() -> np.ndarray:
canvas = np.ones((image_height, image_width, ), dtype=np.uint8) *
pose poses:
keypoints = pose[]
skeleton_pairs = [
(, ), (, ), (, ), (, ),
(, ), (, ), (, ),
(, ), (, ), (, ),
(, ), (, ), (, )
]
start_idx, end_idx skeleton_pairs:
start_idx < (keypoints) end_idx < (keypoints):
start = keypoints[start_idx]
end = keypoints[end_idx]
x1 = (start[] * image_width)
y1 = (start[] * image_height)
x2 = (end[] * image_width)
y2 = (end[] * image_height)
cv2.line(canvas, (x1, y1), (x2, y2), (, , ), )
kpt keypoints:
x = (kpt[] * image_width)
y = (kpt[] * image_height)
cv2.circle(canvas, (x, y), , (, , ), -)
canvas.astype(np.float32) /
() -> np.ndarray:
canvas = np.ones((image_height, image_width, ), dtype=np.uint8) *
colors = [(, , ), (, , ), (, , ), (, , )]
idx, box (boxes):
bbox = box[]
label = box[]
x1 = (bbox[] * image_width)
y1 = (bbox[] * image_height)
x2 = (bbox[] * image_width)
y2 = (bbox[] * image_height)
color = colors[idx % (colors)]
cv2.rectangle(canvas, (x1, y1), (x2, y2), color, )
font = cv2.FONT_HERSHEY_SIMPLEX
cv2.putText(canvas, label, (x1, y1 - ), font, , color, )
canvas.astype(np.float32) /
() -> np.ndarray:
weights :
weights = [ / (canvases)] * (canvases)
merged = np.zeros_like(canvases[])
canvas, weight (canvases, weights):
merged += canvas * weight
np.clip(merged, , )
