duetsvg-multimodal-svg

name	duetsvg-multimodal-svg
title	DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.10894
keywords	["SVG generation","multimodal models","visual guidance","vector graphics","test-time scaling"]
description	Generate SVGs through simultaneous image and SVG token generation with internal visual guidance. DuetSVG overcomes text-only limitations by leveraging visual predictions to enhance SVG coherence—ideal when visual quality and geometric correctness matter.

Overview

DuetSVG addresses limitations in vision-language-only SVG generation by simultaneously generating both image tokens and SVG tokens in an integrated process. Test-time scaling uses the model's own visual predictions as guidance to enhance generation quality.

When to Use

• Scalable vector graphics generation from text/images
• Need for visually coherent SVGs
• Scenarios requiring geometric correctness
• Applications with diverse SVG generation tasks
• Need for semantic coherence and visual appeal

When NOT to Use

• Raster image generation (use diffusion models)
• Simple geometric shapes (simpler approaches sufficient)
• Real-time SVG generation with latency constraints

Core Technique

Unified multimodal generation with test-time scaling:

# DuetSVG: Unified multimodal SVG generation
class DuetSVGGenerator:
    def __init__(self):
        self.image_encoder = ImageEncoder()
        self.svg_tokenizer = SVGTokenizer()
        self.multimodal_decoder = MultimodalDecoder()

    def generate_svg_with_visual_guidance(self, text_prompt):
        """
        Generate SVG jointly with image tokens.
        Use visual predictions to guide SVG generation.
        """
        # Initialize generation
        generated_image_tokens = []
        generated_svg_tokens = []

        for step in range(max_steps):
            # Joint decoding: image and SVG tokens together
            next_image_token = self.multimodal_decoder.predict_image_token(
                generated_image_tokens,
                generated_svg_tokens,
                text_prompt
            )

            next_svg_token = self.multimodal_decoder.predict_svg_token(
                generated_image_tokens,
                generated_svg_tokens,
                text_prompt
            )

            # Internal visual guidance: use predicted image to guide SVG
             .should_apply_visual_guidance(step):
                
                partial_image = .decode_image_tokens(
                    generated_image_tokens + [next_image_token]
                )

                
                visual_features = .extract_visual_features(
                    partial_image
                )

                
                next_svg_token = .rescore_svg_token(
                    next_svg_token,
                    visual_features,
                    partial_image
                )

            generated_image_tokens.append(next_image_token)
            generated_svg_tokens.append(next_svg_token)

            
             .is_complete(generated_svg_tokens):
                

        
        final_svg = .svg_tokenizer.decode(generated_svg_tokens)
        final_image = .decode_image_tokens(generated_image_tokens)

         final_svg, final_image

     ():
        
        features = {
            : .extract_colors(image),
            : .extract_layout(image),
            : .detect_objects(image),
            : .analyze_style(image)
        }
         features

     ():
        
        
        alternatives = .generate_alternatives(token)

        scores = []
         alt  alternatives:
            
            test_svg = .simulate_svg_addition(alt, partial_image)

            
            consistency = .compute_visual_consistency(
                test_svg,
                visual_features,
                partial_image
            )

            scores.append(consistency)

        
        best_token = alternatives[torch.argmax(torch.tensor(scores))]
         best_token

     ():
        
        candidates = []

         sample_idx  (budget):
            
            svg, image = .generate_svg_with_visual_guidance(
                text_prompt
            )

            candidates.append((svg, image))

        
        best_svg = .select_best_candidate(
            candidates,
            text_prompt
        )

         best_svg

     ():
        
        scores = []

         svg, image  candidates:
            
            visual_quality = .score_visual_quality(image)
            semantic_match = .score_semantic_match(svg, text_prompt)
            internal_consistency = .score_consistency(svg, image)

            total_score = (
                 * visual_quality +
                 * semantic_match +
                 * internal_consistency
            )

            scores.append(total_score)

        best_idx = torch.argmax(torch.tensor(scores))
         candidates[best_idx][]