calligrapher-text-style-customization

name	calligrapher-text-style-customization
title	Calligrapher: Freestyle Text Image Customization
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.24123
keywords	["TextGeneration","StyleTransfer","DiffusionModels","Typography","ImageCustomization"]
description	Automates artistic typography customization through self-distilled learning and localized style injection. Generates stylized text images by encoding reference style and injecting it into diffusion denoising. Use for digital design workflows, text-based visual content creation, or applications needing artistic typography control without manual annotation.

Calligrapher: Automated Typography Customization Through Style-Aware Diffusion

Typography is an art form, but designing custom stylized text requires skilled typographers and extensive manual work. Most text generation systems either ignore style entirely or require parallel paired examples. Calligrapher solves this by leveraging self-distillation to create synthetic paired training data and implementing localized style injection—encoding reference style information and fusing it into the generation process at inference time. The result is a system that generates high-quality typographic designs matching arbitrary reference styles without human annotation.

The key innovation is recognizing that style is local and learnable from single examples. Rather than requiring hundreds of parallel examples, Calligrapher extracts style features from a single reference image and conditions the entire generation process, enabling one-shot style transfer for text.

Core Concept

Calligrapher uses three complementary mechanisms to solve the typography customization problem:

1. Self-Distillation: Leverages pre-trained diffusion models and LLMs to synthesize paired training data with style annotations, avoiding expensive manual labeling.

2. Style Encoder: A learnable module (combining Qformer and linear projections) that extracts style features from reference images, producing conditioned representations for the generation pipeline.

3. Localized Style Injection: Integrates style information directly into the diffusion denoising process through modified cross-attention, ensuring style consistency throughout generation while allowing flexible text content.

The framework supports three customization modes: self-reference (edit text while preserving original style), cross-reference (apply style from one image to different text), and style transfer from non-text images.

Architecture Overview

• Style Encoder: Vision transformer backbone (Qformer) extracting semantic style features, projected to cross-attention dimensions
• Self-Distillation Module: Generates synthetic paired training data using pre-trained diffusion models and LLMs
• Modified Cross-Attention Layer: Replaces standard text conditioning with style-aware attention that fuses style and content
• Spatial Concatenation: In-context generation embedding reference images directly in the latent space
• Diffusion Backbone: Standard U-Net denoiser adapted to accept style-conditioned cross-attention

Implementation

The style encoder extracts robust style features from reference images: