text-aware-image-restoration

name	text-aware-image-restoration
title	Text-Aware Image Restoration with Diffusion Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.09993
keywords	["image restoration","diffusion models","OCR","text preservation","multi-task learning"]
description	Restore degraded images while preserving textual fidelity using TeReDiff, a multi-task diffusion framework integrating text spotting with U-Net features and VLM-verified dataset curation.

Text-Aware Image Restoration with Diffusion Models

Core Concept

Text-Aware Image Restoration (TAIR) addresses the overlooked problem of preserving textual fidelity alongside visual quality during image restoration. Existing diffusion methods generate visually plausible but textually incorrect content ("text-image hallucination"). TeReDiff solves this via multi-task learning: a diffusion U-Net handles visual restoration while a parallel text-spotting module detects and preserves text instances, guided by dynamically generated text prompts during inference.

Architecture Overview

• TeReDiff Three-Module Design: Lightweight degradation removal (SwinIR-based), diffusion-based restoration (U-Net + ControlNet), transformer-based text spotting
• Three-Stage Training: Stage 1 trains diffusion components; Stage 2 trains text-spotting using diffusion features; Stage 3 jointly optimizes both
• SA-Text Dataset: 100K high-quality images with dense text annotations, VLM-verified using Qwen2.5-VL and OVIS2
• Text Prompt Guidance: During inference, detected texts dynamically generate prompts (e.g., "A realistic scene where the texts [sign], [board]... appear clearly")
• Feature Innovation: Diffusion U-Net features provide superior text detection compared to traditional ResNet backbones

Implementation

Step 1: Dataset Curation Pipeline

import torch
from PIL import Image
import requests
import json

class SA_TextCurationPipeline:
    """
    Creates high-quality TAIR dataset via automated text detection,
    dual VLM verification, and blur filtering.
    """

    def __init__(self):
        self.text_detector = TextDetector()  # CRAFT or similar
        self.vlm_verifiers = [
            VLMModel('Qwen2.5-VL'),
            VLMModel('OVIS2')
        ]

    def curate_single_image(self, image_path, image_quality_threshold=0.7):
        """
        Process single image through curation pipeline.
        Returns (image, text_annotations, quality_score) or None if rejected.
        """

        image = Image.open(image_path)

        # Step 1: Full image text detection
        full_texts = self.text_detector.detect(image)

        # Step 2: Crop-based re-detection for small text
        crops = self._create_crops(image, crop_size=512)
        cropped_texts = []

        for crop in crops:
            crop_texts = self.text_detector.detect(crop)
            cropped_texts.extend(crop_texts)

        # Merge detections, remove duplicates
        all_texts = self._deduplicate_detections(full_texts + cropped_texts)

        # Step 3: VLM verification of detected text
        verified_texts = []

        for detected_text in all_texts:
            # Verify with multiple VLMs
            verifications = []
             vlm  .vlm_verifiers:
                recognition = vlm.recognize_text(image, detected_text[])
                confidence = ._compute_match_confidence(
                    detected_text[],
                    recognition
                )
                verifications.append(confidence)

            
            avg_confidence = (verifications) / (verifications)
             avg_confidence > :
                verified_texts.append({
                    : detected_text[],
                    : detected_text[],
                    : avg_confidence
                })

        
        blur_score = ._assess_blur_via_vlm()
         blur_score < quality_threshold:
               

         {
            : image,
            : verified_texts,
            : blur_score,
            : (verified_texts)
        }

     ():
        
        crops = []
        w, h = image.size
        stride = crop_size // 

         y  (, h - crop_size, stride):
             x  (, w - crop_size, stride):
                crop = image.crop((x, y, x + crop_size, y + crop_size))
                crops.append(crop)

         crops

     ():
        
        unique = []

         det  detections:
            is_duplicate = 
             existing  unique:
                iou = ._compute_iou(det[], existing[])
                 iou > :
                    is_duplicate = 
                    

              is_duplicate:
                unique.append(det)

         unique

     ():
        
         detected == recognized:
             

        
        max_len = ((detected), (recognized))
        edit_dist = ._levenshtein(detected, recognized)

          - (edit_dist / max_len)

     ():
        
        
         

     ():
        
         (s1) < (s2):
             ._levenshtein(s2, s1)

         (s2) == :
             (s1)

        previous_row = ((s2) + )
         i, c1  (s1):
            current_row = [i + ]
             j, c2  (s2):
                insertions = previous_row[j + ] + 
                deletions = current_row[j] + 
                substitutions = previous_row[j] + (c1 != c2)
                current_row.append((insertions, deletions, substitutions))
            previous_row = current_row

         previous_row[-]

     ():
        
        x1_inter = (box1[], box2[])
        y1_inter = (box1[], box2[])
        x2_inter = (box1[], box2[])
        y2_inter = (box1[], box2[])

        inter_area = (, x2_inter - x1_inter) * (, y2_inter - y1_inter)

        box1_area = (box1[] - box1[]) * (box1[] - box1[])
        box2_area = (box2[] - box2[]) * (box2[] - box2[])

        union_area = box1_area + box2_area - inter_area

         inter_area / union_area  union_area >   

Step 2: TeReDiff Architecture

import torch
import torch.nn as nn

class TeReDiff(nn.Module):
    """
    Multi-task diffusion framework for text-aware restoration.
    Integrates visual restoration (U-Net) with text preservation (text spotting).
    """

    def __init__(self, pretrained_unet_path=None):
        super().__init__()

        # Module 1: Degradation removal (lightweight)
        self.degradation_remover = SwinIRModule()

        # Module 2: Diffusion restoration (U-Net + ControlNet)
        self.diffusion_unet = UNetModel()
        self.control_net = ControlNet()

        # Module 3: Text-spotting module (uses diffusion features)
        self.text_spotting = TextSpottingModule()

    def forward_stage1(self, degraded_image, text_prompts, timesteps, noise):
        """
        Stage 1: Train diffusion components (U-Net, ControlNet)
        with text prompts as conditioning.
        """

        # Degrade and restore
        cleaned = self.degradation_remover(degraded_image)

        # Diffusion denoising conditioned on text prompts
        features = self.diffusion_unet(
            cleaned,
            timesteps=timesteps,
            context=text_prompts  # Text conditioning
        )

        # ControlNet adds spatial guidance
        control_features = self.control_net(cleaned)
        restored = features + control_features

        return restored

    def forward_stage2(self, degraded_image, text_gt, timesteps):
        """
        Stage 2: Train text-spotting module using diffusion features
        as input representations (instead of ResNet backbone).
        """

        
         torch.no_grad():
            diffusion_features = .diffusion_unet.extract_features(
                degraded_image,
                timesteps=timesteps
            )

        
        detected_texts = .text_spotting(diffusion_features)

        
        spotting_loss = ._compute_spotting_loss(detected_texts, text_gt)

         spotting_loss

     ():
        

        
        cleaned = .degradation_remover(degraded_image)
        restored = .diffusion_unet(
            cleaned,
            timesteps=timesteps,
            context=text_prompts
        )

        diffusion_features = .diffusion_unet.extract_features(
            degraded_image,
            timesteps=timesteps
        )
        detected_texts = .text_spotting(diffusion_features)

        
        diffusion_loss = ._compute_diffusion_loss(restored, degraded_image)
        spotting_loss = ._compute_spotting_loss(detected_texts, text_gt)

        total_loss = diffusion_loss +  * spotting_loss

         total_loss

     ():
        

        
         torch.no_grad():
            detected_texts = .text_spotting.detect(degraded_image)

        
        text_list = [t[]  t  detected_texts]
         text_list:
            prompt = 
        :
            prompt = 

        
        restored = ._diffusion_infer(
            degraded_image,
            prompt,
            num_steps=num_inference_steps
        )

         restored

     ():
        
         t  ((num_steps)):
            timestep = torch.tensor([t])
            noise_pred = .diffusion_unet(image, timestep, context=prompt)
            image = image - noise_pred
         image

     ():
         torch.nn.functional.mse_loss(restored, original)

     ():
        
         torch.tensor()  

Step 3: Text Prompt Generation

class TextPromptGenerator:
    """
    Dynamically generate natural language prompts based on detected texts
    to guide diffusion restoration.
    """

    def generate_prompt(self, detected_texts, scene_context=None):
        """
        Generate natural text-aware restoration prompt.
        Example: "A realistic scene where the texts [sign], [board]... appear clearly"
        """

        if not detected_texts:
            return "Generate a high-quality restored image"

        # Extract text strings and locations
        text_strings = [t['text'] for t in detected_texts]
        locations = [t.get('location', 'in the image') for t in detected_texts]

        # Build prompt with text instances
        texts_str = ', '.join(text_strings)

        prompt = (
            f"A realistic scene where the texts {texts_str} appear clearly "
            f"on signs, boards, buildings, and other surfaces in the image. "
            f"Restore the image to high quality while preserving text legibility."
        )

        return prompt

    def generate_negative_prompt(self):
        """Generate negative prompt to avoid hallucinated text."""
        return (
            "Blurry text, distorted characters, illegible fonts, "
            "text artifacts, hallucinated text, corrupted writing"
        )

Practical Guidance

Dataset Creation:

• Start with SA-1B (1.1B images) or similar large vision datasets
• Apply automated text detection (CRAFT, FOTS, or similar)
• Use overlapping crops (512×512) to catch small text instances
• Dual VLM verification (Qwen+OVIS) ensures high quality; target 0.95+ agreement

Training Strategy:

• Stage 1: Warm up diffusion with text prompts (100K steps)
• Stage 2: Introduce text spotting module with frozen diffusion (50K steps)
• Stage 3: Joint fine-tuning with combined losses (20K steps)
• Recommend α=0.5 for loss balance

Inference Optimization:

• Text detection is bottleneck; run once on degraded image to identify texts
• Generate prompts once, reuse across diffusion steps
• Batch multiple images to amortize text detection cost

When to Use:

• Document scanning (preserve printed text)
• Historical image restoration (maintain labels/captions)
• Signage/street view restoration (preserve visible text)
• OCR post-processing pipeline (first restore, then extract)

Reference

• Text hallucination: Diffusion models generate plausible but incorrect visual content; grounding with detected text prevents this
• Multi-task learning: Joint training on restoration and detection improves both tasks
• Conditional diffusion: Text prompts guide generation trajectory toward text-aware outputs