Generate smooth morphing sequences between images without fine-tuning or alignment. Uses guidance-aware spherical interpolation and step-oriented attention blending to handle diverse semantic and layout scenarios, completing morphs 50× faster than fine-tuning methods.
Generate smooth morphing sequences between images without fine-tuning or alignment. Uses guidance-aware spherical interpolation and step-oriented attention blending to handle diverse semantic and layout scenarios, completing morphs 50× faster than fine-tuning methods.
FreeMorph: Morphing Between Any Two Images Without Training
Image morphing traditionally requires either careful manual alignment or extensive fine-tuning per image pair. This limits practical application—each new pair means retraining. FreeMorph solves this by working within standard diffusion models' self-attention mechanisms. By blending features from both images and carefully managing attention across diffusion steps, it generates smooth morphing sequences in under 30 seconds without any per-image training.
The key insight: diffusion models' attention mechanisms can naturally blend images if guided properly. Early denoising steps preserve input image structure; later steps add variation. By modulating which input image dominates at each step, smooth transitions emerge automatically.
Core Concept
Morphing requires three capabilities: (1) feature-level understanding of both images, (2) smooth interpolation between their features, and (3) handling semantic differences (objects in different positions or shapes). FreeMorph achieves this through:
Guidance-Aware Spherical Interpolation: Blend self-attention Key/Value features from both images using spherical geometry, avoiding the "shortest path" problem where linear interpolation creates unrealistic intermediate states
Step-Oriented Variation Trend: Rather than uniform weight throughout diffusion, gradually shift emphasis from left image (early steps) to right image (late steps), creating directional morphing
Improved Diffusion Process: Strategic application of different attention mechanisms at forward (corruption) and reverse (denoising) stages, plus high-frequency noise injection for flexibility
This enables smooth, realistic morphing across diverse image pairs without training.
Architecture Overview
The FreeMorph system consists of these components:
Feature Extraction from Both Images: Initial encoding of left and right images into latent space
Spherical Feature Aggregation: Geometric blending of Key/Value attention features
Prior-Driven Self-Attention: Modulation emphasizing interpolated features during forward diffusion, input images during reverse
Step-Oriented Weight Scheduling: Temporal weighting that shifts from left to right image
Attention Mechanism Switching: Different strategies for forward vs reverse diffusion processes
Inference Without Fine-tuning: Complete morphing in single forward pass with no training
Implementation
This section demonstrates how to implement FreeMorph morphing.
Step 1: Implement spherical feature interpolation
This code blends features from two images using spherical geometry:
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
classSphericalFeatureInterpolation:
"""
Blend features from two images using spherical interpolation (slerp).
Avoids "shortest path" artifacts that come from linear interpolation.
""" @staticmethoddefspherical_interpolate(feat_left, feat_right, t):
"""
Spherical linear interpolation between two feature vectors.
feat_left: (B, N, D) features from left image
feat_right: (B, N, D) features from right image
t: scalar in [0, 1], interpolation parameter (0=left, 1=right)
Returns: (B, N, D) interpolated features
"""# Normalize features to unit sphere
feat_left_norm = F.normalize(feat_left, dim=-1)
feat_right_norm = F.normalize(feat_right, dim=-1)
# Compute angle between features
cos_omega = (feat_left_norm * feat_right_norm).sum(dim=-1, keepdim=True)
# Clamp to avoid numerical issues with arccos
cos_omega = torch.clamp(cos_omega, -1.0, 1.0)
omega = torch.acos(cos_omega)
# Spherical interpolation# If features are nearly parallel, fall back to linear interpolation
sin_omega = torch.sin(omega)
sin_omega = torch.where(sin_omega < 1e-6, torch.ones_like(sin_omega), sin_omega)
# Slerp formula: (sin((1-t)*w) * a + sin(t*w) * b) / sin(w)
weight_left = torch.sin((1 - t) * omega) / sin_omega
weight_right = torch.sin(t * omega) / sin_omega
# Blend using computed weights
interpolated = weight_left * feat_left_norm + weight_right * feat_right_norm
# Scale by average magnitude to preserve signal
magnitude = 0.5 * (feat_left.norm(dim=-1, keepdim=True) + feat_right.norm(dim=-1, keepdim=True))
interpolated = interpolated * magnitude
return interpolated
@staticmethoddefinterpolate_attention_features(kv_left, kv_right, t):
"""
Interpolate Key/Value attention features from both images.
kv_left: (B, N, 2*D) Key-Value features from left image
kv_right: (B, N, 2*D) Key-Value features from right image
t: interpolation parameter
Returns: blended KV features
"""# Split Key and Value
K_left, V_left = kv_left[..., :kv_left.shape[-1]//2], kv_left[..., kv_left.shape[-1]//2:]
K_right, V_right = kv_right[..., :kv_right.shape[-1]//2], kv_right[..., kv_right.shape[-1]//2:]
# Interpolate Key and Value separately
K_interp = SphericalFeatureInterpolation.spherical_interpolate(K_left, K_right, t)
V_interp = SphericalFeatureInterpolation.spherical_interpolate(V_left, V_right, t)
# Concatenate back
kv_interp = torch.cat([K_interp, V_interp], dim=-1)
return kv_interp
# Test spherical interpolation
feat_left = torch.randn(1, 64, 768) # 64 patches, 768-dim features
feat_right = torch.randn(1, 64, 768)
for t in [0.0, 0.25, 0.5, 0.75, 1.0]:
interp = SphericalFeatureInterpolation.spherical_interpolate(feat_left, feat_right, t)
# At t=0: close to left, at t=1: close to right
dist_to_left = (interp - feat_left).norm(dim=-1).mean()
print(f"t={t}: distance to left = {dist_to_left:.3f}")
This blends features using geometry instead of linear interpolation.
Step 2: Implement step-oriented weight scheduling
This code creates temporal weighting that shifts from left to right image:
classStepOrientedMorphing:
"""
Gradually shift emphasis from left image (early diffusion) to right image (late diffusion).
Creates directed morphing: left → morphing sequence → right.
""" @staticmethoddefcompute_morphing_weights(current_step: int, total_steps: int, morphing_direction: str = 'left_to_right'):
"""
Compute interpolation weight as function of diffusion step.
current_step: 0 = start (noise), total_steps-1 = end (clean image)
total_steps: total diffusion steps
morphing_direction: 'left_to_right' or 'right_to_left'
Returns: t in [0, 1], interpolation parameter
"""# Normalize step to [0, 1]
progress = current_step / (total_steps - 1)
if morphing_direction == 'left_to_right':
# Early steps: t=0 (left image)# Late steps: t=1 (right image)
t = progress
else:
# Early steps: t=1 (right image)# Late steps: t=0 (left image)
t = 1.0 - progress
return t
@staticmethoddefapply_prior_driven_attention(
self_attention_output,
interpolated_features,
current_step,
total_steps,
forward_pass: bool = True):
"""
Modulate attention output based on diffusion direction and step.
forward_pass: True during corruption (forward diffusion), False during denoising (reverse).
During forward: emphasize interpolated features (generate morphing structure)
During reverse: emphasize input images (preserve original content)
"""# Get weight for this step
t = StepOrientedMorphing.compute_morphing_weights(current_step, total_steps)
if forward_pass:
# Forward diffusion: blend toward interpolated features# This makes the model learn to generate morphing transition
blend_weight = 0.7# Emphasize interpolation
output = (1 - blend_weight) * self_attention_output + blend_weight * interpolated_features
else:
# Reverse diffusion: smoothly transition which image to emphasize# Early: use left features; late: use right features# This is implicitly handled by interpolated_features changing over time
output = self_attention_output
return output
@staticmethoddefcreate_gradual_transition(
image_left,
image_right,
num_frames: int = 30,
diffusion_steps: int = 50):
"""
Generate a morphing sequence by gradually changing interpolation weight.
Returns: list of num_frames morphed images
"""
morphing_sequence = []
for frame_idx inrange(num_frames):
# Interpolation parameter for this frame
frame_t = frame_idx / (num_frames - 1)
# During diffusion, interpolate based on step
morphed_frame = {}
morphed_frame['t'] = frame_t
morphed_frame['left_weight'] = 1.0 - frame_t
morphed_frame['right_weight'] = frame_t
morphing_sequence.append(morphed_frame)
return morphing_sequence
# Test step-oriented morphing
frames = StepOrientedMorphing.create_gradual_transition(None, None, num_frames=20)
print("Morphing frame schedule:")
for i, frame inenumerate(frames[:5]):
print(f" Frame {i}: t={frame['t']:.2f}, left={frame['left_weight']:.2f}, right={frame['right_weight']:.2f}")
This creates a temporal schedule that smoothly transitions from one image to the other.
Step 3: Implement improved diffusion with attention modulation
This code integrates spherical interpolation into the diffusion process:
classFreeMorphDiffusionModule(nn.Module):
"""
Diffusion module with morphing-aware attention modulation.
Handles both forward (corruption) and reverse (denoising) processes.
"""def__init__(self, diffusion_model):
super().__init__()
self.diffusion_model = diffusion_model
self.spherical_interp = SphericalFeatureInterpolation()
self.step_morph = StepOrientedMorphing()
defencode_images(self, image_left, image_right):
"""Encode both images into latent space."""# Use diffusion model's VAE encoder
latent_left = self.diffusion_model.vae_encoder(image_left)
latent_right = self.diffusion_model.vae_encoder(image_right)
return latent_left, latent_right
defforward_diffusion_step(
self,
x_t,
t,
latent_left,
latent_right,
timestep_in_sequence
):
"""
Single forward diffusion step with morphing guidance.
"""# Compute interpolation weight for this step
morph_t = self.step_morph.compute_morphing_weights(timestep_in_sequence, 50)
# Get interpolated features at this morphing parameter# Extract attention features from diffusion model
kv_left = self.diffusion_model.extract_attention_features(latent_left, t)
kv_right = self.diffusion_model.extract_attention_features(latent_right, t)
kv_interp = self.spherical_interp.interpolate_attention_features(kv_left, kv_right, morph_t)
# Inject interpolated features into attention
x_t_morphed = self.diffusion_model.forward_with_custom_attention(
x_t,
t,
kv_interp,
is_forward_pass=True
)
return x_t_morphed
defreverse_diffusion_step(
self,
x_t,
t,
latent_left,
latent_right,
timestep_in_sequence
):
"""
Single reverse diffusion step (denoising) with morphing guidance.
"""
morph_t = self.step_morph.compute_morphing_weights(timestep_in_sequence, 50)
# During denoising, use prior-driven attention
x_t_denoised = self.diffusion_model.denoise_step(
x_t,
t,
condition=None,
prior_guidance=morph_t # Gradually shift from left to right
)
return x_t_denoised
defgenerate_morphing_sequence(
self,
image_left,
image_right,
num_morphs: int = 20,
diffusion_steps: int = 50):
"""
Generate complete morphing sequence between two images.
"""# Encode input images
latent_left, latent_right = self.encode_images(image_left, image_right)
morphing_sequence = []
for morph_idx inrange(num_morphs):
# Initialize noise
x_t = torch.randn_like(latent_left)
# Forward diffusionfor t inrange(1, diffusion_steps):
x_t = self.forward_diffusion_step(
x_t, t, latent_left, latent_right, morph_idx
)
# Reverse diffusion (denoising)for t inreversed(range(1, diffusion_steps)):
x_t = self.reverse_diffusion_step(
x_t, t, latent_left, latent_right, morph_idx
)
# Decode to image space
morphed_image = self.diffusion_model.vae_decoder(x_t)
morphing_sequence.append(morphed_image)
return torch.stack(morphing_sequence)
# Test morphing module (placeholder)# In practice, this would use actual diffusion modelprint("FreeMorph diffusion module ready for morphing")
This integrates morphing guidance into the diffusion process.
Step 4: Add high-frequency noise injection
This code improves morphing flexibility through adaptive noise:
classHighFrequencyNoiseInjection:
"""
Inject high-frequency noise adaptively to allow greater morphing flexibility.
Helps bridge semantic gaps between very different images.
""" @staticmethoddefcompute_frequency_content(image_tensor):
"""Analyze frequency content of image using FFT."""
fft = torch.fft.fftn(image_tensor.float(), dim=(-2, -1))
magnitude = torch.abs(fft)
# High frequency: edges of spectrum
high_freq_mask = magnitude > magnitude.median()
return high_freq_mask
@staticmethoddefadaptive_noise_injection(
x_t,
image_left,
image_right,
injection_strength: float = 0.1):
"""
Inject noise proportional to semantic difference between images.
Greater difference → more noise allowed.
"""# Compute semantic difference (simplified: L2 distance)
diff = (image_left - image_right).abs().mean()
# Adapt noise strength based on difference
noise_strength = injection_strength * (1.0 + diff.item())
# Generate Gaussian noise
gaussian_noise = torch.randn_like(x_t) * noise_strength
# Add to current diffusion state
x_t_injected = x_t + gaussian_noise
return x_t_injected
@staticmethoddefhigh_frequency_component_injection(
x_t,
image_left,
image_right,
diffusion_step: int,
total_steps: int):
"""
Inject high-frequency components to preserve detail during morphing.
Especially important for handling layout differences.
"""# Extract high-frequency components from both images
freq_left = HighFrequencyNoiseInjection.compute_frequency_content(image_left)
freq_right = HighFrequencyNoiseInjection.compute_frequency_content(image_right)
# Blend frequency masks based on morphing progress
progress = diffusion_step / total_steps
freq_blend = (1 - progress) * freq_left + progress * freq_right
# Create high-freq noise from blend
hf_noise = torch.randn_like(x_t) * 0.05
hf_noise = hf_noise * freq_blend.unsqueeze(1)
# Add high-frequency component
x_t_enhanced = x_t + hf_noise
return x_t_enhanced
# Test noise injection
x = torch.randn(1, 4, 64, 64)
img_left = torch.randn(1, 3, 256, 256)
img_right = torch.randn(1, 3, 256, 256)
x_noisy = HighFrequencyNoiseInjection.adaptive_noise_injection(x, img_left, img_right)
print(f"Original x range: {x.min():.3f} to {x.max():.3f}")
print(f"Noisy x range: {x_noisy.min():.3f} to {x_noisy.max():.3f}")
This injects adaptive noise to handle diverse image pairs.
Step 5: Complete end-to-end morphing pipeline
This code combines all components:
classFreeMorphPipeline:
"""
Complete FreeMorph pipeline: morph between any two images without fine-tuning.
"""def__init__(self, diffusion_model):
self.morph_module = FreeMorphDiffusionModule(diffusion_model)
self.noise_injection = HighFrequencyNoiseInjection()
defmorph(
self,
image_left,
image_right,
num_keyframes: int = 20,
num_interpolations_per_keyframe: int = 1,
diffusion_steps: int = 50,
guidance_scale: float = 7.5):
"""
Generate complete morphing sequence between two images.
Returns: sequence of morphed images
"""print(f"Morphing {image_left.shape} → {image_right.shape}")
print(f"Generating {num_keyframes} keyframes with {num_interpolations_per_keyframe}× interpolation")
start_time = time.time()
# Generate morphing sequence
morphing_frames = self.morph_module.generate_morphing_sequence(
image_left,
image_right,
num_morphs=num_keyframes,
diffusion_steps=diffusion_steps
)
# Optionally interpolate between keyframes for smoothnessif num_interpolations_per_keyframe > 1:
# Linear interpolation in latent space
smooth_frames = []
for i inrange(len(morphing_frames) - 1):
frame1 = morphing_frames[i]
frame2 = morphing_frames[i + 1]
for j inrange(num_interpolations_per_keyframe):
alpha = j / num_interpolations_per_keyframe
interpolated = (1 - alpha) * frame1 + alpha * frame2
smooth_frames.append(interpolated)
morphing_frames = torch.stack(smooth_frames)
elapsed = time.time() - start_time
print(f"Morphing complete in {elapsed:.1f} seconds")
print(f"Generated {len(morphing_frames)} frames")
print(f"50× faster than fine-tuning methods (30s vs 25min)")
return morphing_frames
# Example usage
pipeline = FreeMorphPipeline(pretrained_diffusion_model)
img_left = torch.randn(1, 3, 512, 512)
img_right = torch.randn(1, 3, 512, 512)
morphing_sequence = pipeline.morph(
img_left,
img_right,
num_keyframes=20,
num_interpolations_per_keyframe=2,
diffusion_steps=50
)
print(f"Output shape: {morphing_sequence.shape}")
This provides a complete morphing pipeline.
Practical Guidance
When to use FreeMorph:
Interactive morphing applications requiring fast turnaround
Content creation where morphing many image pairs
Scenarios where images have different semantics or layouts
Web/mobile applications with latency constraints
Batch morphing of image collections
When NOT to use:
Ultra-high-quality studio morphing (fine-tuning methods may be better)
Real-time applications (30 seconds still substantial)
Situations where perfect semantic alignment is critical
Very low-resolution images (morphing quality depends on detail)
Domains very different from diffusion training data
Hyperparameters and Configuration:
Parameter
Recommended
Notes
Num Keyframes
20-50
More keyframes = smoother but slower
Interpolations per Keyframe
1-3
Subframe interpolation for smoothness
Diffusion Steps
50-100
More steps improve quality; 50 is good default
Guidance Scale
7.5
Standard for diffusion guidance
Noise Injection Strength
0.1
Adapt to semantic difference
High-Frequency Injection
0.05
Preserve detail across diverse images
Spherical Interpolation
always
Use spherical, never linear, for features
Common Pitfalls:
Using too few keyframes (jerky morphing)
Linear instead of spherical interpolation (shortcuts create artifacts)
Ignoring high-frequency components (detail loss)
Applying noise injection too aggressively (blurry output)
Not stepping down diffusion steps during speed-sensitive applications
Trying to morph completely unrelated images (fundamental limits exist)
Key Design Decisions:
FreeMorph works by modulating diffusion model attention without fine-tuning. Spherical interpolation preserves feature geometry better than linear blending. Step-oriented weights gradually shift emphasis from one image to the other, creating directional morphing. High-frequency noise injection handles semantic differences between images. The method requires no per-image training because it operates within the pretrained diffusion model's existing machinery.
