dcm-dual-expert-consistency

name	dcm-dual-expert-consistency
title	DCM: Dual-Expert Consistency Model for Efficient and High-Quality Video Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.03123
keywords	["video-generation","consistency-distillation","dual-experts","efficiency","diffusion-models"]
description	Accelerate video generation through dual-expert consistency distillation, using separate denoisers for semantic layout/motion and detail refinement to resolve conflicting optimization gradients.

DCM: Dual-Expert Consistency Model for Efficient and High-Quality Video Generation

Core Concept

Consistency distillation for video synthesis faces a fundamental optimization conflict: early diffusion steps require rapid semantic changes (layout, motion) while later steps need gradual detail refinement. A single student model cannot simultaneously optimize for both dynamics, creating conflicting gradients. DCM solves this by employing two specialized expert denoisers—one for semantic/motion reasoning, one for detail synthesis—enabling efficient 4-step video generation while maintaining quality near 50-step original models.

The approach maintains parameter efficiency through LoRA adapters and frozen semantic experts, achieving VBench scores of 83.83 (vs. 80.33 baseline) with 92% latency reduction.

Architecture Overview

• Semantic Expert (SemE): Operates on high-noise samples (timesteps 1-25), captures layout and motion dynamics
• Detail Expert (DetE): Operates on low-noise samples (timesteps 26-50), refines fine details through adversarial training
• Parameter Efficiency: Freezes semantic expert, adds LoRA + timestep embeddings to detail expert only
• Specialized Loss Functions: Temporal coherence for motion consistency, GAN + feature matching for detail quality
• Inference Pipeline: Routes timesteps to appropriate expert during 4-step generation

Implementation

1. Expert Specialization: Train denoisers on distinct noise ranges

# Expert training specification
def create_dual_expert_curriculum(diffusion_model, train_data):
    """
    Partition training into high-noise (semantic) and low-noise (detail) phases.
    Each expert specializes in different denoising dynamics.
    """
    semantic_expert = copy_model(diffusion_model)  # Will be frozen
    detail_expert = copy_model(diffusion_model)   # Will be adapted with LoRA

    # High-noise timesteps for semantic learning (t: 1-25, high sigma)
    semantic_batch = filter_timesteps(train_data, timestep_range=(1, 25))

    # Low-noise timesteps for detail learning (t: 26-50, low sigma)
    detail_batch = filter_timesteps(train_data, timestep_range=(26, ))

     {
        : semantic_expert,
        : detail_expert,
        : semantic_batch,
        : detail_batch
    }