speed-by-simplicity

name	speed-by-simplicity
title	Speed by Simplicity: Unified Backbone for Audio-Video-Text Streaming
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.21986
keywords	["Architecture Design","Multimodal","Single-Stream","Audio","Video"]
description	Replace multi-stream modality-specific pathways with a unified Transformer backbone processing text, video, and audio tokens in shared sequence via self-attention. Achieves superior visual quality (4.80 vs 4.76), 75% better speech clarity (14.6% WER vs 19.23%), and 80% human preference wins—particularly strong for human-centric scenarios with expressive facial performance and audio-video sync.

Component Identification

Old Design (Multi-Stream Baseline)

• Separate pathways for video and audio tokens
• Dedicated fusion blocks for modality alignment
• Modality-specific preprocessing and embedding layers
• Complex cross-attention mechanisms for temporal coordination

New Design (Single-Stream) Unified Transformer backbone with shared self-attention sequence combining text, video, and audio tokens.

Motivation & Problem Statement

Leading open-source models rely on heavily specialized designs that increase parameter count, training complexity, and latency. The simplified approach tests whether a single shared backbone—treating all modalities as sequential tokens—can achieve competitive or superior performance while reducing architectural overhead.

The Modification

The core architectural change replaces modality-specific pathways with token-level fusion:

# Single-stream processing: all modalities in shared sequence
# Text tokens: [CLS] w1 w2 ... wN [SEP]
# Video tokens: [VID] v1 v2 ... vM [SEP]
# Audio tokens: [AUD] a1 a2 ... aK [SEP]

# Combined sequence processed by single Transformer backbone
sequence = [text_tokens + video_tokens + audio_tokens]
output = transformer_backbone(sequence)  # Unified self-attention

# Modality-specific decoders applied to output
text_out = text_decoder(output[text_indices])
video_out = video_decoder(output[video_indices])
audio_out = audio_decoder(output[audio_indices])

This eliminates dedicated fusion blocks and modality-specific alignment layers, allowing emergent cross-modal interaction through the shared attention mechanism.

Ablation Results with Exact Numbers

Visual Quality

• daVinci-MagiHuman: 4.80
• LTX 2.3: 4.76
• Ovi 1.1: 4.73

Speech Clarity (Word Error Rate)

• daVinci-MagiHuman: 14.60%
• LTX 2.3: 19.23%
• Ovi 1.1: 40.45%

Human Preference

• 80% preference vs Ovi 1.1
• 60.9% preference vs LTX 2.3

Domain-Specific Performance

Strong in human-centric scenarios:

• Expressive facial performance
• Natural speech-expression coordination
• Realistic body motion synthesis
• Precise audio-video synchronization

Multilingual capability: Six major languages with consistent quality

Conditions of Applicability

Works well when:

• Modality tokens share temporal alignment (speech-to-video sync is critical)
• Model capacity is sufficient (unified backbone needs adequate parameters)
• Training data includes diverse modality combinations
• Human-centric content is primary (facial expression, gesture, speech coordination critical)

Less optimal when:

• Modalities operate at very different timescales
• Extreme modality imbalance in training data
• Real-time processing with hard latency constraints (unified attention still requires full sequence processing)

Drop-In Replacement Checklist

• Maintains same input/output interface (text + video + audio conditioning)
• Compatible with existing video diffusion decoders
• No architectural changes required to downstream components
• KV-cache compatible for streaming inference
• Can be trained with standard cross-entropy losses
• Inference latency improved over multi-stream baselines