Skip to main content 홈 크리에이터 adu2021 skillxiv motion-stream-real-time-interactive-video
motion-stream-real-time-interactive-video Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation.
설치로 이동 Skills Marketplace 커뮤니티가 만든 AI 스킬을 발견하고 탐색하세요.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
직접 명령은 검토 Prompt를 거치지 않습니다. 실행하기 전에 소스를 확인하세요.
npx skills add https://github.com/ADu2021/skillXiv --skill motion-stream-real-time-interactive-video명령은 한 줄로 유지됩니다. 복사하기 전에 가로로 스크롤해 전체 내용을 확인하세요.
로컬 사본을 원하시나요? SkillsMP에서 현재 제공할 수 있는 파일을 다운로드하세요.
Zip 다운로드 다운로드 중... name motion-stream-real-time-interactive-video title MotionStream: Real-Time Video Generation with Interactive Motion Controls version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2511.01266 keywords ["Video Generation","Real-Time Rendering","Motion Control","Distillation","Attention Mechanisms"] description Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation.
Title: Enable Sub-Second, Infinitely-Long Video Generation With Motion Control
Traditional video generation models either lack motion control or sacrifice speed for quality. MotionStream achieves both through a two-stage approach: a bidirectional teacher model with motion guidance is distilled into a fast causal student. The student uses sliding-window attention to handle arbitrary lengths at constant latency, with attention sinks preventing quality degradation during extended generation.
The result is interactive video generation where users paint trajectories and see results in real-time.
Core Concept
Real-Time Motion-Controlled Video Generation :
Teacher Model : Bidirectional, motion-guided, high-quality video generationStudent Model : Causal, single-forward-pass, fast inferenceDistribution Matching Distillation : Bake teacher guidance into student learningSliding-Window Attention : Fixed context window prevents O(n²) memory growthAttention Sinks : Preserve image coherence by pinning initial frame tokens
Architecture Overview
Motion Conditioning : Lightweight sinusoidal embeddings for trajectory inputsTeacher Architecture : Text + motion encoder producing guidance signalsStudent Architecture : Causal video generation with sliding-window attentionAttention Sink Mechanism : Fixed anchor tokens for initializationTraining : Distillation with self-rollout validation during training
Implementation Steps
1. Design Motion Conditioning System
Encode user trajectories into guidance signals for video generation.
class MotionConditioner (nn.Module):
def __init__ (self, hidden_dim=512 ):
self .position_encoding = PositionalEncoding(hidden_dim)
self .trajectory_encoder = nn.LSTM(2 , hidden_dim, batch_first=True
self
def
encode_trajectory
self, trajectory_points
self
self
self
return
def
forward
self, text, trajectory=None
self
if
is
not
None
self
else
return
2. Implement Teacher-Student Distillation
Train student to replicate teacher guidance with single forward pass.
class TeacherStudentDistillation (nn.Module):
def __init__ (self, teacher_model, student_model ):
self .teacher = teacher_model
self .student = student_model
def forward_teacher (self, condition, num_frames=16 ):
video = self .teacher(condition, bidirectional=True )
return video
def forward_student (self, condition, num_frames=16 ):
video = self .student(condition, bidirectional=False )
return video
def compute_distillation_loss (self, condition, num_frames=16 ):
teacher_video = self .forward_teacher(condition, num_frames)
student_video = self .forward_student(condition, num_frames)
teacher_feat = self .teacher.encode(teacher_video)
student_feat = self .student.encode(student_video)
feature_loss = F.mse_loss(student_feat, teacher_feat)
teacher_frames = teacher_video
student_frames = student_video
perceptual_loss = self .compute_perceptual_loss(teacher_frames, student_frames)
return feature_loss + perceptual_loss
def compute_perceptual_loss (self, teacher_frames, student_frames ):
teacher_perc = self .vgg(teacher_frames)
student_perc = self .vgg(student_frames)
return F.mse_loss(teacher_perc, student_perc)
3. Implement Sliding-Window Causal Attention
Enable constant-latency inference for arbitrary-length generation.
class SlidingWindowCausalAttention (nn.Module):
def __init__ (self, hidden_dim, window_size=8 , num_heads=8 ):
self .window_size = window_size
self .hidden_dim = hidden_dim
self .query_projection = nn.Linear(hidden_dim, hidden_dim)
self .key_projection = nn.Linear(hidden_dim, hidden_dim)
self .value_projection = nn.Linear(hidden_dim, hidden_dim)
self .output_projection = nn.Linear(hidden_dim, hidden_dim)
def forward (self, x, cache=None ):
batch_size, seq_len, hidden_dim = x.shape
Q = self .query_projection(x)
K = self .key_projection(x)
V = self .value_projection(x)
if cache is not None :
past_K, past_V = cache
K = torch.cat([past_K, K], dim=1 )
V = torch.cat([past_V, V], dim=1 )
if K.shape[1 ] > self .window_size:
K = K[:, -self .window_size:, :]
V = V[:, -self .window_size:, :]
else :
K = K[:, :self .window_size, :]
V = V[:, :self .window_size, :]
scores = torch.matmul(Q, K.transpose(-2 , -1 )) / np.sqrt(hidden_dim)
weights = F.softmax(scores, dim=-1 )
output = torch.matmul(weights, V)
output = self .output_projection(output)
cache = (K, V)
return output, cache
4. Implement Attention Sinks for Temporal Coherence
Prevent quality degradation during extended generation by anchoring to initial frames.
class AttentionSinkAttention (nn.Module):
def __init__ (self, hidden_dim, num_sink_tokens=4 ):
self .num_sinks = num_sink_tokens
self .hidden_dim = hidden_dim
self .sink_tokens = nn.Parameter(torch.randn(1 , num_sink_tokens, hidden_dim))
def forward (self, x, initial_frame=None ):
batch_size, seq_len, hidden_dim = x.shape
if initial_frame is not None :
self .sink_tokens.data = initial_frame[:, :self .num_sinks, :]
combined = torch.cat([self .sink_tokens.expand(batch_size, -1 , -1 ), x], dim=1 )
return combined
5. Train with Self-Rollout During Training
Simulate inference-time behavior (rolling KV cache) during training.
def train_streaming_video_model (student_model, teacher_model, num_steps=10000 ):
optimizer = torch.optim.Adam(student_model.parameters(), lr=1e-4 )
for step in range (num_steps):
condition = sample_condition_batch()
num_frames = 16
cache = None
student_loss = 0
for frame_idx in range (num_frames):
frame_student, cache = student_model.generate_frame(
condition, frame_idx, cache
)
full_video_teacher = teacher_model(condition)
frame_teacher = full_video_teacher[:, frame_idx, :, :, :]
loss_frame = F.mse_loss(frame_student, frame_teacher)
student_loss += loss_frame
optimizer.zero_grad()
student_loss.backward()
optimizer.step()
if step % 100 == 0 :
print (f"Step {step} : Loss {student_loss.item():.4 f} " )
Practical Guidance
Real-time interactive video generation
Streaming video applications
Motion-conditioned synthesis (UI automation, animation)
window_size: 8 (balance latency vs. coherence)
num_sink_tokens: 4 (anchor quality)
distillation_weight: 0.8 (relative to perception loss)
Applications requiring frame-by-frame editing flexibility
Scenarios needing post-hoc video modifications
Very long sequences (beyond 1-2 minutes)
KV cache overflow : Window must fit in GPU memory; adjust for your hardwareSink token initialization : Poor initialization causes early convergence to wrong attractorSelf-rollout mismatch : Training with rolling cache but evaluating differently causes distribution mismatch Integration Point : Deploy as interactive layer in video editing/creation tools.
Reference 