Skip to main content Home Creators adu2021 skillxiv latent-particle-world-models
latent-particle-world-models Learn world models by decomposing scenes into latent particles with per-particle dynamics. Each particle represents an object with position, scale, and appearance. Learn distributed latent actions governing per-particle transitions, enabling multimodal video generation from identical initial conditions.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill latent-particle-world-modelsThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name latent-particle-world-models title Latent Particle World Models: Self-Supervised Object-Centric Stochastic Dynamics Modeling version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2603.04553 keywords ["World Models","Object-Centric Learning","Latent Dynamics","Stochastic Generation","Video Understanding"] description Learn world models by decomposing scenes into latent particles with per-particle dynamics. Each particle represents an object with position, scale, and appearance. Learn distributed latent actions governing per-particle transitions, enabling multimodal video generation from identical initial conditions.
Latent Particle World Models: Object-Centric Stochastic Dynamics
Traditional world models predict full video frames (wasteful) or use global latent actions (insufficient for multi-entity scenes). Latent Particle World Models (LPWM) decomposes video into object-centric particles with independent dynamics. The key innovation is per-particle latent actions: instead of a single global action determining the full scene, each particle has its own latent action distribution, enabling rich multi-entity interactions and genuine stochastic behavior.
The core insight is that complex scenes with multiple agents naturally decompose into independent actors, each with their own latent policy. This enables realistic multi-modal generation where identical initial conditions lead to different outcomes (e.g., multiple agents choosing different actions).
Core Concept
LPWM operates through three coordinated mechanisms:
Object-Centric Decomposition : Segment video scenes into latent particles, each representing an object or agentPer-Particle Latent Actions : Learn distributions over latent actions specific to each particle, not global to the sceneFlexible Conditioning : Support diverse control modalities (language goals, external actions, user instructions) by mapping to per-particle latent actions
Architecture Overview
Input : Video sequences or action-annotated trajectoriesScene Decomposition : Latent VAE decomposes scenes into particles (position, scale, appearance)Dynamics Model : Temporal transitions with per-particle learned policiesLatent Action Sampler : Sample per-particle actions from learned distributionsReconstruction : Render predicted particle states back to image spaceOutput : Multimodal video predictions, controllable generation
Implementation Steps
Step 1: Design particle representation
Define compact per-particle state encoding.
class Particle :
"""
Represents a single object/entity in a scene.
"""
def __init__ (self, position, scale, appearance, opacity=1.0 ):
"""
position: (x, y) coordinates (normalized 0-1)
scale: (h, w) relative to image size
appearance: learned embedding (e.g., 16-dim vector)
opacity: alpha channel (0-1)
"""
self
self
self
self
def
to_state_vector
self
"""Concatenate into single state vector."""
return
self
self
self
self
@classmethod
def
from_state_vector
cls, state_vector
"""Reconstruct particle from state vector."""
2
2
4
4
20
20
return
class
SceneDecomposition
"""Decompose video frames into particles."""
def
__init__
self, num_particles=4 , video_vae=None
self
self
def
decompose_frame
self, frame
"""
Extract particles from a single frame.
"""
self
self
self
for
in
self
self
self
self
return
def
_slot_attention
self, latent, num_slots
"""Decompose latent into K slot representations."""
return
64
def
_decode_position
self, slot
"""Extract position coordinates from slot."""
return
0
1.0
1
1.0
def
_decode_scale
self, slot
"""Extract scale from slot."""
return
0.1
2
1.0
0.8
0.1
3
1.0
0.8
def
_decode_appearance
self, slot
"""Extract appearance embedding from slot."""
return
4
20
def
_decode_opacity
self, slot
"""Extract opacity from slot."""
return
0.5
20
1.0
0.5
Step 2: Model per-particle latent action distributions
Define learned stochastic policies for each particle.
class ParticleLatentPolicy :
"""
Learn distribution over latent actions for a specific particle type.
"""
def __init__ (self, latent_action_dim=4 ):
self .latent_action_dim = latent_action_dim
self .policy_net = nn.Sequential(
nn.Linear(21 , 64 ),
nn.ReLU(),
nn.Linear(64 , 64 ),
nn.ReLU(),
nn.Linear(64 , latent_action_dim * 2 )
)
def sample_action (self, particle_state ):
"""
Sample latent action for this particle.
Returns: latent action vector (e.g., 4D)
"""
output = self .policy_net(particle_state)
mean = output[:self .latent_action_dim]
log_std = output[self .latent_action_dim:]
std = torch.exp(log_std)
epsilon = torch.randn_like(std)
latent_action = mean + std * epsilon
return latent_action
def compute_logprob (self, particle_state, latent_action ):
"""Compute log-probability of action under this policy."""
output = self .policy_net(particle_state)
mean = output[:self .latent_action_dim]
log_std = output[self .latent_action_dim:]
std = torch.exp(log_std)
logprob = -0.5 * ((latent_action - mean) / std) ** 2 - log_std
return logprob.sum ()
class DynamicsModel :
"""
Update particle states based on latent actions.
z_i^{t+1} = f(z_i^t, a_i^t, context)
"""
def __init__ (self, state_dim=21 , latent_action_dim=4 ):
self .state_dim = state_dim
self .latent_action_dim = latent_action_dim
self .transition_net = nn.Sequential(
nn.Linear(state_dim + latent_action_dim, 128 ),
nn.ReLU(),
nn.Linear(128 , 128 ),
nn.ReLU(),
nn.Linear(128 , state_dim)
)
def predict_next_state (self, particle_state, latent_action ):
"""Predict next particle state given current state and action."""
input_vector = torch.cat([particle_state, latent_action], dim=-1 )
state_delta = self .transition_net(input_vector)
next_state = particle_state + state_delta
next_state = self ._apply_constraints(next_state)
return next_state
def _apply_constraints (self, state ):
"""Enforce valid particle state ranges."""
state[:, :2 ] = torch.clamp(state[:, :2 ], 0 , 1 )
state[:, 2 :4 ] = torch.clamp(state[:, 2 :4 ], 0.05 , 0.95 )
state[:, -1 ] = torch.clamp(state[:, -1 ], 0 , 1 )
return state
Step 3: Implement scene rendering from particles
Convert particle states back to image space.
class ParticleRenderer :
"""Render particles back to image space."""
def __init__ (self, image_size=64 ):
self .image_size = image_size
def render_scene (self, particles ):
"""
Compose particles into single image.
"""
canvas = np.zeros((self .image_size, self .image_size, 3 ))
sorted_particles = sorted (particles,
key=lambda p: np.mean(p.appearance))
for particle in sorted_particles:
canvas = self ._composite_particle(canvas, particle)
return canvas
def _composite_particle (self, canvas, particle ):
"""Composite single particle onto canvas with alpha blending."""
texture = self ._appearance_to_texture(particle.appearance)
center_x = int (particle.position[0 ] * self .image_size)
center_y = int (particle.position[1 ] * self .image_size)
h = int (particle.scale[0 ] * self .image_size)
w = int (particle.scale[1 ] * self .image_size)
x_min = max (0 , center_x - w // 2 )
x_max = min (self .image_size, center_x + w // 2 )
y_min = max (0 , center_y - h // 2 )
y_max = min (self .image_size, center_y + h // 2 )
if x_max > x_min and y_max > y_min:
texture_region = texture[:h, :w]
alpha = particle.opacity
canvas[y_min:y_max, x_min:x_max] = \
(1 - alpha) * canvas[y_min:y_max, x_min:x_max] + \
alpha * texture_region[:y_max - y_min, :x_max - x_min]
return canvas
def _appearance_to_texture (self, appearance_embedding ):
"""Convert appearance embedding to rendered texture."""
np.random.seed(hash (appearance_embedding.tobytes()) % (2 ** 32 ))
return np.random.randn(32 , 32 , 3 ) * 0.5 + 0.5
Step 4: Define training objective
Train to reconstruct videos and match target distributions.
def compute_lpwm_loss (model, video_batch, latent_action_dim=4 ):
"""
LPWM training loss combining reconstruction and KL divergence.
"""
batch_size, seq_len, H, W, C = video_batch.shape
first_particles = model.decomposer.decompose_frame(video_batch[:, 0 ])
total_loss = 0.0
for t in range (seq_len - 1 ):
current_frame = video_batch[:, t]
target_frame = video_batch[:, t + 1 ]
particles = model.decomposer.decompose_frame(current_frame)
latent_actions = []
log_probs = []
for i, particle in enumerate (particles):
particle_state = torch.tensor(particle.to_state_vector(), dtype=torch.float32)
action = model.policies[i].sample_action(particle_state)
latent_actions.append(action)
logprob = model.policies[i].compute_logprob(particle_state, action)
log_probs.append(logprob)
next_particles = []
for i, (particle, action) in enumerate (zip (particles, latent_actions)):
particle_state = torch.tensor(particle.to_state_vector(), dtype=torch.float32)
next_state = model.dynamics.predict_next_state(particle_state, action)
next_particle = Particle.from_state_vector(next_state.detach().numpy())
next_particles.append(next_particle)
predicted_frame = model.renderer.render_scene(next_particles)
recon_loss = torch.nn.functional.mse_loss(
torch.tensor(predicted_frame),
torch.tensor(target_frame)
)
kl_loss = -torch.mean(torch.stack(log_probs))
total_loss += recon_loss + 0.01 * kl_loss
return total_loss / seq_len
Step 5: Generation and evaluation
Generate multimodal videos and benchmark on control tasks.
def generate_multimodal_video (model, initial_frame, num_steps=10 , num_samples=4 ):
"""
Generate multiple video trajectories from identical initial frame.
Demonstrates stochasticity from per-particle latent actions.
"""
initial_particles = model.decomposer.decompose_frame(initial_frame)
trajectories = []
for sample_idx in range (num_samples):
particles = initial_particles
frames = [initial_frame]
for step in range (num_steps):
latent_actions = [
model.policies[i].sample_action(
torch.tensor(p.to_state_vector(), dtype=torch.float32)
)
for i, p in enumerate (particles)
]
next_particles = [
Particle.from_state_vector(
model.dynamics.predict_next_state(
torch.tensor(p.to_state_vector(), dtype=torch.float32),
action
).detach().numpy()
)
for p, action in zip (particles, latent_actions)
]
next_frame = model.renderer.render_scene(next_particles)
frames.append(next_frame)
particles = next_particles
trajectories.append(frames)
return trajectories
Practical Guidance Hyperparameter Selection:
Number of particles : 2-6. More = finer detail; higher computational cost.Latent action dimension : 2-8. Higher = more expressive per-particle policies.Appearance embedding dimension : 8-16. Larger = more visual diversity.KL weight : 0.001-0.1. Higher = stronger stochasticity pressure.
Multi-agent or multi-entity scene modeling
Stochastic video generation where different outcomes are valid
Controllable generation with per-entity control
Settings requiring object-centric representations
Single-object scenes (global latent actions sufficient)
Deterministic environments where stochasticity is harmful
Real-time systems requiring fast generation
Scenes with many small, indistinct objects (decomposition becomes ambiguous)
Slot collapse : Some particle slots become inactive. Add auxiliary loss encouraging slot diversity.Poor appearance learning : If appearance embeddings are uninformative, increase embedding dimension or add contrastive loss.Policy convergence : Latent action policies may become deterministic (log_std → -∞). Add entropy regularization.Rendering artifacts : Alpha compositing can cause color oversaturation. Use pre-multiplied alpha or other blending modes.
Reference 