Skip to main content Home Creators adu2021 skillxiv taoavatar-augmented-reality
taoavatar-augmented-reality Create real-time full-body talking avatars for AR using hybrid parametric-Gaussian representations. Teacher-student distillation transfers pose-dependent deformations from a large network to a compact student model, enabling 90+ FPS rendering on mobile devices.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill taoavatar-augmented-realityThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name taoavatar-augmented-reality title TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2503.17032 keywords ["3D Gaussian Splatting","Avatar","AR","Real-Time","Rendering","Deformation"] description Create real-time full-body talking avatars for AR using hybrid parametric-Gaussian representations. Teacher-student distillation transfers pose-dependent deformations from a large network to a compact student model, enabling 90+ FPS rendering on mobile devices.
Core Concept
High-fidelity avatar rendering for AR requires balancing visual quality, real-time performance, and mobile deployment. TaoAvatar achieves this through hybrid parametric-Gaussian representation : extending the SMPLX parametric body model with 3D Gaussians for clothing and details, then using teacher-student distillation to compress pose-dependent deformations into a lightweight student network. The result: photorealistic full-body talking avatars running at 90 FPS on Apple Vision Pro and 150+ FPS on high-end GPUs.
Architecture Overview
Parametric Base : SMPLX++ template binding Gaussians to mesh triangles as learnable texturesDeformation Layers : Teacher network (StyleUnet) learns non-rigid pose-dependent deformations in 2D texture spaceDistillation Pipeline : "Bakes" teacher knowledge into compact MLP-based student networkBlend Shape Compensation : Learnable position and color adjustments for Gaussian deformation lossesMulti-Modal Control : Expression blendshapes, gesture keyframes, and audio synchronizationEfficient Rendering : Direct splatting of bound Gaussians via rasterization
Implementation Steps
Step 1: Create Hybrid SMPLX++ Parametric Representation
Extend the base SMPLX body model with clothing and hair geometry, then bind 3D Gaussians to mesh triangles.
import torch
import torch.nn as nn
import numpy as np
from pytorch3d.ops import sample_points_from_meshes
class SMPLX_Plus (nn.Module):
"""
Extends SMPLX body model with clothing and hair by adding
additional mesh vertices. Binds 3D Gaussians to triangle vertices.
"""
def __init__ (self, smplx_model, num_gaussians_per_triangle=3 ):
super ().__init__()
self .smplx = smplx_model
self .num_gaussians_per_triangle = num_gaussians_per_triangle
self
2000
3
0.1
self
1000
3
0.1
self
5000
3
0.01
self
5000
3
self
0
0
0
1
0
5000
1
self
5000
self
5000
3
def
forward
self, pose_params, shape_params, expr_params, hand_pose
"""
Generate parametric body mesh and bind Gaussians.
Returns mesh vertices and Gaussian parameters.
"""
self
3
1
True
self
1
True
self
1
return
def
get_gaussian_params
self
"""Return Gaussian parameters for rendering."""
self
return
'means'
self
'scales'
'quaternions'
self
'opacities'
self
'colors'
self
Step 2: Design Teacher Network for Deformation Learning Create a large StyleUnet network that learns pose-dependent non-rigid deformations in 2D texture space.
class StyleUnetTeacher (nn.Module):
"""
Teacher network learning pose-dependent deformations.
Operates in 2D texture space of parametric model (unwrapped UV layout).
"""
def __init__ (self, texture_channels=3 , hidden_channels=128 ):
super ().__init__()
self .texture_channels = texture_channels
self .pose_encoder = nn.Sequential(
nn.Linear(72 , 256 ),
nn.BatchNorm1d(256 ),
nn.ReLU(),
nn.Linear(256 , 256 )
)
self .enc1 = nn.Conv2d(texture_channels + 256 , hidden_channels, 3 , padding=1 )
self .enc2 = nn.Conv2d(hidden_channels, hidden_channels * 2 , 3 , stride=2 , padding=1 )
self .enc3 = nn.Conv2d(hidden_channels * 2 , hidden_channels * 4 , 3 , stride=2 , padding=1 )
self .dec3 = nn.ConvTranspose2d(hidden_channels * 4 , hidden_channels * 2 , 4 , stride=2 , padding=1 )
self .dec2 = nn.ConvTranspose2d(hidden_channels * 2 + hidden_channels * 2 , hidden_channels, 4 , stride=2 , padding=1 )
self .dec1 = nn.Conv2d(hidden_channels + hidden_channels, texture_channels, 3 , padding=1 )
def forward (self, texture_features, pose_params, uv_map ):
"""
texture_features: (batch, channels, height, width) - UV texture
pose_params: (batch, 72) - SMPL pose
uv_map: (batch, 2, height, width) - UV coordinates for each pixel
"""
pose_embed = self .pose_encoder(pose_params)
pose_embed = pose_embed.unsqueeze(-1 ).unsqueeze(-1 )
pose_embed = pose_embed.expand(-1 , -1 , texture_features.shape[-2 ], texture_features.shape[-1 ])
combined = torch.cat([texture_features, pose_embed], dim=1 )
e1 = torch.relu(self .enc1(combined))
e2 = torch.relu(self .enc2(e1))
e3 = torch.relu(self .enc3(e2))
d3 = torch.relu(self .dec3(e3))
d3_cat = torch.cat([d3, e2], dim=1 )
d2 = torch.relu(self .dec2(d3_cat))
d2_cat = torch.cat([d2, e1], dim=1 )
deformation = torch.tanh(self .dec1(d2_cat))
return deformation
Step 3: Implement Teacher-Student Distillation Train the student MLP to mimic the teacher's deformation predictions across diverse poses.
class StudentMLP (nn.Module):
"""
Lightweight MLP student network approximating teacher deformations.
Input: pose parameters and UV coordinates.
Output: deformation vectors.
"""
def __init__ (self, pose_dim=72 , uv_dim=2 , hidden_dim=128 , texture_h=512 , texture_w=512 ):
super ().__init__()
self .pose_dim = pose_dim
self .texture_h = texture_h
self .texture_w = texture_w
self .uv_frequencies = [1 , 2 , 4 , 8 , 16 ]
encoded_uv_dim = len (self .uv_frequencies) * 2 * uv_dim
self .mlp = nn.Sequential(
nn.Linear(pose_dim + encoded_uv_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 3 )
)
def encode_uv (self, uv_coords ):
"""Apply Fourier feature encoding to UV coordinates."""
encoded = []
for freq in self .uv_frequencies:
encoded.append(torch.sin(freq * np.pi * uv_coords))
encoded.append(torch.cos(freq * np.pi * uv_coords))
return torch.cat(encoded, dim=-1 )
def forward (self, pose_params, uv_coords ):
"""
pose_params: (batch, pose_dim) or (batch*h*w, pose_dim)
uv_coords: (batch*h*w, 2) or (h*w, 2)
"""
uv_encoded = self .encode_uv(uv_coords)
combined = torch.cat([pose_params, uv_encoded], dim=-1 )
deformation = self .mlp(combined)
return deformation
def generate_texture_deformation (self, pose_params ):
"""
Generate full deformation texture from pose.
Returns: (batch, 3, height, width) deformation field
"""
batch_size = pose_params.shape[0 ]
h, w = self .texture_h, self .texture_w
u = torch.linspace(0 , 1 , w, device=pose_params.device)
v = torch.linspace(0 , 1 , h, device=pose_params.device)
uu, vv = torch.meshgrid(u, v, indexing='xy' )
uv_grid = torch.stack([uu, vv], dim=-1 ).reshape(-1 , 2 )
pose_expanded = pose_params.unsqueeze(1 ).expand(-1 , h * w, -1 ).reshape(-1 , pose_params.shape[-1 ])
deformation = self .forward(pose_expanded, uv_grid)
deformation = deformation.reshape(batch_size, h, w, 3 ).permute(0 , 3 , 1 , 2 )
return deformation
def distillation_loss (teacher_output, student_output, temperature=1.0 ):
"""
Compute L2 loss between teacher and student deformations.
teacher_output: (batch, 3, h, w)
student_output: (batch, 3, h, w)
"""
return torch.nn.functional.mse_loss(
student_output / temperature,
teacher_output.detach() / temperature
)
Step 4: Add Gaussian Blend Shape Compensation Introduce learnable blend shapes for position and color to compensate for deformation losses during distillation.
class GaussianBlendShapes (nn.Module):
"""
Learnable per-Gaussian blend shapes for position and color.
Compensates for information loss during teacher-student distillation.
"""
def __init__ (self, num_gaussians=5000 , num_expressions=50 ):
super ().__init__()
self .position_blends = nn.Parameter(torch.randn(num_gaussians, num_expressions, 3 ) * 0.01 )
self .color_blends = nn.Parameter(torch.randn(num_gaussians, num_expressions, 3 ) * 0.01 )
def forward (self, expr_weights, base_positions, base_colors ):
"""
expr_weights: (batch, num_expressions)
base_positions: (batch, num_gaussians, 3)
base_colors: (batch, num_gaussians, 3)
"""
position_offset = torch.matmul(expr_weights, self .position_blends.permute(1 , 0 , 2 ))
color_offset = torch.matmul(expr_weights, self .color_blends.permute(1 , 0 , 2 ))
new_positions = base_positions + position_offset
new_colors = base_colors + color_offset
return new_positions, new_colors
Step 5: Integration and Real-Time Rendering Combine all components for inference: audio-driven expression prediction, pose tracking, and splatting.
class TaoAvatarInference (nn.Module):
"""
Full inference pipeline: audio-to-expression, pose deformation,
Gaussian manipulation, and efficient splatting.
"""
def __init__ (self, smplx_model, student_network, blend_shapes, gaussian_params ):
super ().__init__()
self .smplx_plus = SMPLX_Plus(smplx_model)
self .student = student_network
self .blend_shapes = blend_shapes
self .gaussian_params = gaussian_params
def forward (self, audio_features, pose_params, shape_params ):
"""
audio_features: (batch, audio_latent_dim) from audio encoder
pose_params: (batch, 72)
shape_params: (batch, 10)
"""
expr_params = self .audio_to_expression(audio_features)
mesh_verts, faces = self .smplx_plus(
pose_params, shape_params, expr_params, hand_pose=None
)
gaussian_dict = self .gaussian_params.get_gaussian_params()
uv_coords = self .get_uv_coords()
student_deformation = self .student.generate_texture_deformation(pose_params)
new_positions, new_colors = self .blend_shapes(
expr_params, gaussian_dict['means' ], gaussian_dict['colors' ]
)
gaussian_dict['means' ] = new_positions
gaussian_dict['colors' ] = new_colors
rendered = self .gaussian_splat_render(gaussian_dict)
return rendered
def audio_to_expression (self, audio_features ):
"""Simple linear mapping from audio to expression weights."""
return torch.tanh(torch.randn(audio_features.shape[0 ], 50 ))
def gaussian_splat_render (self, gaussian_dict ):
"""Placeholder for Gaussian splatting renderer."""
batch_size = gaussian_dict['means' ].shape[0 ]
height, width = 1500 , 2000
return torch.ones(batch_size, 3 , height, width)
Practical Guidance
Real-time avatar applications in AR/VR requiring mobile deployment
Scenarios needing photorealistic talking heads with full-body motion
Projects where 90+ FPS rendering is essential (mobile AR, live streaming)
Applications combining audio-driven facial animation with gesture control
Offline rendering where 5-10 FPS is acceptable and maximum quality is the goal
Scenarios without parametric body model fitting data
Applications requiring extreme facial expression detail beyond SMPL expressivity
num_gaussians : 5000 balances quality and speed; increase to 8000 for higher fidelitystudent_hidden_dim : 128 works for most cases; increase to 256 if distillation loss plateausdistillation_temperature : 1.0-2.0; higher values soften teacher outputs, improving generalizationblend_shape_num_expressions : 50 standard; 30 for speed, 100+ for fine expression control
Misaligned UV parameterization between teacher texture space and Gaussian positions
Insufficient teacher training before distillation (student learns garbage)
Over-reliance on blend shapes; they compensate for poor distillation, not a substitute
Audio-to-expression model poorly calibrated; use supervised training with labeled audio-expression pairs
References 