Combine diffusion-based parallel drafting with autoregressive sampling in a single forward pass using structured attention masks—achieving 5x+ token throughput while maintaining autoregressive-level output quality through hybrid generation.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Combine diffusion-based parallel drafting with autoregressive sampling in a single forward pass using structured attention masks—achieving 5x+ token throughput while maintaining autoregressive-level output quality through hybrid generation.
Hybrid Diffusion-Autoregressive Generation for High-Throughput Language Models
Autoregressive language models are high-quality but slow (one token per forward pass). Diffusion models generate tokens in parallel but with lower quality. TiDAR merges both paradigms: diffusion generates candidate tokens in parallel (drafting phase), then autoregression selects final outputs sequentially (refinement phase)—all in a single forward pass using structured attention.
The approach achieves 4.71x to 5.91x tokens per second compared to pure autoregression while maintaining comparable quality, solving a fundamental speed-quality tradeoff.
Core Concept
TiDAR operates in two phases within one neural forward pass:
Thinking (Diffusion) - Parallel iterative refinement generates k candidate tokens for each position
Talking (Autoregression) - Sequential sampling selects final tokens using context and diffusion candidates
Structured attention masks enable this hybrid within a single transformer: diffusion layers have all-to-all connectivity (parallel thinking), while autoregressive layers have causal masks (sequential talking). The architecture transitions smoothly between thinking and talking phases.
Architecture Overview
Diffusion Thinking Layers: Parallel token generation with iterative refinement
Structured Attention Masks: All-to-all for diffusion; causal for autoregression
Candidate Representation: Stores k candidate tokens per position for AR selection
Autoregressive Refinement: Sequential sampling from diffusion candidates
Hybrid Router: Decides when to transition from diffusion to autoregressive phase
Efficient Masking: Single forward pass enables gradient flow through both paradigms
Implementation Steps
Step 1: Diffusion-Based Candidate Generation
Generate multiple token candidates per position through iterative diffusion.
self, vocab_size: int, embed_dim: int, num_candidates: int = 8,
num_diffusion_steps: int = 4
"""
Args:
vocab_size: Size of vocabulary
embed_dim: Embedding dimension
num_candidates: Number of candidate tokens per position
num_diffusion_steps: Refinement iterations
"""
super
self
self
self
self
# Learnable noise scheduler
self
1.0
0.0
# Refinement layers
self
for
in
range
def
generate_candidates
self, hidden_states: torch.Tensor
"""
Generate candidate tokens through diffusion.
Args:
hidden_states: Model hidden states [batch_size, seq_len, embed_dim]
Returns:
candidates: Candidate logits [batch_size, seq_len, num_candidates, vocab_size]
"""
# Initialize candidates with noise
# Start from uniform random; refine toward true distribution
self
# Iterative refinement (diffusion steps)
for
in
range
self
self
# Refine candidates with context
# Attend to hidden states to bias candidates toward relevant tokens
2
# Add context bias
self
# Gradually remove noise
1
# Project to logits
# Map embedding space to vocabulary
self
return
Step 2: Structured Attention Masks
Create attention patterns enabling diffusion (all-to-all) and autoregression (causal) in one forward pass.
defcreate_hybrid_attention_mask(batch_size: int, seq_len: int, num_candidates: int,
diffusion_layers: int, ar_layers: int,
device: torch.device) -> Dict[str, torch.Tensor]:
"""
Create structured attention masks for hybrid architecture.
Args:
batch_size: Batch size
seq_len: Sequence length
num_candidates: Number of candidates per position
diffusion_layers: Number of diffusion (all-to-all) layers
ar_layers: Number of autoregressive (causal) layers
device: torch device
Returns:
masks: {diffusion_mask, ar_mask, candidate_mask}
"""# Diffusion mask: all-to-all connectivity (thinking phase)# Every position can attend to every other position
diffusion_mask = torch.ones(
batch_size, seq_len, seq_len,
device=device, dtype=torch.bool
)
# Autoregressive mask: causal (talking phase)# Each position attends to itself and previous positions only
ar_mask = torch.tril(
torch.ones(seq_len, seq_len, device=device, dtype=torch.bool)
).unsqueeze(0).expand(batch_size, -1, -1)
# Candidate mask: connections between AR positions and diffusion candidates# AR layer can attend to all candidate positions from previous step
candidate_mask = torch.ones(
batch_size, seq_len, num_candidates, seq_len,
device=device, dtype=torch.bool
)
return {
'diffusion_mask': diffusion_mask,
'ar_mask': ar_mask,
'candidate_mask': candidate_mask
}
classHybridAttentionLayer(nn.Module):
"""
Single attention layer supporting both diffusion and AR patterns.
"""def__init__(self, embed_dim: int, num_heads: int = 8):
super().__init__()
self.embed_dim = embed_dim
self.num_heads = num_heads
self.head_dim = embed_dim // num_heads
self.query = nn.Linear(embed_dim, embed_dim)
self.key = nn.Linear(embed_dim, embed_dim)
self.value = nn.Linear(embed_dim, embed_dim)
self.output = nn.Linear(embed_dim, embed_dim)
defforward(self, hidden_states: torch.Tensor, attention_mask: torch.Tensor,
phase: str = 'diffusion') -> torch.Tensor:
"""
Apply hybrid attention.
Args:
hidden_states: [batch, seq_len, embed_dim]
attention_mask: Attention mask (diffusion or AR)
phase: 'diffusion' or 'autoregressive'
Returns:
output: Attended states [batch, seq_len, embed_dim]
"""
batch_size, seq_len, embed_dim = hidden_states.shape
# Compute Q, K, V
Q = self.query(hidden_states)
K = self.key(hidden_states)
V = self.value(hidden_states)
# Reshape for multi-head attention
Q = Q.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
K = K.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
V = V.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
# Scaled dot-product attention
scores = torch.matmul(Q, K.transpose(-2, -1)) / (self.head_dim ** 0.5)
# Apply maskif attention_mask isnotNone:
# Mask shape: [batch, 1, seq_len, seq_len]
scores = scores.masked_fill(~attention_mask.unsqueeze(1), float('-inf'))
# Softmax and dropout
attn_weights = F.softmax(scores, dim=-1)
attn_output = torch.matmul(attn_weights, V)
# Reshape back
attn_output = attn_output.transpose(1, 2).contiguous()
attn_output = attn_output.view(batch_size, seq_len, embed_dim)
# Output projection
output = self.output(attn_output)
return output
Step 3: Autoregressive Refinement from Candidates
Select final tokens from diffusion candidates using autoregressive sampling.
classAutoregressiveRefinement(nn.Module):
"""
Refines diffusion candidates through autoregressive sampling.
"""def__init__(self, vocab_size: int, embed_dim: int):
super().__init__()
self.vocab_size = vocab_size
self.embed_dim = embed_dim
# Selection network: learns to pick best candidateself.selector = nn.Sequential(
nn.Linear(embed_dim + vocab_size, embed_dim),
nn.ReLU(),
nn.Linear(embed_dim, 1)
)
defselect_tokens(self, candidates_logits: torch.Tensor,
context_hidden: torch.Tensor) -> torch.Tensor:
"""
Select best candidate tokens given context.
Args:
candidates_logits: [batch, seq_len, num_candidates, vocab_size]
context_hidden: [batch, seq_len, embed_dim]
Returns:
selected_tokens: [batch, seq_len, vocab_size]
"""
batch_size, seq_len, num_candidates, vocab_size = candidates_logits.shape
# Compute candidate probabilities
candidate_probs = F.softmax(candidates_logits, dim=-1)
# Score each candidate position
scores = []
for c inrange(num_candidates):
# Get probabilities for this candidate set
cand_probs = candidate_probs[:, :, c, :]
# Combine with context
combined = torch.cat([
context_hidden,
cand_probs
], dim=-1)
# Compute selection score
score = self.selector(combined).squeeze(-1)
scores.append(score)
scores = torch.stack(scores, dim=-1) # [batch, seq_len, num_candidates]# Select highest-scoring candidate per position
selected_idx = torch.argmax(scores, dim=-1) # [batch, seq_len]# Gather selected logits
selected_logits = torch.gather(
candidates_logits,
2,
selected_idx.unsqueeze(-1).unsqueeze(-1).expand(-1, -1, 1, vocab_size)
).squeeze(2)
return selected_logits
Step 4: Unified Forward Pass
Combine diffusion thinking and AR talking into single forward pass.
