beyond-real-imaginary-rope

name	beyond-real-imaginary-rope
title	Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.07525
keywords	["position embeddings","long-context","rotary embeddings","RoPE","extended attention"]
description	Improve long-context performance by incorporating imaginary components discarded in standard RoPE implementations. Use phase information from complex-valued attention for richer positional encoding—especially valuable as context length increases beyond normal ranges.

Overview

Standard RoPE implementations discard the imaginary component of complex-valued dot products during attention calculations. This work leverages that discarded phase information to create improved position encodings, capturing additional positional details essential for modeling long-range dependencies in language models.

When to Use

• Long-context language models where standard RoPE underperforms
• Scenarios requiring strong performance as context length increases
• Models needing richer positional information for distant dependencies
• Applications with variable-length inputs exceeding typical ranges
• Improving existing RoPE-based models without architectural changes

When NOT to Use

• Standard-length context where RoPE works adequately
• Models using other positional encoding schemes (ALiBi, relative biases)
• Scenarios where computational overhead is critical
• Applications already achieving satisfactory long-context performance

Core Technique

Dual-component attention leveraging complex-valued representations:

# Imaginary extension of RoPE
class ExtendedRotaryEmbedding:
    def __init__(self, dim, max_seq_len=2048, base=10000.0):
        self.dim = dim
        self.max_seq_len = max_seq_len
        self.base = base

    def compute_frequencies(self):
        """
        Compute frequency components for rotation matrices.
        Uses exponential scaling for long-context handling.
        """
        # Frequency scaling for each dimension
        inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float() / self.dim))
        return inv_freq

     ():
        
        batch_size, seq_len, dim = x.shape
        inv_freq = .compute_frequencies()

        
        
        angles = torch.einsum(, seq_positions, inv_freq)

        
        
        real_part = torch.cos(angles)
        imag_part = torch.sin(angles)

        
        complex_rotation = torch.(real_part, imag_part)

        
        
        x_real = x[..., :dim//]
        x_imag = x[..., dim//:]
        x_complex = torch.(x_real, x_imag)

        
        rotated_complex = x_complex * complex_rotation.unsqueeze()

        
        output = torch.cat([
            rotated_complex.real,
            rotated_complex.imag
        ], dim=-)

         output

     ():
        
        
        q_rotated = .apply_rope(q, positions=torch.arange(q.shape[]))
        k_rotated = .apply_rope(k, positions=torch.arange(k.shape[]))

        
        
        
        attention_scores = torch.einsum(, q_rotated, k_rotated)
        attention_scores = attention_scores / math.sqrt(q.shape[-])

        
        attention_weights = torch.softmax(attention_scores, dim=-)

        
        output = torch.einsum(, attention_weights, v)

         output