grape-group-position-encoding

name	grape-group-position-encoding
title	Group Representational Position Encoding
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.07805
keywords	["position embeddings","group actions","rotary embeddings","long-context transformers","attention mechanisms"]
description	Unify positional encoding methods via group action theory, encompassing RoPE and ALiBi as special cases. GRAPE enables exploration of cross-subspace feature coupling—ideal when you need principled positional encoding beyond standard implementations.

Overview

GRAPE provides a unified mathematical framework for positional encoding based on group actions. It encompasses two mechanism families: multiplicative rotations in SO(d) and additive logit biases from unipotent actions in GL. This unification reveals that RoPE and ALiBi are special cases of a broader framework, enabling theoretical understanding and novel extensions.

When to Use

• Designing new positional encoding methods beyond RoPE and ALiBi
• Understanding theoretical properties of position encodings
• Exploring cross-subspace feature coupling in attention
• Research into positional encoding mechanisms
• Applications requiring principled positional encoding foundations

When NOT to Use

• Standard RoPE implementations already meeting needs
• Applications where existing methods work adequately
• Scenarios where theoretical elegance doesn't improve practical performance
• Real-time inference requiring minimal overhead

Core Technique

Group action theory for unified positional encoding:

# Group Representational Position Encoding
class GroupRepresentationalPositionalEncoding:
    def __init__(self, dim, max_seq_len=2048):
        self.dim = dim
        self.max_seq_len = max_seq_len

    # Multiplicative GRAPE using SO(d) rotations
    def multiplicative_grape(self, x, positions):
        """
        Multiplicative position-dependent rotation matrices.
        Uses rank-2 skew-symmetric generators for efficient computation.
        """
        batch_size, seq_len, dim = x.shape

        # Generate skew-symmetric matrices
        # L is skew-symmetric generator (antisymmetric)
        omega = self.compute_omega(dim)  # Base frequency matrix
        L = self.compute_skew_symmetric(omega)

        # Matrix exponential: G(n) = exp(n * omega * L)
        # For position n, compute rotation matrix
        rotation_matrices = []
         position  (seq_len):
            
            exp_matrix = torch.linalg.matrix_exp(position * L)
            rotation_matrices.append(exp_matrix)

        rotation_matrices = torch.stack(rotation_matrices)

        
        
        output = torch.einsum(
            ,
            rotation_matrices,
            x
        )

        
         output

     ():
        
        
        
        dim = omega.shape[]
        L = torch.zeros((dim, dim))

        
         i  (dim):
             j  (i+, dim):
                L[i, j] = omega[i, j]
                L[j, i] = -omega[i, j]

         L

    
     ():
        
        
        

        batch_size, seq_len, dim = logits.shape

        
        position_bias = torch.zeros(seq_len, seq_len)

         q_pos  (seq_len):
             k_pos  (seq_len):
                
                distance = q_pos - k_pos

                
                

                
                

                
                bias_value = .compute_unipotent_bias(distance)
                position_bias[q_pos, k_pos] = bias_value

        
        
        output_logits = logits + position_bias.unsqueeze()

         output_logits

     ():
        
         distance == :
             

        
        
        slope =   

        
        linear_bias = -(distance) * slope

        
        exp_bias = -slope * ( - torch.exp(torch.tensor(distance * )))

        
        alpha =   
        mixed_bias = alpha * linear_bias + ( - alpha) * exp_bias

         mixed_bias

    
     ():
        
        batch_size, seq_len, total_dim = x.shape
        num_heads = total_dim // head_dim

        
        heads = x.reshape(batch_size, seq_len, num_heads, head_dim)

        
        encoded_heads = []
         head_idx  (num_heads):
            head = heads[:, :, head_idx, :]

            
            positions = torch.arange(seq_len)
             head_idx %  == :
                
                encoded = .multiplicative_grape(head, positions)
            :
                
                encoded = .additive_grape(head, positions)

            encoded_heads.append(encoded)

        
        output = .couple_heads(encoded_heads, head_dim)

         output

     ():
        
        
        stacked = torch.stack(heads, dim=)
        batch_size, seq_len, num_heads, head_dim = stacked.shape

        
        coupling_matrix = .compute_coupling_weights(num_heads)

        
        coupled = torch.einsum(
            ,
            coupling_matrix,
            stacked
        )

        
        output = coupled.reshape(batch_size, seq_len, -)

         output