long-context

name	long-context
description	Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.
category	llm-tools
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Emerging Techniques","Long Context","RoPE","YaRN","ALiBi","Position Interpolation","Extended Context","Rotary Embeddings","Attention Bias","Context Extension","Positional Encoding"]
dependencies	["transformers","torch","flash-attn"]

Long Context: Extending Transformer Context Windows

When to Use This Skill

Use Long Context techniques when you need to:

• Process long documents (32k, 64k, 128k+ tokens) with transformer models
• Extend context windows of pre-trained models (LLaMA, Mistral, etc.)
• Implement efficient positional encodings (RoPE, ALiBi)
• Train models with length extrapolation capabilities
• Deploy models that handle variable-length inputs efficiently
• Fine-tune existing models for longer contexts with minimal compute

Key Techniques: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation

Papers: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)

Installation

# HuggingFace Transformers (includes RoPE, YaRN support)
pip install transformers torch

# For custom implementations
pip install einops  # Tensor operations
pip install rotary-embedding-torch  # Standalone RoPE

# Optional: FlashAttention for efficiency
pip install flash-attn --no-build-isolation

Quick Start

RoPE (Rotary Position Embeddings)

import torch
import torch.nn as nn

class RotaryEmbedding(nn.Module):
    """Rotary Position Embeddings (RoPE)."""

    def __init__(self, dim, max_seq_len=8192, base=10000):
        super().__init__()
        # Compute inverse frequencies
        inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
        self.register_buffer(, inv_freq)
        .max_seq_len = max_seq_len

     ():
        
        t = torch.arange(seq_len, device=device).type_as(.inv_freq)

        
        freqs = torch.outer(t, .inv_freq)  

        
        emb = torch.cat((freqs, freqs), dim=-)  
         emb.cos(), emb.sin()

 ():
    
    x1, x2 = x.chunk(, dim=-)
     torch.cat((-x2, x1), dim=-)

 ():
    
    
    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)
     q_embed, k_embed


rope = RotaryEmbedding(dim=, max_seq_len=)
cos, sin = rope(seq_len=, device=)


q_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)