arc-encoder-compression

name	arc-encoder-compression
title	ARC-Encoder: learning compressed text representations for LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.20535
keywords	["Compression","Efficiency","Inference","Encoder","Embeddings"]
description	Reduces inference cost by compressing context into continuous representations using a separate encoder. Generates 4-8x fewer representations than token embeddings while maintaining model performance. Works with any decoder LLM without modification or fine-tuning.

ARC-Encoder: Efficient Text Compression for LLM Inference

Long contexts increase inference latency and memory. ARC-Encoder compresses text into dense continuous representations that substitute token embeddings, reducing sequence length without sacrificing language model performance.

The encoder generalizes across decoder models, enabling a single compression component to work with multiple LLMs.

Core Concept

Key innovation: replace token-level embeddings with compressed continuous representations:

• Separate encoder compresses context into continuous embeddings
• Outputs 4-8x fewer representations than input tokens
• Maintains semantic information despite aggressive compression
• Compatible with any pretrained decoder without architecture changes

This decouples compression from language modeling, enabling reuse across models.

Architecture Overview

• Text encoder (e.g., BERT-style or custom architecture)
• Compression ratio design (typically 4x or 8x)
• Continuous representation substitution in decoder embeddings
• Optional dimension reduction for efficiency

Implementation Steps

Build an encoder that compresses arbitrary text into dense fixed-size representations. The encoder should preserve semantic content while reducing cardinality:

class TextCompressionEncoder(nn.Module):
    def __init__(self, input_dim=768, output_dim=768, compression_ratio=4):
        super().__init__()
        self.compression_ratio = compression_ratio

        # Encoder: standard transformer blocks
        self.encoder = TransformerEncoder(
            d_model=input_dim,
            num_layers=4,
            num_heads=12
        )

        # Compression projection
        self.compress_proj = nn.Linear(input_dim * compression_ratio, output_dim)
        self.norm = nn.LayerNorm(output_dim)

    def forward():
        
        
        encoded = .encoder(token_embeddings)

        
        seq_len = encoded.shape[]
        
        grouped = []
         i  (, seq_len, .compression_ratio):
            chunk = encoded[:, i:i+.compression_ratio, :]
            
            chunk_flat = chunk.reshape(chunk.shape[], -)
            compressed = .compress_proj(chunk_flat)
            grouped.append(compressed)

        
        output = torch.stack(grouped, dim=)
        output = .norm(output)

         output