zip2zip-adaptive-vocab

name	zip2zip-adaptive-vocab
title	zip2zip: Inference-Time Adaptive Vocabularies for Language Models via Token Compression
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.01084
keywords	["token compression","adaptive vocabulary","LLM inference","Lempel-Ziv","efficiency"]
description	Reduce token count by 15-40% at inference through context-adaptive compression, merging frequent token sequences into hypertokens using online Lempel-Ziv-Welch compression without retraining entire models.

zip2zip: Inference-Time Adaptive Vocabularies for Language Models via Token Compression

Core Concept

zip2zip addresses the inefficiency of static, general-purpose tokenizers in language models. Rather than using a fixed vocabulary optimized for average cases, zip2zip dynamically adapts tokenization at inference time by identifying and merging frequently co-occurring token sequences into compact "hypertokens."

The approach uses Lempel-Ziv-Welch (LZW) compression on-the-fly to detect compression patterns in the current context and create hypertokens that match the input's actual token distribution. A lightweight dynamic embedding layer computes representations for newly formed hypertokens at runtime, reducing both input and output token counts by 15-40% without full model retraining.

Architecture Overview

• Online LZW Compression: Dynamically identify and merge frequent token sequences during inference
• Hypertokenization: Replace multiple fragmented tokens with single merged tokens
• Dynamic Embedding Layer: Compute embeddings for novel hypertokens without precomputed vocabulary
• Context-Adaptive Encoding: Adjust compression strategy to match current input distribution
• Minimal Retraining: Fine-tune embedding layer in ~10 GPU-hours; preserve base model weights
• Seamless Integration: Works with existing LLM architectures (Transformers, etc.)

Implementation

The following steps outline how to implement adaptive tokenization via online compression:

1. Monitor token stream - Track input tokens as they arrive during inference
2. Detect compression patterns - Apply LZW algorithm to identify frequent sequences
3. Merge tokens into hypertokens - Create new token IDs for merged sequences
4. Compute hypertoken embeddings - Dynamically generate representations using a lightweight layer
5. Process through model - Run standard model inference with hypertokens
6. Decompress output - Convert hypertokens back to original tokens for downstream use

import torch
import torch.nn as nn
from typing import List, Tuple, Dict
 collections  defaultdict

 :
     ():
        .vocab_size = vocab_size
        .max_code = max_code
        .dictionary = defaultdict()
        .reset()

     ():
        
        .dictionary = {i: (i)  i  (.vocab_size)}
        .next_code = .vocab_size

     () -> []:
        
          token_ids:
             []

        compressed = []
        w = (token_ids[])

         token  token_ids[:]:
            wc = w +  + (token)
             wc  .dictionary:
                w = wc
            :
                compressed.append(._lookup(w))
                 .next_code < .max_code:
                    .dictionary[wc] = .next_code
                    .next_code += 
                w = (token)

        compressed.append(._lookup(w))
         compressed

     () -> :
        
         .dictionary.get(code_str, (code_str.split()[]))


 (nn.Module):
     ():
        ().__init__()
        .base_vocab_size = base_vocab_size
        .embedding_dim = embedding_dim
        .base_embeddings = nn.Embedding(base_vocab_size, embedding_dim)
        
        .hypertoken_projection = nn.Linear(embedding_dim * , embedding_dim)

     () -> torch.Tensor:
        
         token_id < .base_vocab_size:
             .base_embeddings(torch.tensor(token_id))
         component_ids:
            
            component_embeddings = [.base_embeddings(torch.tensor(cid))  cid  component_ids]
            combined = torch.cat(component_embeddings[:], dim=-)  
             .hypertoken_projection(combined)
        :
             .base_embeddings(torch.tensor(.base_vocab_size - ))  

     () -> torch.Tensor:
        
        embeddings = []
         tid  token_ids:
            tid_val = tid.item()  (tid, torch.Tensor)  tid
             hypertoken_map  tid_val  hypertoken_map:
                embeddings.append(.get_embedding(tid_val, hypertoken_map[tid_val]))
            :
                embeddings.append(.get_embedding(tid_val))
         torch.stack(embeddings)


 :
     ():
        .compressor = LZWCompressor(base_vocab_size)
        .embedding_layer = DynamicEmbedding(base_vocab_size, embedding_dim)
        .hypertoken_map = {}

     () -> [[], ]:
        
        compressed = .compressor.compress_sequence(token_ids)
        
        .hypertoken_map = ._build_mapping(token_ids, compressed)
         compressed, .hypertoken_map

     () -> [, []]:
        
        mapping = {}
        
         mapping