calm-continuous-autoregressive-language-models

name	calm-continuous-autoregressive-language-models
title	Continuous Autoregressive Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.27688
keywords	["Continuous Embeddings","Autoregressive Models","Token Compression","Likelihood-Free","Energy Scoring"]
description	Replace discrete token prediction with continuous vector prediction by training a high-fidelity autoencoder to compress K tokens into single latent vectors, enabling K-fold sequence length reduction while maintaining likelihood-free generation through energy-based scoring rules.

Title: Predict Continuous Token-Chunk Vectors for Efficient Language Generation

Traditional autoregressive language models predict one discrete token at a time, requiring billions of forward passes for lengthy generations. CALM (Continuous Autoregressive Language Model) reduces generation steps by a factor of K by compressing K tokens into a single continuous vector, then predicting vectors directly. The key innovation is a likelihood-free training framework using energy scoring that enables stable continuous-space learning without explicit probability densities.

The approach combines a high-fidelity variational autoencoder for token compression with an energy-based generative head that uses strictly proper scoring rules for training.

Core Concept

Continuous-Space Next-Vector Prediction:

• Compress K consecutive tokens into a single latent vector via a learned autoencoder
• Train the model to predict next latent vector autoregressively: p(Z) = ∏p(zᵢ|z<i)
• Use energy scoring (likelihood-free) instead of explicit probability distributions
• K-fold reduction in sequence length with >99.9% reconstruction fidelity

This sidesteps diffusion or flow-matching complexity by using energy-based models that evaluate alignment between predictions and observations through sample distances rather than probability densities.

Architecture Overview

• Autoencoder: High-fidelity token compression (VQ-VAE style, ~75M parameters)
• Variational Regularization: KL divergence smoothing latent space with posterior collapse prevention
• Energy Transformer Head: Residual MLP blocks (10% of model parameters) generating predictions via energy scoring
• Strictly Proper Scoring: Energy Score for training, BrierLM for evaluation
• Two-Stage Training: Autoencoder pre-training, then CALM model training

Implementation Steps

1. Train High-Fidelity Autoencoder

Build a symmetric encoder-decoder that compresses K=4 tokens into l=128-dimensional latent vectors. The encoder maps token embeddings through feed-forward networks and flattens to a latent vector. Use variational regularization with KL clipping to prevent posterior collapse.

# Autoencoder architecture
class TokenAutoencoder(nn.Module):
    def __init__(self, vocab_size, hidden_dim=, latent_dim=):
        .encoder = nn.Sequential(
            nn.Linear(vocab_size, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, latent_dim * )  
        )
        .decoder = nn.Linear(latent_dim, vocab_size)

     ():
        
        embeddings = .embed(token_ids)
        chunks = embeddings.reshape(-, k, -)
        
        z_params = .encoder(chunks.flatten())
        z = z_params[:, :.latent_dim]
         z

     ():
        logits = .decoder(z)
         logits