llm-local-linear-mappings

name	llm-local-linear-mappings
title	Large Language Models are Locally Linear Mappings
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2505.24293
keywords	["Interpretability","LLM Mechanisms","Linear Systems","Transformer Analysis"]
description	Interpret LLM behavior as locally linear mappings between hidden representations, enabling mechanistic understanding of computation without examining individual weights or attention patterns.

Interpret LLMs as Locally Linear Computational Systems

Despite advances in transformer interpretability, understanding how LLMs actually compute remains elusive. This work shows that LLM inference for a given input sequence can be accurately reconstructed as a linear system mapping from input embeddings to output logits. This linearity holds locally (for similar inputs) and provides an interpretable, mechanistic view of LLM computation.

The key insight is that the complex, nonlinear transformer can be well-approximated as a linear map in the region around a specific input. By studying these linear approximations, you can understand what information flows through the model and how it's combined to produce outputs—without analyzing millions of weight matrices or attention patterns individually.

Core Concept

The locally linear model represents LLM computation as:

• Input embedding space: Tokenized text encoded as high-dimensional vectors
• Linear transformation matrix: Maps input embeddings to output logits
• Local linearity: Approximation holds for inputs near the original
• Interpretable weights: Weight values directly show information flow
• Reconstruction error < 10%: Linear model very accurately reproduces predictions
• Mechanistic insight: Understanding this linear map reveals how model processes information

This is profoundly different from weight analysis: instead of examining billions of parameters, you study a single linear map that captures the computation for your input.

Architecture Overview

• Input tokenization and embedding: Standard LLM input processing
• Sequence representation: Tracking how information evolves through layers
• Linear reconstruction module: Fitting linear model to map embeddings → logits
• Error analysis: Measuring where linearity breaks down
• Contribution analysis: Which input elements matter most
• Generalization bounds: Understanding where linear approximation fails
• Visualization and probing: Making the linear structure interpretable

Implementation

Build a framework for analyzing LLMs as locally linear systems:

# Locally Linear LLM Analysis
import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM, AutoTokenizer
from sklearn.linear_model import LinearRegression
import numpy as np

class LocallyLinearLLMAnalyzer:
    """
    Analyze LLM computation as locally linear mappings.
    """
    def __init__(self, model_name: str):
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model.eval()

    def extract_hidden_states(self, prompt: str):
        """
        Extract hidden states at each layer for analysis.
        """
        input_ids = self.tokenizer.encode(prompt, return_tensors='pt')

        with torch.no_grad():
            outputs = self.model(
                input_ids,
                output_hidden_states=True,
                return_dict=True
            )

        hidden_states = outputs.hidden_states  # Tuple of (num_layers + 1, batch, seq, hidden_dim)
        logits = outputs.logits

        return hidden_states, logits, input_ids

    def fit_linear_model(self, prompt: str, candidate_outputs: torch.Tensor = None):
        """
        Fit a linear model from input embeddings to output logits.
        """
        hidden_states, logits, input_ids = .extract_hidden_states(prompt)

        
        input_embeddings = hidden_states[]  
        batch_size, seq_len, hidden_dim = input_embeddings.shape

        
        X = input_embeddings.reshape(batch_size * seq_len, hidden_dim).numpy()

        
        
        y = logits[:, -, :].numpy()  

        
        
        linear_model = LinearRegression()

        
        linear_models = []
        predictions = []

         vocab_idx  (y.shape[-]):
            target = y[:, vocab_idx]

            
            
             batch_size == :
                
                X_samples = []
                y_samples = []

                 _  ():  
                    .model.train()  
                     torch.no_grad():
                        sample_hidden = .extract_hidden_states(prompt)[]
                    sample_embeddings = sample_hidden[, -, :].numpy()  
                    X_samples.append(sample_embeddings)

                    sample_logits = .extract_hidden_states(prompt)[][, -, vocab_idx].numpy()
                    y_samples.append(sample_logits)

                X_regression = np.array(X_samples)
                y_regression = np.array(y_samples)
            :
                X_regression = X
                y_regression = target

            
            model = LinearRegression()
            model.fit(X_regression, y_regression)
            linear_models.append(model)

            
            pred = model.predict(X_regression)
            predictions.append(pred)

        predictions = np.array(predictions).T

         linear_models, X, y

     ():
        
        hidden_states, logits, input_ids = .extract_hidden_states(prompt)

        
        actual_logits = logits[:, -, :].detach().cpu().numpy()

        
        linear_models, X, _ = .fit_linear_model(prompt)

        
        input_embeddings = hidden_states[][:, -, :].numpy()
        predicted_logits = np.array([
            model.predict(input_embeddings.reshape(, -))[]
             model  linear_models
        ])

        
        mse = np.mean((actual_logits - predicted_logits) ** )
        relative_error = mse / (np.mean(actual_logits ** ) + )

         {
            : mse,
            : relative_error,
            : predicted_logits,
            : actual_logits
        }

     ():
        
        linear_models, X, _ = .fit_linear_model(prompt)

        
        
        weights = np.array([model.coef_  model  linear_models])  

        
        importance = np.(weights).(axis=)  
        important_indices = np.argsort(importance)[-:]  

         {
            : weights,
            : importance,
            : important_indices,
            : importance[important_indices]
        }

     ():
        
        results = []

        
        linear_models, X_train, y_train = .fit_linear_model(prompts[])

        
         prompt  prompts[:]:
            hidden_states, logits, _ = .extract_hidden_states(prompt)
            input_embeddings = hidden_states[][:, -, :].numpy()
            actual_logits = logits[:, -, :].detach().cpu().numpy()

            
            predictions = np.array([
                model.predict(input_embeddings.reshape(, -))[]
                 model  linear_models
            ])

            error = np.mean((actual_logits - predictions) ** )
            results.append({: prompt, : error})

         results

Implement analysis visualization and interpretation tools:

def interpret_linear_computation(analyzer: LocallyLinearLLMAnalyzer, prompt: str):
    """
    Interpret what the linear model reveals about LLM computation.
    """
    # Measure linearity
    linearity = analyzer.measure_linearity_error(prompt)
    print(f"Linearity Error (relative): {linearity['relative_error']:.2%}")
    print(f"Model reconstructs output with {100 * (1 - linearity['relative_error']):.1f}% accuracy")

    # Analyze information flow
    flow = analyzer.analyze_information_flow(prompt)
    print(f"\nTop 5 important input dimensions: {flow['important_dimensions'][:5]}")

    # Generalization analysis
    related_prompts = generate_related_prompts(prompt, num_variations=5)
    generalization = analyzer.compare_linear_approximation([prompt] + related_prompts)

    avg_gen_error = np.mean([r['generalization_error'] for r in generalization[1:]])
    print(f"\nGeneralization error on similar prompts: {avg_gen_error:.4f}")

    return {
        'linearity': linearity,
        'information_flow': flow,
        'generalization': generalization
    }

def generate_related_prompts(prompt: str, num_variations: int) -> :
    
    
    variations = [
        prompt,
        prompt.replace(, ),  
         + prompt,
        prompt + ,
    ]
     variations[:num_variations]

 () -> :
    
    errors = []
    generalization_distances = []

     prompt  test_prompts:
        error = analyzer.measure_linearity_error(prompt)
        errors.append(error[])

    
    mean_error = np.mean(errors)
    std_error = np.std(errors)

    ()
    ()
    ()
    ()

     {
        : mean_error,
        : std_error,
        : np.percentile(errors, ),
        : errors
    }

 ():
    
    hidden_states, logits, input_ids = analyzer.extract_hidden_states(prompt)
    linear_models, X, _ = analyzer.fit_linear_model(prompt)

    
    weights = linear_models[target_token].coef_  

    
    top_indices = np.argsort(np.(weights))[-:]

    explanation = 
     explanation

Practical Guidance

Aspect	Recommendation	Notes
Number of samples for fitting	10 - 100	More samples = better fit, but more compute
Dimensionality of hidden states	Full	Don't reduce; linearity relies on full expressivity
Distance for local validity	Small ε	Linearity breaks down as you move far from base point
Relative error threshold	<10%	Linearity approximation valid if error is small
Generalization distance	Same model family	Linear fit may not transfer to different models

When to use locally linear analysis:

• Need mechanistic understanding of LLM computation
• Want interpretable explanation of specific predictions
• Studying how information flows through models
• Analyzing failure modes or adversarial examples
• Building approximations or distillations

When NOT to use:

• Only care about final accuracy (interpretability not needed)
• Models exhibit strong non-linear behavior (check linearity error first)
• Computational budget for fitting linear models is tight
• Need global understanding (linearity is local, not global)
• Studying learned representations independent of computation

Common pitfalls:

• Assuming linearity holds globally (only true locally)
• Fitting on too few samples (unstable linear models)
• Not measuring reconstruction error (can't validate linearity)
• Using raw logits instead of normalizing (scale sensitivity)
• Generalizing insights beyond neighborhood of fitting point
• Confusing linear approximation with actual mechanism (still approximate)
• Not comparing to baselines (what's the accuracy gain from interpretability?)

Reference

Large Language Models are Locally Linear Mappings https://arxiv.org/abs/2505.24293