layer-cake-contrastive-decoding

name	layer-cake-contrastive-decoding
title	LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.04404
keywords	["Decoding","Contrastive Learning","Factuality","Token-Aware","No-Training Required"]
description	Improve factual accuracy in LLM generation through decoding-time layer-wise attention suppression. Selectively suppress attention to specific token types at their most influential transformer depths without training or model modifications. Use when you need to reduce hallucinations and improve factual correctness at inference time.

LayerCake: Token-Aware Contrastive Decoding for Enhanced Factuality

Large language models frequently generate factual errors and hallucinations despite strong pretraining. LayerCake addresses this through a novel decoding-time approach that exploits the internal structure of Transformers. The key observation is that different token types (punctuation, concepts, entities) have dominant influence at specific layer depths: early layers handle surface-level tokens like punctuation, while intermediate layers drive semantic reasoning.

By selectively suppressing attention to token types at their most influential depths, the method creates contrastive signals that guide generation toward factual outputs. The approach requires no training, no model modifications, and no fine-tuning—it operates purely at decoding time through attention manipulation.

Core Concept

Transformers process information hierarchically: early layers capture surface patterns (punctuation, formatting), middle layers reason about concepts and relationships, and deeper layers synthesize high-level decisions. LayerCake identifies which token types exert maximum influence at each depth, then strategically suppresses attention to those tokens at that depth.

This creates a controlled factual degradation that generates contrastive signals: the model learns which token types are critical for accurate generation and avoids over-relying on spurious correlations. The technique combines two insights: (1) token types have differential importance across layers, and (2) suppressing specific signals at their critical depths causes the model to find alternative, more robust reasoning paths.

Architecture Overview

• Attention Pattern Analysis: Identify which token categories receive dominant attention at each transformer layer (punctuation in early layers, concepts in middle)
• Layer-Token Importance Mapping: Build per-layer matrices showing influence of each token type on output quality
• Selective Suppression Module: At inference, suppress attention to identified token types at their critical depths
• Contrastive Signal Generation: Controlled suppression creates factual gaps that guide model toward more grounded generation
• Zero-Training Design: All operations at decoding time without model parameter updates

Implementation

Attention Pattern Analysis Across Layers

Analyze which tokens receive maximum attention at each transformer depth.

import torch
import torch.nn.functional as F
from typing import , , 
 transformers  AutoModelForCausalLM, AutoTokenizer

 :
    

     ():
        .model = AutoModelForCausalLM.from_pretrained(model_name, output_attentions=)
        .tokenizer = AutoTokenizer.from_pretrained(model_name)
        .model.()

        
        .token_types = {
            : .identify_punctuation_tokens(),
            : .identify_common_words(),
            : .identify_entity_indicators(),
            : .identify_concept_words()
        }

     () -> :
        
        punctuation = ()
         token_id  (.tokenizer.vocab_size):
            token_str = .tokenizer.decode([token_id])
             (token_str.strip()) <=    token_str.isalnum():
                punctuation.add(token_id)
         punctuation

     () -> :
        
        common_ids = (.tokenizer.convert_tokens_to_ids(
            [, , , , , , , , , , ]
        ))
         common_ids

     () -> :
        
        entity_indicators = ()
         token_id  ((.tokenizer.vocab_size, )):
            token_str = .tokenizer.decode([token_id])
             (token_str) >   token_str[].isupper():
                entity_indicators.add(token_id)
         entity_indicators

     () -> :
        
        
        
        concept_ids = ()
         token_id  ((.tokenizer.vocab_size, )):
            token_str = .tokenizer.decode([token_id]).strip()
             (token_str) >=   token_str.isalpha():
                concept_ids.add(token_id)
         concept_ids

     () -> [, [, ]]:
        
        input_ids = .tokenizer.encode(prompt, return_tensors=)

         torch.no_grad():
            outputs = .model(input_ids, output_attentions=)
            attentions = outputs.attentions  

        layer_importance = {}

         layer_idx  (((attentions), num_layers_to_analyze)):
            layer_attn = attentions[layer_idx]  
            batch_size, num_heads, seq_len, _ = layer_attn.shape

            
            avg_attn = layer_attn.mean(dim=(, ))  

            
            token_importance = avg_attn.(dim=)  

            
            type_importance = {}

             token_type, token_ids  .token_types.items():
                type_scores = []

                 pos, token_id  (input_ids[]):
                     token_id.item()  token_ids:
                        type_scores.append(token_importance[pos].item())

                type_importance[token_type] = (
                    (type_scores) / (type_scores)  type_scores  
                )

            layer_importance[layer_idx] = type_importance

         layer_importance