lightweight-memory-augmented

name	lightweight-memory-augmented
title	LightMem: Lightweight and Efficient Memory-Augmented Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.18866
keywords	["memory augmentation","efficient generation","token reduction","context compression","RAG"]
description	Reduce inference tokens by up to 38× and API calls by 30× through three-stage memory system: sensory compression, short-term consolidation, and offline long-term storage.

Technique: Three-Stage Memory System — Efficient Context Management

Multi-turn conversations and long-horizon tasks require remembering context, but naive approaches accumulate tokens exponentially—each turn adds conversation history, doubling context length. LightMem addresses this through a three-stage memory hierarchy inspired by human memory: rapid filtering (sensory), structured consolidation (short-term), and offline processing (long-term).

The system filters irrelevant information early, organizes remaining context by topic, and separates online inference from offline consolidation. This achieves 38× token reduction and 30× fewer API calls while improving QA accuracy by up to 7.7%.

Core Concept

LightMem operates on three principles:

• Sensory Memory: Lightweight compression that rapidly filters irrelevant information and groups by topic
• Short-term Memory: Organized consolidation of topic groups into structured summaries
• Long-term Memory: Offline procedures that separate consolidation from real-time inference
• Lazy Retrieval: Only fetch long-term memories when needed, don't load everything

The three-stage approach prevents context explosion while maintaining access to historical information when necessary.

Architecture Overview

• Input Processor: Compress incoming text to key tokens/entities
• Topic Detector: Classify information into semantic topics
• Sensory Buffer: Lightweight sliding window that groups information
• Short-term Consolidator: Summarize topic groups on-demand
• Long-term Storage: Offline database of compressed memories
• Retriever: Fetch relevant long-term memories based on current query
• Context Manager: Assemble final context from sensory + retrieved long-term

Implementation Steps

The key is implementing efficient filtering at each stage. This example shows the three-stage pipeline.

import torch
from collections import defaultdict
from typing import List, Dict

class SensoryMemory:
    """
    Stage 1: Lightweight rapid filtering and topic grouping.
    """

    def __init__(self, max_tokens: int = 500, max_topics: int = 10):
        self.max_tokens = max_tokens
        self.max_topics = max_topics
        self.buffer = []
        self.topic_groups = defaultdict(list)

    def compress_and_group(self, text: str, topic_classifier) -> None:
        """
        Rapidly filter and group incoming information by topic.
        """
        # Extract key entities/tokens (fast, heuristic-based)
        key_tokens = extract_key_tokens(text, max_tokens=50)

        # Classify into topic
        topic = topic_classifier.predict(text)

        # Store in sensory buffer
        self.buffer.append({
            "text": text,
            "tokens": key_tokens,
            "topic": topic,
            "timestamp": len(self.buffer)
        })

        # Group by topic
        self.topic_groups[topic].append(len(self.buffer) - )

        
        total_tokens = ((item[])  item  .buffer)
         total_tokens > .max_tokens  .buffer:
            old_item = .buffer.pop()
            total_tokens -= (old_item[])


 :
    

     ():
        .summarizer = summarizer_model
        .topic_summaries = {}

     () -> :
        
          topic_texts:
             

        
        combined = .join(topic_texts)

        
        summary_prompt = 
        summary = .summarizer.generate(summary_prompt, max_tokens=)
        .topic_summaries[topic] = summary
         summary

     () -> :
        
        context_parts = []
         topic  active_topics:
             topic  .topic_summaries:
                context_parts.append()
         .join(context_parts)


 :
    

     ():
        .embedder = embedding_model
        .db_path = db_path
        .memory_vectors = []
        .memory_texts = []

     () -> :
        
        
        embedding = .embedder.embed(consolidated_text)

        
        .memory_vectors.append(embedding)
        .memory_texts.append(consolidated_text)

     () -> []:
        
        
        query_embedding = .embedder.embed(query)

        
        similarities = torch.nn.functional.cosine_similarity(
            torch.tensor(query_embedding).unsqueeze(),
            torch.tensor(.memory_vectors)
        )

        
        top_indices = torch.topk(similarities, k=(top_k, (.memory_texts)))[]
        relevant = [.memory_texts[i]  i  top_indices.tolist()]

         relevant


 :
    

     ():
        .sensory = SensoryMemory()
        .short_term = ShortTermMemory(summarizer)
        .long_term = LongTermMemory(embedder)
        .topic_classifier = topic_classifier

     () -> :
        
        
        .sensory.compress_and_group(text, .topic_classifier)

        
         (.sensory.buffer) > :
            
             topic, indices  .sensory.topic_groups.items():
                topic_texts = [.sensory.buffer[i][]  i  indices]
                .short_term.consolidate_topic_group(topic, topic_texts)

            
             topic, summary  .short_term.topic_summaries.items():
                 (summary) > :  
                    .long_term.add_to_long_term(summary)

     () -> :
        
        
        sensory_context = .join(
            item[][:]  item  .sensory.buffer[-:]  
        )

        
        active_topics = (.sensory.topic_groups.keys())[:]
        short_term_context = .short_term.get_short_term_context(active_topics)

        
        long_term_context = .join(
            .long_term.retrieve_relevant_memories(query, top_k=)
        )

        
        full_context = 
         full_context


 ():
    
     user_input  user_inputs:
        
        memory_manager.process_new_information(user_input)

        
        context = memory_manager.generate_context_for_query(user_input)

        
        response = model.generate(context, max_tokens=)
        ()

The three-stage design is crucial: Stage 1 filters fast (heuristics), Stage 2 organizes on-demand (summaries only when needed), Stage 3 stores and retrieves offline (no latency hit during inference).

Practical Guidance

Scenario	Token Reduction	API Call Reduction	Accuracy Change
Multi-turn QA	20-30×	20-30×	+5-8%
Document RAG	10-15×	10-15×	+3-5%
Online learning	15-25×	15-25×	+2-4%

When to Use:

• Long-running conversations or sessions
• Token costs matter (pay-per-token APIs)
• Need to maintain historical context without explosion
• Multi-document or multi-topic interactions

When NOT to Use:

• Single-turn or short conversations
• Real-time applications where consolidation latency is problematic
• Tasks where full context is always needed
• Budget not constrained (simpler full-context approaches work)

Common Pitfalls:

• Sensory buffer too aggressive in filtering → loses important details
• Short-term summaries too compressed → lose nuance needed for reasoning
• Long-term retrieval misses relevant context → validate retriever quality
• Not separating online/offline → adds latency to inference
• Topic classifier too granular → fragments related information

Reference

LightMem: Lightweight and Efficient Memory-Augmented Generation