memory-agent-evaluation

name	memory-agent-evaluation
title	Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.05257
keywords	["Memory Evaluation","LLM Agents","Multi-Turn Interactions","Knowledge Retrieval","Test-Time Learning"]
description	Evaluate and improve memory capabilities in LLM agents across four competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Identifies critical gaps in how agents store, update, and revise information.

MemoryAgentBench: Comprehensive Memory Evaluation for LLM Agents

Current LLM agents excel at reasoning and planning but fail at memory—the ability to compress, store, update, and selectively revise information from multi-turn interactions. While long-context models can process large inputs, they lack the incremental memory mechanisms that real agents need. MemoryAgentBench addresses this gap by introducing a unified evaluation framework that assesses four core memory competencies agents must master.

Memory is fundamentally different from simply having access to long context windows. True memory involves selectively compressing information, updating beliefs when contradicted, and retrieving relevant details on demand—mimicking how human agents learn and adapt over time.

Core Concept

The framework evaluates memory through four distinct competencies that agents encounter in real deployments:

Accurate Retrieval (AR): Can the agent extract specific information snippets when asked? This tests whether information is properly stored and indexed.

Test-Time Learning (TTL): Can the agent acquire new skills or knowledge during deployment without retraining? This simulates learning from user demonstrations mid-deployment.

Long-Range Understanding (LRU): Can the agent integrate information across extremely long contexts (100k+ tokens) to answer questions requiring cross-document synthesis?

Selective Forgetting (SF): Can the agent revise or remove outdated information when presented with contradictions? This is critical for maintaining consistency as the world state changes.

Architecture Overview

• Multi-turn dialogue simulation: Textual information presented incrementally across many turns, mimicking real agent interactions
• Four competency benchmarks: Separate datasets for each memory capability with increasing difficulty levels
• Three agent architectures: Long-context models, RAG-based systems, and agentic memory agents with external modules
• Quantitative assessment: Brier scores and task-completion metrics to identify which agents handle which competencies
• Multi-hop reasoning requirements: Selective forgetting tests require reasoning across multiple conversational turns

Implementation

The evaluation framework constructs scenarios where agents must demonstrate each competency. Here's how to set up a basic memory evaluation:

# Load a multi-turn scenario for accurate retrieval testing
scenario = {
    "turns": [
        {: , : },
        {: , : },
        {: , : }
    ],
    : ,
    : 
}


contradiction_scenario = {
    : [
        {: , : },
        {: , : },
        {: , : }
    ],
    : ,
    : 
}