mmpo-metacognitive-memory-policy

name	mmpo-metacognitive-memory-policy
description	Meta-Cognitive Memory Policy Optimization (MMPO) for long-horizon LLM agents using Belief Entropy as self-supervised proxy.
arxiv_id	2605.30159
authors	Ziyan Liu, Zhezheng Hao, Yeqiu Chen, Hong Wang
published	2026-05-29T00:00:00.000Z
categories	["Artificial Intelligence","LLM Agents","Memory Optimization","Metacognitive"]
tags	["LLM","agents","memory","reinforcement-learning","belief-entropy","long-horizon","metacognitive"]
activation_keywords	["memory policy optimization","belief entropy","long-horizon LLM agent","metacognitive","MMPO","memory-augmented agent"]
related_skills	["agent-memory-framework","agent-memory-management","indexed-memory"]

Meta-Cognitive Memory Policy Optimization (MMPO)

Overview

Meta-Cognitive Memory Policy Optimization (MMPO) is a novel approach for training memory-augmented LLM agents on long-horizon tasks. It addresses the critical problem of belief deviation in recursive memory summarization by introducing Belief Entropy — a self-supervised proxy that measures epistemic uncertainty about the latent task state.

Key Innovation: Instead of relying solely on sparse outcome-based reinforcement learning signals, MMPO provides fine-grained, memory-specific supervision by explicitly penalizing summaries that induce high epistemic uncertainty.

arXiv: 2605.30159

Problem Statement

Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However:

1. Outcome-based RL fails to localize intermediate memory quality degradation
2. Ambiguous recursive summaries progressively discard task-relevant information
3. Semantic noise accumulates, exacerbating belief deviation
4. Agents lose accurate estimates of latent task states
5. Long-horizon reasoning derails due to accumulated uncertainty

Core Concepts

Belief Entropy

Belief Entropy is a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory:

• Definition: Measures epistemic uncertainty in belief state induced by memory
• Purpose: Identify memory summaries that obscure task understanding
• Mechanism: Self-supervised — no external labels required
• Application: Fine-grained supervision signal for memory policy optimization

MMPO Framework

Metacognitive Memory Policy Optimization consists of:

1. Memory Policy: Agent's strategy for summarizing interaction trajectories
2. Belief Entropy Estimation: Quantify uncertainty about latent task state
3. Penalty Mechanism: Penalize high-uncertainty memory summaries
4. Combined Optimization: Outcome-based RL + Belief Entropy supervision

Implementation Methodology

Step-by-Step Process