defermem-evidence-distillation

# DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA **Paper:** arXiv:2605.22411 (May 2026) ## Overview DeferMem is a long-term memory framework for LLM-based agents that addresses a fundamental tension in memory-augmented QA: retrieving too much context drowns the model in noise and token budgets, while retrieving too little misses critical evidence. DeferMem decouples memory into two distinct phases: 1. **High-recall candidate retrieval** — cast a wide net across raw history using a lightweight structural index 2. **Query-conditioned evidence distillation** — an RL-trained *memory distiller* selects and rewrites the noisy candidates into concise, faithful, self-contained evidence that directly answers the query The key insight is that *raw retrieval can be coarse and high-recall* (cheap, broad) while the burden of precision is shifted to a *trainable distillation step* that produces clean evidence for the downstream LLM. This mirrors "deferred" computation — hence the name DeferMem. ## Segment-Link Structure The raw conversation history is organized into a lightweight **segment-link** structure: - The conversation is divided into **segments** — contiguous blocks of dialogue turns (typically 2–8 turns each). - **Links** are bidirectional edges between segments, weighted by semantic or temporal adjacency (e.g., discourse continuity, topic shift detection, timestamp proximity). - The segment-link graph is built **offline** during conversation ingestion and updated incrementally as new history arrives. - At query time, **spreading activation** starting from the last few segments traverses the link graph to retrieve a broad candidate set (typically 50–200 segments) — intentionally sacrificing precision for high recall. - This structure is lightweight compared to full vector-database indexes: segments are plain text with metadata, links are sparse edges, and no dense embeddings are stored persistently. ## Dual-Phase Workflow ### Phase 1: Candidate Retrieval (High-Recall, Zero-Learning) 1. Identify the most recent N segments as seed nodes in the segment-link graph. 2. Run spreading activation with decaying relevance scores across link hops (1–3 hops, configurable). 3. Collect all activated segments as the raw candidate pool (targeting recall >95% on held-out queries). 4. No learned ranking or dense retrieval — just structural graph traversal. ### Phase 2: Evidence Distillation (RL-Trained, Query-Conditioned) 1. The raw candidates + the user query are fed into the **memory distiller** (a lightweight transformer, e.g., 7B–8B parameters). 2. The distiller produces structured output: a selected subset of messages + optionally rewritten evidence text. 3. The evidence is appended to the context window of the downstream LLM for final answer generation. 4. The distillation is done at **query time** — the distiller runs once per query, not during ingestion. ## DistillPO Algorithm DistillPO (Distillation Policy Optimization) is the RL algorithm used to train the memory distiller. It formulates post-retrieval evidence distillation as a structured RL problem. ### Structured Action Space Each action consists of two components: - **Message Selection (`a_sel`):** A binary mask over retrieved candidates, choosing which raw messages to keep (filtering out noise, redundancy, and irrelevant context). - **Evidence Rewriting (`a_rw`):** An optional rewrite step that condenses, paraphrases, or reformulates the selected messages into self-contained evidence that directly answers the query. The distiller can choose to pass through verbatim (no rewrite) when the selected messages are already suitable. The combined action is `a = (a_sel, a_rw)`, allowing the distiller to both *filter* and *reshape* retrieved content. ### Decomposed-and-Gated Reward Pipeline The reward signal is decomposed into three components, each assessing a distinct quality dimension: | Reward Component | Symbol | Assesses | |---|---|---| | **Faithfulness** | `R_faith` | Whether the distilled evidence is factually consistent with the source segments (using an NLI-based verifier) | | **Answerability** | `R_ans` | Whether the distilled evidence, when given to the downstream LLM (frozen decoder), enables a correct answer to the query | | **Conciseness** | `R_conc` | Token efficiency — penalizes unnecessarily long evidence relative to a budget | The gated combination is: ``` R = g_faith * R_faith + g_ans * R_ans + g_conc * R_conc ``` Where `g_* ∈ {0, 1}` are gating signals. The gates are computed heuristically: `g_faith = 1` always; `g_ans = 1` if `R_faith` exceeds a threshold (no point rewarding answerability if evidence is unfaithful); `g_conc = 1` only when both previous gates are active and the evidence length exceeds the budget. This **deferred gating** ensures that reward components only activate when prerequisite quality dimensions are satisfied, preventing the distiller from gaming the system (e.g., producing concise-but-wrong evidence). ### Advantage Assignment & Training - Trajectories are collected by running the distiller on training queries paired with ground-truth answers. - The reward model (frozen downstream LLM + NLI verifier) scores each distillation output. - **Advantage** is computed per action step using a learned value baseline (a small MLP over distiller hidden states), following a REINFORCE-with-baseline setup. - The policy gradient objective maximizes the expected advantage: ``` ∇J(θ) = E[ A(s, a) · ∇θ log π_θ(a|s) ] ``` - Ties are broken behaviorally: when no candidate is clearly superior, the distiller learns to prefer conciseness via the gated reward structure. ## Key Results Reported on long-context QA benchmarks (e.g., LongBench, NarrativeQA, and an internal multi-session conversation dataset): | Metric | DeferMem | Best Baseline | Δ | |---|---|---|---| | **QA Accuracy (F1)** | **73.4%** | 68.1% (Full-Context) | +5.3 pp | | **QA Accuracy (EM)** | **61.2%** | 55.8% (RAG-50) | +5.4 pp | | **Evidence Token Budget** | **~1,200 tokens** | ~8,000 tokens (Full-Context) | **6.7× reduction** | | **End-to-End Latency** | **1.2×** | 1.0× (no memory) | only 20% overhead | | **Distillation False Positives** | **↓ 62%** | — | vs. Frozen RAG | Key findings: - DeferMem matches or exceeds the accuracy of *full-context* approaches while using ~15% of the tokens. - The segment-link retrieval achieves 96% recall on evidence-containing segments across held-out queries. - DistillPO-trained distiller significantly outperforms supervised-finetuning (SFT) baselines for evidence distillation, especially on the faithfulness and conciseness axes. - The decomposed reward structure is critical: ablating any single component degrades accuracy by 2–6 pp. - The gating mechanism prevents reward hacking: without gating, the distiller learns to produce miniaturized-but-wrong evidence that exploits conciseness rewards. ## When to Use DeferMem / DistillPO - **Long-term memory QA:** Systems where an agent accumulates conversation history over extended sessions (days/weeks) and needs to answer queries grounded in that history. - **RAG with long conversations:** Standard RAG breaks down with hundreds of conversation turns due to embedding drift and token limits; DeferMem's segment-link structure handles it more naturally. - **Evidence distillation as a learned skill:** When you want to train a model to condense retrieved context specifically for downstream task performance, rather than using generic summarization. - **RL for retrieval/reranking pipelines:** DistillPO's decomposed reward and gating approach is transferable to other RL-for-retrieval settings (multi-hop QA, tool-use memory, etc.). ## Activation Keywords DeferMem, DistillPO, evidence distillation, query-time distillation, long-term memory QA, memory system, segment-link graph, spreading activation, reinforcement learning for retrieval, decomposed reward, gated reward, post-retrieval distillation, RL for RAG, evidence selection, evidence rewriting, conversational memory, multi-session QA, high-recall retrieval, query-conditioned distillation, REINFORCE with baseline, faithful evidence generation, memory distiller, structured action space, NLI-based faithfulness verification, deferred computation, learning to distill retrieval.