{"category":"AI & Agents","tags":["memory","algorithms","vector-search","knowledge-graphs","HNSW","episodic-memory","agents","data-structures"],"provenance":{"kind":"first-party","owners":["port-daddy"]},"pairs-with":[{"skill":"always-on-agent-architecture","reason":"Always-on agents require persistent memory across sessions"},{"skill":"always-on-agent-inputs","reason":"Input streams feed into memory; memory informs input processing"},{"skill":"agent-interchange-formats","reason":"Memory retrieval results must be serialized for agent consumption"},{"skill":"agentic-infrastructure-2026","reason":"Once the memory algorithm is chosen here, that skill picks the vendor/framework to run it on"}],"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
You are an expert in the data structures and algorithms that give AI agents memory. You understand the full stack from vector search to knowledge graphs to temporal retrieval. You can design memory systems that balance recall accuracy, latency, storage cost, and token budget.
Decision Points
1. Memory Architecture Selection
What type of facts does your agent need to remember?
│
├─ User identity + preferences ONLY
│ └─ Use: Core memory (always in context) + simple vector store
│ Algorithm: Flat search or small HNSW (<10K entries)
│ Reasoning: Static facts don't need complex temporal tracking
│
├─ Past conversations + events
│ ├─ Need fact change tracking over time?
│ │ ├─ YES → Bi-temporal knowledge graph (Zep/Graphiti approach)
│ │ │ Algorithm: Triple store with (validFrom, ingestedAt) timestamps
│ │ │ Reasoning: Legal/medical/preference changes require audit trail
│ │ └─ NO → Vector store + simple timestamps
│ │ Algorithm: HNSW + exponential decay scoring
│ │ Reasoning: Most conversational agents don't need change history
│ │
│ └─ Need multi-hop reasoning ("who did X introduce me to")?
│ ├─ YES → Knowledge graph + vector store hybrid
│ │ Algorithm: Triple store + HNSW with graph traversal
│ │ Reasoning: Relational queries require structured relationships
│ └─ NO → Vector store with entity extraction
│ Algorithm: HNSW with named entity metadata
│ Reasoning: Simpler, faster, good enough for most cases
│
└─ Learned skills + behaviors
└─ Use: Procedural memory (prompt templates, tool definitions)
Algorithm: Static prompt injection + skill registry
Reasoning: Skills change infrequently, need immediate access
2. Vector Search Algorithm Selection
How many memories will you store?
│
├─ <100K entries
│ ├─ Need real-time updates? → HNSW (efConstruction=200, M=32)
│ └─ Batch updates only? → Flat search (brute force cosine similarity)
│
├─ 100K - 10M entries
│ ├─ Memory constrained? → IVF-PQ (nlist=4096, m=64, nbits=8)
│ └─ Memory abundant? → HNSW (efConstruction=400, M=64)
│
└─ >10M entries
└─ Use: DiskANN or distributed IVF-PQ
Reasoning: Single-machine HNSW becomes prohibitively expensive
3. Retrieval Scoring Strategy
What's the query context?
│
├─ User asked specific question about past
│ └─ Prioritize: relevance=0.6, recency=0.2, importance=0.2
│ Reasoning: Accuracy matters more than freshness for specific recall
│
├─ Agent deciding what context to include
│ └─ Prioritize: importance=0.5, relevance=0.3, recency=0.2
│ Reasoning: Include high-value facts even if not perfectly relevant
│
├─ Real-time conversation flow
│ └─ Prioritize: recency=0.5, relevance=0.3, importance=0.2
│ Reasoning: Recent context usually most relevant to ongoing discussion
│
└─ Cold start (new session beginning)
└─ Prioritize: importance=0.7, relevance=0.3, recency=0.0
Reasoning: Load core identity facts regardless of when they happened
4. Forgetting Policy Assignment
What's the importance score of this memory? (1-10 scale)
│
├─ 9-10 (Core identity, critical preferences)
│ └─ Policy: never_forget
│ Reasoning: Agent loses coherent identity without these facts
│
├─ 7-8 (Important decisions, key relationships)
│ └─ Policy: decay with halfLife=720 hours (30 days)
│ Reasoning: Important enough to remember long-term but not forever
│
├─ 4-6 (Interesting conversations, moderate preferences)
│ └─ Policy: decay with halfLife=168 hours (7 days)
│ Reasoning: Useful for context but not critical to preserve
│
├─ 1-3 (Small talk, transient mentions)
│ └─ Policy: decay with halfLife=24 hours (1 day)
│ Reasoning: Clutter that wastes token budget if retained
│
└─ Temporary working notes
└─ Policy: session_scoped (delete on session end)
Reasoning: Only useful within current conversation context
Failure Modes
1. Memory Retrieval Hallucination
Symptoms: Agent confidently states facts that aren't in retrieved memories, or misattributes information.
Detection Rule: If cosine similarity < 0.7 but agent states specific facts, you've hit retrieval hallucination.
Diagnosis: The retrieval scored poorly but LLM filled gaps with plausible-sounding information.
Fix:
Set hard similarity threshold (0.75+) for factual claims
Include confidence scores in retrieved memory context
Add "I don't have specific information about..." fallback responses
Log retrieval scores for manual audit of false positives
2. Token Budget Overflow
Symptoms: Context window consistently full, important instructions truncated, degraded performance.
Detection Rule: If retrieved memories + recent messages > 80% of context window, you've hit token overflow.
Diagnosis: Too many low-quality memories retrieved, no prioritization by token cost vs. value.
Deduplicate similar triples: merge (user, likes, coffee) and (user, enjoys, coffee)
Limit extraction to high-importance messages (importance >= 5)
Worked Examples
Example 1: Personal Assistant Memory Design (1M memories, <200ms retrieval)
Scenario: Building memory for a personal assistant agent that helps with scheduling, email, and personal tasks. Needs to remember user preferences, past decisions, and ongoing projects across 2-year timespan.
Step 1: Architecture Decision
Query: "What type of facts?" → User preferences + past conversations + learned behaviors
Decision: Hybrid system (core memory + vector store + knowledge graph)
Reasoning: Need identity facts always available, episodic search, and relationship tracking
Step 2: Scale Planning
Expected growth: ~1,500 memories/day = 1M memories over 2 years
Latency requirement: <200ms P95 (user-facing)
Decision: HNSW with efConstruction=400, M=64, efSearch=100
Step 3: Forgetting Strategy
Core identity (importance 9-10): never_forget
"User is vegetarian, lives in SF, works at Google"
Work preferences (importance 7-8): 30-day half-life
"Prefers morning meetings, dislikes back-to-back calls"
Project context (importance 5-6): 7-day half-life
"Working on Q3 planning, deadline is July 15"
Small talk (importance 1-3): 1-day half-life
"Mentioned liking the weather yesterday"
Step 4: Retrieval Tuning
# Morning briefing (load important context)
recall(query="daily tasks", weights={
importance: 0.6, # Prioritize high-value facts
relevance: 0.3,
recency: 0.1# Old todos still matter
})
# Answering specific question
recall(query="when is my dentist appointment", weights={
relevance: 0.7, # Exact match most important
recency: 0.2, # Recent scheduling changes
importance: 0.1
})
What a novice would miss: Setting uniform weights (0.33 each) regardless of query type, no importance scoring, storing entire emails instead of extracted facts.
What an expert catches: Query-adaptive scoring, importance assessment at write time, chunking long content before embedding, token budget management.
Example 2: Customer Service Agent (Enterprise Scale)
Scenario: Support agent for SaaS product. Needs to remember customer issues, feature requests, and resolutions. 10M+ customer interactions, need bi-temporal tracking for compliance.
Step 1: Compliance Requirements
Must track "when did issue occur" vs "when did we log it"
# Customer context loadingdefload_customer_context(customer_id: str, as_of: str = None):
# Get all facts valid at time as_of (or now)
facts = knowledge_graph.query(
subject=customer_id,
valid_at=as_of or datetime.now()
)
# Semantic search for related issues
similar_cases = vector_store.search(
embedding=embed(f"customer {customer_id} context"),
filter={"customer_tier": facts.get("subscription_tier")}
)
return merge_and_rank(facts, similar_cases)
What a novice would miss: No compliance tracking, single timestamp per fact, no customer segmentation in retrieval.
What an expert catches: Bi-temporal model, fact superseding logic, customer-tier-aware retrieval, audit trail preservation.
Quality Gates
Memory system is production-ready when:
Retrieval latency P95 < 300ms for target memory count
Recall@10 > 0.85 on evaluation dataset of real queries
Core memory facts (importance >= 9) never get forgotten
Token budget for retrieved memories < 30% of context window
Importance scoring is consistent (same content gets same score ±1)
Forgetting policy removes >90% of low-importance memories within TTL
Knowledge graph has <5% invalid triples (broken references, low confidence)
Memory consolidation reduces storage by >50% without losing high-importance facts
Cold start retrieval loads user identity within 3 queries
Bi-temporal queries return correct facts for "what did we know at time T"
Contradiction detection flags conflicting facts with >0.8 confidence
System degrades gracefully when vector index is empty (fallback to keyword search)
NOT-FOR Boundaries
Do NOT use this skill for:
RAG document search: For retrieving chunks from static documents, use ai-engineer instead. Agent memory is for dynamic, temporal facts about ongoing relationships.
Infrastructure selection: For choosing between Pinecone vs Weaviate vs Qdrant, use agentic-infrastructure-2026 instead. This skill covers algorithms, not vendor comparison.
Conversation protocol design: For structuring agent-to-agent communication, use agent-conversation-protocols instead. Memory retrieval format ≠ interchange format.
Prompt engineering: For optimizing how memories are injected into context, use prompt-engineer instead. This skill covers storage/retrieval, not context formatting.
General vector database usage: For embedding search in recommendation systems or semantic search over documents, use ai-engineer instead. Agent memory has unique temporal and importance requirements.
When to delegate:
"How should I structure memory in prompts?" → prompt-engineer
"Which vector database should I deploy?" → agentic-infrastructure-2026
"How do agents exchange memory between each other?" → agent-conversation-protocols
"How do I build a RAG system?" → ai-engineer
Skill Bundle Index
Every file in this skill, and when to open it. Auto-generated; run scripts/index_references.py --fix.
