// Strategies for managing LLM context windows effectively in AI agents. Use when building agents that handle long conversations, multi-step tasks, tool orchestration, or need to maintain coherence across extended interactions.
| name | context-engineering |
| description | Strategies for managing LLM context windows effectively in AI agents. Use when building agents that handle long conversations, multi-step tasks, tool orchestration, or need to maintain coherence across extended interactions. |
Context engineering is the discipline of curating and maintaining the optimal set of tokens during LLM inference. Unlike prompt engineering (crafting individual prompts), context engineering focuses on what information enters the context window and when.
LLMs have limited "attention budgets." As context length increases, models experience context rot—decreased ability to accurately recall information. The goal is finding the smallest possible set of high-signal tokens that maximize desired outcomes.
Effective Context = Relevant Information / Total Tokens
Key insight: More context isn't better. The right context is better.
Every token added to context has costs:
Drop older conversation turns, keeping only the last N turns.
| Aspect | Details |
|---|---|
| Mechanism | Sliding window over conversation history |
| Pros | Deterministic, zero latency, preserves recent context verbatim |
| Cons | Abrupt loss of long-range context, "amnesia" effect |
| Best for | Independent tasks, short interactions, predictable workflows |
def trim_context(messages: list, keep_last_n: int = 10) -> list:
"""Keep system message + last N turns."""
system_msgs = [m for m in messages if m["role"] == "system"]
other_msgs = [m for m in messages if m["role"] != "system"]
return system_msgs + other_msgs[-keep_last_n:]
Compress prior messages into structured summaries.
| Aspect | Details |
|---|---|
| Mechanism | LLM generates summary of older context |
| Pros | Retains long-range memory, smoother UX, scalable |
| Cons | Summarization bias risk, added latency, potential compounding errors |
| Best for | Complex multi-step tasks, long-horizon interactions |
SUMMARIZATION_PROMPT = """Summarize the conversation so far, preserving:
1. Key decisions made
2. Important context established
3. Current task state and goals
4. Any constraints or preferences expressed
Be concise but complete. Output as structured markdown."""
async def summarize_context(messages: list, model) -> str:
"""Generate a summary of conversation history."""
conversation_text = format_messages_for_summary(messages)
response = await model.generate(
system=SUMMARIZATION_PROMPT,
user=conversation_text
)
return response.content
Combine trimming and summarization for optimal balance.
class HybridContextManager:
def __init__(
self,
keep_recent: int = 5, # Recent turns to keep verbatim
summary_threshold: int = 20, # When to trigger summarization
):
self.keep_recent = keep_recent
self.summary_threshold = summary_threshold
self.running_summary = ""
def process(self, messages: list) -> list:
if len(messages) < self.summary_threshold:
return messages
# Summarize older messages
old_messages = messages[:-self.keep_recent]
self.running_summary = summarize(old_messages, self.running_summary)
# Return summary + recent messages
return [
{"role": "system", "content": f"Previous context:\n{self.running_summary}"},
*messages[-self.keep_recent:]
]
Persist reusable facts, preferences, and task state outside the context window. Load only the relevant slice for the current turn.
| Aspect | Details |
|---|---|
| Mechanism | External store keyed by user, session, task, or resource |
| Pros | Recovers long-range context without carrying all history |
| Cons | Requires retrieval, freshness, and deletion policies |
| Best for | Agents, project work, personalization, long-running workflows |
Separate durable memory from ephemeral scratchpads. Durable memory should contain stable facts and explicit decisions, not every intermediate thought.
# Role
You are [specific role] that [primary function].
# Capabilities
- [Capability 1 with scope]
- [Capability 2 with scope]
# Constraints
- [Hard constraint]
- [Preference]
# Output Format
[Specific format requirements]
Instead of front-loading all possible context, load information dynamically as needed.
# Anti-pattern: Loading everything upfront
context = load_all_user_data() # Large, mostly unused
context += load_all_documents() # Even larger
# Better: Just-in-time retrieval
tools = [
Tool(
name="get_user_preference",
description="Get specific user preference by key",
# Only fetches what's needed when asked
),
Tool(
name="search_documents",
description="Search documents by query",
# Returns relevant subset
),
]
# Well-designed tool
def search_codebase(query: str, max_results: int = 5) -> str:
"""Search codebase for relevant code snippets.
Args:
query: Natural language description of what to find
max_results: Maximum snippets to return (default 5)
Returns:
Formatted code snippets with file paths and line numbers,
or 'No results found' if nothing matches.
"""
results = perform_search(query, limit=max_results)
if not results:
return "No results found for query."
return format_results(results) # Concise, structured output
Periodically compress conversation history to reclaim context space.
async def compaction_loop(agent, messages, task):
while not task.complete:
# Process next step
response = await agent.run(messages)
messages.append(response)
# Compact when approaching limit
if estimate_tokens(messages) > TOKEN_LIMIT * 0.8:
summary = await summarize_context(messages[:-3])
messages = [
{"role": "system", "content": agent.system_prompt},
{"role": "assistant", "content": f"Summary of progress:\n{summary}"},
*messages[-3:] # Keep recent context
]
return messages
Agent maintains external notes, retrieving as needed.
class NoteTakingAgent:
def __init__(self):
self.notes = {} # Key-value store outside context
async def run(self, messages):
tools = [
Tool("save_note", self.save_note, "Save information for later"),
Tool("get_note", self.get_note, "Retrieve saved information"),
Tool("list_notes", self.list_notes, "List all saved note keys"),
]
return await self.agent.run(messages, tools=tools)
def save_note(self, key: str, content: str) -> str:
self.notes[key] = content
return f"Saved note: {key}"
def get_note(self, key: str) -> str:
return self.notes.get(key, f"No note found for key: {key}")
Delegate focused tasks to specialized agents with clean context.
class OrchestratorAgent:
def __init__(self):
self.sub_agents = {
"researcher": ResearchAgent(),
"coder": CodingAgent(),
"reviewer": ReviewAgent(),
}
async def delegate(self, task: str, agent_type: str) -> str:
"""Delegate to sub-agent, receive condensed summary."""
agent = self.sub_agents[agent_type]
# Sub-agent works with fresh context
result = await agent.run(task)
# Return only essential findings to main context
return result.summary # Not the full conversation
Benefits:
class SessionMemory:
def __init__(
self,
keep_last_n_turns: int = 5,
context_limit: int = 100_000, # tokens
summarizer = None,
):
self.keep_last_n_turns = keep_last_n_turns
self.context_limit = context_limit
self.summarizer = summarizer
self.messages = []
self.summary = ""
async def add_message(self, message: dict):
self.messages.append(message)
await self._maybe_compact()
async def _maybe_compact(self):
current_tokens = estimate_tokens(self.messages)
if current_tokens > self.context_limit * 0.8:
# Summarize all but recent messages
old_messages = self.messages[:-self.keep_last_n_turns]
new_summary = await self.summarizer.summarize(
old_messages,
previous_summary=self.summary
)
self.summary = new_summary
self.messages = self.messages[-self.keep_last_n_turns:]
def get_context(self) -> list:
context = []
if self.summary:
context.append({
"role": "system",
"content": f"Conversation summary:\n{self.summary}"
})
context.extend(self.messages)
return context
def estimate_tokens(messages: list) -> int:
"""Rough token estimation (4 chars ≈ 1 token for English)."""
total_chars = sum(
len(m.get("content", ""))
for m in messages
)
return total_chars // 4
def estimate_tokens_accurate(messages: list, model: str) -> int:
"""Accurate token count using tiktoken."""
import tiktoken
encoding = tiktoken.encoding_for_model(model)
return sum(
len(encoding.encode(m.get("content", "")))
for m in messages
)
Treat context as precious: Every token has a cost. Include only information that improves task performance.
Use progressive disclosure: Start minimal, expand context only when needed via tools.
Design for recoverability: Agents should be able to reconstruct critical context from external sources.
Monitor context health: Track token usage, retrieval accuracy, and task completion rates.
Prefer structured over raw data: JSON, markdown tables, and clear formatting improve information density.
Implement graceful degradation: When context limits approach, prioritize recent and high-signal information.
Test with long conversations: Validate agent behavior after many turns, not just initial interactions.
Separate concerns: Use different context regions for system instructions, user history, and tool outputs.
Version your summaries: When compacting, maintain enough structure to debug summarization issues.
Measure and iterate: Context engineering is empirical—test what information actually improves outcomes.
Separate memory classes: Keep instructions, recent messages, durable memory, retrieved documents, and tool results in distinct sections so conflicts are easier to detect.
Patterns and architectures for building AI agents and workflows with LLMs. Use when designing systems that involve tool use, multi-step reasoning, autonomous decision-making, or orchestration of LLM-driven tasks.
Parameter-efficient fine-tuning with Low-Rank Adaptation (LoRA). Use when fine-tuning large language models with limited GPU memory, creating task-specific adapters, or when you need to train multiple specialized models from a single base.
Running and fine-tuning LLMs on Apple Silicon with MLX. Use when working with models locally on Mac, converting Hugging Face models to MLX format, fine-tuning with LoRA/QLoRA on Apple Silicon, or serving models via HTTP API.
Crafting effective prompts for LLMs. Use when designing prompts, improving output quality, structuring complex instructions, or debugging poor model responses.
Building and training neural networks with PyTorch. Use when implementing deep learning models, training loops, data pipelines, model optimization with torch.compile, distributed training, or deploying PyTorch models.
Memory-efficient fine-tuning with 4-bit quantization and LoRA adapters. Use when fine-tuning large models (7B+) on consumer GPUs, when VRAM is limited, or when standard LoRA still exceeds memory. Builds on the lora skill.