rlm-processing

name

rlm-processing

description

Process arbitrarily large text files that exceed context window limits using the Recursive Language Model (RLM) paradigm. Use when the user asks to analyze, summarize, search, or extract information from large files (logs, codebases, legal documents, datasets, books) that are too big to fit in context. Triggers include requests like "analyze this large file", "process all my logs", "summarize this book", "find patterns across this codebase", "extract all X from this document", or any task involving files larger than ~100KB. Requires the rlm_mcp MCP server to be installed.

RLM Processing

Process files of unlimited size by treating them as external environment variables that Claude iteratively explores.

Core Concept

Instead of trying to fit large files into context, RLM:

1. Loads the file into an external "environment" via MCP tools
2. Claude examines the file piece by piece through tool calls
3. Claude aggregates findings across all pieces
4. Claude itself acts as both root and sub-LM - no external API needed

Available MCP Tools

Tool	Purpose
rlm_load_context	Load a file into RLM environment
rlm_get_chunk	Get chunk N for processing (iterate 0, 1, 2...)
rlm_search_context	Find patterns/text with regex or literal search
rlm_slice_context	Extract specific line or character ranges
rlm_execute_code	Run Python code (regex, counting) on context
rlm_context_info	Check loaded contexts and plan chunking
rlm_clear_context	Free memory when done

Critical: Reasoning Between Tool Calls

You (Claude) ARE the sub-LM. After each tool call, you must:

1. Analyze the returned content for the user's task
2. Extract relevant information and hold it in your reasoning
3. Decide what to do next (get next chunk, search, or aggregate)
4. Accumulate findings across all chunks before responding

Do NOT just call tools mechanically. Each chunk requires active analysis.

Choosing the Right Pattern

Task Type	Use Pattern	Why
Summarize entire document	Full Iteration	Must see everything
Extract ALL instances of X	Full Iteration	Can't miss any
Find specific fact/answer	Search First	Targeted lookup
Count occurrences	Execute Code	Regex is faster
Unknown file structure	Probe First	Explore before deciding

Default to Full Iteration when comprehensive coverage matters.

Workflow Patterns

Pattern 1: Full Document Processing (DEFAULT)

Use when: summarization, comprehensive extraction, analysis requiring full coverage.

1. Load file with rlm_load_context
2. Note total chunks from response  
3. Loop through ALL chunks:
   a. Call rlm_get_chunk(index=N)
   b. STOP and ANALYZE the content - what's relevant to the task?
   c. RECORD findings in your reasoning (don't wait for user)
   d. Call rlm_get_chunk(index=N+1)
4. After final chunk: AGGREGATE all findings
5. Present structured result

Example reasoning flow:

• Chunk 0: "Found references to articles 470-1, 1351. Theme: autorité de chose jugée"
• Chunk 1: "Found article 30-3, nationality dispute. New theme: désuétude"
• Chunk 2: "Terrorism case, articles 706-16-1, L.217-6. Theme: JIVAT competence"
• AGGREGATE: "3 decisions covering 3 distinct legal themes..."

Pattern 2: Search Then Deep-Dive

Use when: looking for specific known patterns, answering targeted questions.

1. Load file with rlm_load_context  
2. Search with rlm_search_context using relevant keywords/regex
3. ANALYZE search results - are they sufficient?
4. If needed, use rlm_slice_context for more context around matches
5. Synthesize findings

Warning: Search can miss context. If comprehensive coverage matters, use Pattern 1.

Pattern 3: Probe Then Process

For unknown file structure (explore before deciding approach):

1. Load file with rlm_load_context (preview shows first 1000 chars)
2. Optionally: rlm_slice_context lines 0-100 to see structure
3. Optionally: rlm_execute_code to count patterns, get statistics
4. Decide: full chunked processing vs targeted search
5. Execute chosen approach

Pattern 4: Multi-File Analysis

For analyzing patterns across multiple files:

1. Load each file with unique context_name: file1, file2, file3...
2. Process each file using Pattern 1 or 2
3. Cross-reference findings across contexts
4. Aggregate into unified analysis

Chunking Strategy

Default chunk size is 50,000 characters (~12K tokens). Adjust based on task:

• Dense analysis (legal, code review): Use 30K chunks, overlap 1000
• Light extraction (logs, data): Use 80K chunks, overlap 200
• Keyword search: Use Pattern 2 instead of full chunking

Example: Log Analysis

User: "Find all errors in this 500MB log file and categorize them"

1. rlm_load_context(file_path="/var/log/app.log", context_name="logs")
   → Shows 10M chars, ~200 chunks

2. rlm_search_context(context_name="logs", query="ERROR|FATAL|Exception", use_regex=True)
   → Returns 150 matches with positions

3. For each error category found, rlm_slice_context to get full stack traces

4. Aggregate: "Found 3 error categories: DatabaseTimeout (45 occurrences), 
   NullPointerException (72 occurrences), AuthenticationFailed (33 occurrences)..."

Example: Document Summarization

User: "Summarize this 800-page PDF transcript"

1. rlm_load_context(file_path="transcript.txt", context_name="doc")
   → Shows 2M chars, ~40 chunks

2. Loop through all 40 chunks with rlm_get_chunk:
   - Chunk 0: "Introduction, speakers introduced, agenda outlined..."
   - Chunk 1: "Discussion of Q1 results, revenue up 15%..."
   - ... continue for all chunks, tracking key points

3. Final aggregation: Synthesize all chunk summaries into coherent document summary

Important Notes

• You ARE the sub-LM: No external API - your reasoning IS the recursive processing
• Reason after EVERY chunk: Don't just fetch mechanically; analyze, extract, accumulate
• State lives in your reasoning: Track findings mentally across all chunks
• Aggregate before responding: User sees only final synthesis, not intermediate chunks
• Natural boundaries: Chunks break at newlines; overlap prevents missing split items
• Memory efficient: Only one chunk in context at a time