map-reduce

name	map-reduce
description	Orchestration pattern for large-scale processing where work is distributed, processed independently, then combined. Map phase splits and processes, Reduce phase aggregates results. Use for codebase-wide analysis, bulk transformations, or any task where "do X to everything, then summarize."
allowed-tools	bash: ls, cat, grep, find file: read, write mcp: task

Map-Reduce

Some tasks are too large for single-pass processing: analyze every file in a codebase, check all endpoints for issues, transform all components to new pattern. Map-Reduce handles this: split the work (map), process in parallel, combine results (reduce). It's batch processing for Claude.

When To Activate

Trigger when:

• Task involves "all files" or "every X"
• Processing a large number of similar items
• Need aggregate statistics or summaries across many items
• Transformation must be applied uniformly
• Analysis must cover entire codebase/dataset

Do NOT trigger for:

• Small number of items (just do them directly)
• Items requiring different processing logic
• When intermediate results affect how later items are processed

The Pattern

                 MAP PHASE                          REDUCE PHASE
              ┌────────────┐
              │ Process A  │──→ Result A ─┐
[All Items]   ├────────────┤              │    ┌─────────────┐
     │        │ Process B  │──→ Result B ─┼──→ │  Aggregate  │──→ [Final]
     │        ├────────────┤              │    └─────────────┘
     └──Split─│ Process C  │──→ Result C ─┘
              ├────────────┤
              │ Process D  │──→ Result D ─┘
              └────────────┘

Instructions

Step 1: Define the Job

Specify what you're processing:

Map-Reduce Job: [Name]

Input: [What collection of items]
Map function: [What to do to each item]
Reduce function: [How to combine results]
Expected output: [What the final result looks like]

Step 2: Enumerate Items (Map Input)

List everything to process:

Items to process:
1. [Item 1] - [brief description]
2. [Item 2] - [brief description]
3. [Item 3] - [brief description]
...
Total: [N] items

For large sets, use patterns:

Items: All files matching src/**/*.ts
Count: ~150 files
Batching: Groups of 10

Step 3: Execute Map Phase

Process each item (parallel when possible):

═══════════════════════════════════════
MAP PHASE: Processing [N] items
═══════════════════════════════════════

Batch 1 (items 1-10):
[Processing...]
- Item 1: [Result]
- Item 2: [Result]
...

Batch 2 (items 11-20):
[Processing...]
...

Map phase complete: [N] items processed
- Succeeded: [X]
- Failed: [Y]
- Skipped: [Z]

Step 4: Collect Intermediate Results

Gather all map outputs:

Intermediate results:
┌──────────┬─────────────────────────────┐
│ Item     │ Map Result                  │
├──────────┼─────────────────────────────┤
│ Item 1   │ [Result summary]            │
│ Item 2   │ [Result summary]            │
│ ...      │ ...                         │
└──────────┴─────────────────────────────┘

Step 5: Execute Reduce Phase

Aggregate results:

═══════════════════════════════════════
REDUCE PHASE: Aggregating results
═══════════════════════════════════════

Reduction strategy: [How combining]

Aggregating...

Categories identified:
- Category A: [N] items
- Category B: [M] items

Statistics:
- Total processed: [X]
- Issues found: [Y]
- Patterns detected: [Z]

Step 6: Produce Final Output

Present combined results:

═══════════════════════════════════════
MAP-REDUCE COMPLETE: [Job Name]
═══════════════════════════════════════

## Summary
[High-level findings]

## Statistics
- Items processed: [N]
- [Metric 1]: [Value]
- [Metric 2]: [Value]

## Categories/Groups
[Breakdown by category]

## Notable Items
[Specific items worth highlighting]

## Recommendations
[Actions based on findings]

Batching Strategies

**By count:** ``` Items: 150 files Batch size: 10 Batches: 15 ```

By directory:

Items: All TypeScript files
Batches: src/api/*, src/components/*, src/utils/*

By type:

Items: All source files
Batches: *.ts, *.tsx, *.css

Adaptive:

Start with batch of 20
If too slow → reduce to 10
If fast → increase to 30

Reduce Functions

**Count:** Total items matching criteria **Sum:** Aggregate numeric values **Group:** Categorize by attribute **Filter:** Surface items matching condition **Top-N:** Highest/lowest by metric **Histogram:** Distribution across buckets **Merge:** Combine into single artifact

NEVER

• Process items that depend on other items' results (use pipeline)
• Ignore failed items - always report what couldn't be processed
• Reduce without the map phase (need individual results first)
• Skip batching for large sets (will timeout/overwhelm)
• Assume uniform results - expect variation

ALWAYS

• Define map and reduce functions before starting
• Batch appropriately for the workload
• Track success/failure per item
• Produce actionable summary, not just raw data
• Include "notable items" - outliers worth attention

Examples

Example 1: Security Audit

User: "Check all API endpoints for authentication issues"

Map-Reduce Job: API Security Audit

Input: All files in src/api/**/*.ts
Map function: Check each file for auth patterns
Reduce function: Group by issue severity

═══════════════════════════════════════
MAP PHASE: Processing 23 files
═══════════════════════════════════════

- users.ts: [Missing rate limit on /login]
- orders.ts: [No auth on /history endpoint]
- products.ts: [Clean]
- admin.ts: [Deprecated auth method]
...

═══════════════════════════════════════
REDUCE PHASE: Aggregating results
═══════════════════════════════════════

By severity:
- Critical: 2 files (orders.ts, payments.ts)
- Warning: 5 files
- Clean: 16 files

═══════════════════════════════════════
MAP-REDUCE COMPLETE: API Security Audit
═══════════════════════════════════════

Summary: 7 of 23 endpoints have auth issues

Critical (fix immediately):
1. orders.ts:45 - GET /history has no auth check
2. payments.ts:23 - POST /refund missing admin check

Recommendations:
1. Add auth middleware to orders router
2. Implement admin check on payments

Example 2: Codebase Metrics

User: "How complex is our codebase? Get metrics on all files."

Map-Reduce Job: Codebase Complexity Analysis

Input: All source files
Map function: Count lines, functions, cyclomatic complexity
Reduce function: Aggregate statistics, find outliers

═══════════════════════════════════════
MAP PHASE: Processing 234 files
═══════════════════════════════════════

[Batched processing of all files...]

═══════════════════════════════════════
REDUCE PHASE: Aggregating results
═══════════════════════════════════════

Totals:
- Lines of code: 45,230
- Functions: 1,847
- Average complexity: 4.2

Distribution:
- Low complexity (<5): 78%
- Medium (5-10): 18%
- High (>10): 4%

═══════════════════════════════════════
MAP-REDUCE COMPLETE
═══════════════════════════════════════

Notable outliers (high complexity):
1. src/utils/parser.ts - complexity 23
2. src/api/legacy/converter.ts - complexity 19

Recommendations:
Consider refactoring top 5 complex files.

Example 3: Bulk Transformation

User: "Convert all class components to functional components"

Map-Reduce Job: Class → Functional Conversion

Input: All React component files
Map function: Convert class to functional if applicable
Reduce function: Track conversions, summarize changes

═══════════════════════════════════════
MAP PHASE
═══════════════════════════════════════

- Header.tsx: Converted (was class)
- Button.tsx: Already functional (skipped)
- Modal.tsx: Converted (was class)
- LegacyForm.tsx: Cannot convert (uses getDerivedStateFromProps)
...

═══════════════════════════════════════
REDUCE PHASE
═══════════════════════════════════════

Summary:
- Converted: 34 components
- Already functional: 56 components
- Cannot convert: 3 components
- Failed: 1 component

Changes made to 34 files.

What DOESN'T work: - No batching on 500+ files: Timeout, context overflow - Vague map function: "Check for issues" → inconsistent results - No reduce strategy: End up with 200 disconnected bullet points - Processing order-dependent items: Results inconsistent - Ignoring failures: Miss important edge cases

Why This Elixir Exists

"Check everything" is easy to say, hard to do well. Without structure, you get incomplete coverage, inconsistent analysis, and no useful summary.

Map-Reduce brings discipline to bulk operations: every item processed uniformly, failures tracked, results aggregated meaningfully. It's the difference between "I looked at some files" and "I analyzed all 234 files, here's what I found."

Scale requires structure. This is that structure.