| name | fluent-iterable |
| description | | Use when this capability is needed. |
fluent-iterable Guide: Functional Data Transformations
A comprehensive guide to mastering @codibre/fluent-iterable for elegant, chainable data transformations in TypeScript/NestJS projects.
Quick Decision Tree
Do you have data to transform?
├─ Is it an AsyncIterable (stream, generator)?
│ ├─ YES → Use fluentAsync()
│ └─ NO → Go to next
├─ Is it a regular array/Map/iterable?
│ ├─ YES → Use fluent()
│ └─ NO → Check if it's an async generator
└─ Will you chain async operations?
├─ YES → Use .forEachAsync() or .flatMapAsync()
└─ NO → Use normal methods (.forEach(), .flatMap())
The Two Main Functions
fluent() - For Synchronous Iterables
Use fluent() with arrays, Map entries, Sets, or any sync iterable.
import { fluent } from '@codibre/fluent-iterable';
const numbers = [1, 2, 3, 4, 5];
const result = fluent(numbers)
.filter(x => x > 2)
.map(x => x * 2)
.toArray();
Key characteristics:
- ✅ Synchronous execution - returns immediately
- ✅ No awaits needed
- ✅ Perfect for data transformation pipelines
- ❌ Cannot handle AsyncIterables
- ❌ Cannot use
.forEachAsync() or .flatMapAsync()
fluentAsync() - For Asynchronous Iterables
Use fluentAsync() with streams, async generators, or database cursors.
import { fluentAsync } from '@codibre/fluent-iterable';
const activeUsers = await fluentAsync(
this.userRepository.streamUsers()
)
.filter(user => user.isActive)
.map(user => user.id)
.toSet();
Key characteristics:
- ✅ Handles AsyncIterables (streams, generators)
- ✅ Returns a Promise you must await
- ✅ Memory efficient - doesn't load all data at once
- ✅ Can use async-specific methods
- ❌ Slightly slower than sync (but necessary for I/O)
Common Patterns
Pattern 1: Stream Processing (Most Efficient for Large Data)
Use case: Process 5000+ records from database without loading all in memory
const successfulUserIds = await fluentAsync(
this.messageRepository.findByAnnouncementIdStream(announcementId)
)
.filter(msg => msg.sendError === null)
.map(msg => msg.userId)
.toSet();
Why this works:
- Uses async generator that yields results one at a time
- Memory usage is O(1) relative to dataset size
- Perfect for pagination without explicit loops
Pattern 2: Complex Transformations with Maps
Use case: Invert a map (userId → tokens) to (token → userId)
const tokenToUserId = fluent(userTokens.entries())
.flatMap(([userId, tokens]) =>
tokens.map(token => [token, userId] as const)
)
.reduce(
(acc, [token, userId]) => {
acc.set(token, userId);
return acc;
},
new Map<string, string>()
);
Why this works:
entries() converts Map to iterable of [key, value] pairs
flatMap() flattens nested arrays
reduce() with initial accumulator (new Map)
- All synchronous - no await needed
Pattern 3: Chunking for Batch Operations
Use case: Process 5000 items in chunks of 100 for API limits
const chunks = fluent(largeList)
.partition(100)
.map(chunk => ({
items: chunk.toArray(),
count: chunk.count()
}))
.toArray();
for (const {items, count} of chunks) {
await firebase.sendBatch(items);
}
Why this works:
partition() returns iterable of iterables
- Each chunk maintains fluent interface
- Only materialize to array when needed
Pattern 4: Parallel Async Operations
Use case: Save 1000 messages to DB in parallel, not sequentially
await fluent(payload.recipients)
.filter(recipient => !alreadyProcessed.has(recipient.userId))
.forEachAsync(async (recipient) => {
await this.messageRepository.saveMessage({
recipientId: recipient.userId,
message: this.renderTemplate(recipient),
});
});
Why this works:
forEachAsync() runs all operations in parallel
- Await only completes when all finish
- No need for Promise.all() boilerplate
- Handles backpressure automatically
Pattern 5: Flatten and Transform with Async
Use case: Depaginate multiple API calls in parallel
const allResellers = await fluent(chunks)
.flatMapAsync(chunk =>
this.api.getResellers({
codes: chunk.codes,
limit: 100
})
)
.toArray();
Why this works:
- Each chunk triggers an API call in parallel
- Results are automatically flattened into single array
- Equivalent to Promise.all() but cleaner
Common Methods Reference
Transformation Methods
| Method | Sync | Async | Use Case |
|---|
.map(fn) | ✅ | ✅ | Transform each element |
.map(fieldName) | ✅ | ✅ | Extract field: map('email') = map(x => x.email) |
.filter(predicate) | ✅ | ✅ | Keep matching elements |
.filter() | ✅ | ✅ | Remove falsy (null/undefined/false/0/'') |
.filter(fieldName) | ✅ | ✅ | Filter + type guard: filter('email') removes undefined |
.flatMap(fn) | ✅ | ❌ | Map + flatten (sync) |
.flatten(fn) | ✅ | ❌ | Flatten nested iterables |
.flatMapAsync(fn) | ❌ | ✅ | Map + flatten (async) |
.partition(size) | ✅ | ✅ | Split into chunks (returns iterable of iterables) |
.group(keyFn) | ✅ | ✅ | Group elements by key |
.group(fieldName) | ✅ | ✅ | Group by field: group('category') = group(x => x.category) |
.distinct() | ✅ | ✅ | Remove duplicates (like toSet() but lazy) |
.sort(compareFn?) | ✅ | ✅ | Sort elements |
.reverse() | ✅ | ✅ | Reverse order |
.first() / .firstAsync() | ✅ | ✅ | Get first element (stops iteration) |
.last() / |
Iteration & Execution Methods
| Method | Sync | Async | Effect |
|---|
.forEach(fn) | ✅ | ❌ | Execute for each, sync |
.forEachAsync(fn) | ❌ | ✅ | Execute in parallel, await (runs all concurrently) |
.execute(fn) | ✅ | ✅ | Side effect (logging) without modifying iterable |
.waitAll(promiseFn) | ❌ | ✅ | Convert to promises and await all (parallel) |
.combine(other, keyFn1, keyFn2) | ✅ | ✅ | Inner join two iterables |
Materialization Methods
| Method | Returns | Use | Memory |
|---|
.toArray() | Array<T> | Final result to array | O(n) |
.toSet() | Set<T> | Deduplicate + set | O(n) |
.toMap(keyFn) | Map<K, T> | Create lookup index | O(n) |
.toMap(keyFn, valueFn) | Map<K, V> | Transform keys and values | O(n) |
.count() | number | Get length (no array allocation) | O(1) or O(n) depending on source |
.any(predicate?) | boolean | Check if any match (short-circuits) | O(1)-O(n) |
.all(predicate?) | boolean | Check if all match (short-circuits) | O(1)-O(n) |
.join(separator) | string | Join to string | O(n) |
String Selectors & Smart Shortcuts
✨ filter() Without Parameters - Clean Null/Falsy Filtering
Direct usage - no callback needed:
const activeIds = await fluentAsync(users)
.filter()
.map(user => user.id)
.toArray();
const activeIds = await fluentAsync(users)
.filter(user => !!user)
.map(user => user.id)
.toArray();
Perfect for streams that might have nulls:
const validRecords = await fluentAsync(asyncGenerator())
.filter()
.map(record => record.process());
✨ String Selectors - Field Names as Shortcuts
Use field names instead of callbacks - cleaner and with auto type guards:
fluent(users)
.map(user => user.email)
.filter(email => !!email)
fluent(users)
.map('email')
.filter('email')
interface User {
id: number;
email?: string;
name: string;
}
const usersWithEmail = fluent(users)
.filter('email')
.map('email');
String selector patterns:
fluent(products)
.map('name')
.toArray()
fluent(users)
.filter('email')
.filter('isActive')
.forEach(user => {
console.log(user.email.toLowerCase());
});
fluent(orders)
.filter('customerId')
.filter('total')
.map('total')
.sum();
Why string selectors are powerful:
- Less boilerplate:
map('field') vs map(x => x.field)
- Auto type guards:
filter('field') narrows type (removes | undefined)
- Cleaner chains: Especially useful in long pipelines
- Readable intent:
filter('email') is clearer than filter(x => x.email)
const validatedUsers = await fluentAsync(streamUsers())
.filter('id')
.filter('email')
.filter('isVerified')
.map('email')
.distinct()
.toArray();
✨ TypeScript Type Guards in Filter Callbacks
Declare type guards with is keyword - fluent-iterable propagates narrowed types!
Type guards are TypeScript's way of narrowing types. When you use is in a filter callback, fluent-iterable understands and propagates the narrowed type through the entire chain.
function isUser(item: User | Admin | null): item is User {
return item !== null && 'email' in item && !('permissions' in item);
}
const emails = fluent(users)
.filter(isUser)
.map(user => user.email)
.toArray();
const emails = fluent(users)
.filter((item): item is User =>
item !== null && 'email' in item && !('permissions' in item)
)
.map(user => user.email)
.toArray();
Real-world examples with type guards:
type ApiResponse = { status: 'success'; data: string } | { status: 'error'; error: Error };
function isSuccess(response: ApiResponse): response is { status: 'success'; data: string } {
return response.status === 'success';
}
const successData = fluent(responses)
.filter(isSuccess)
.map(r => r.data)
.toArray();
interface Product {
id: number;
name: string;
description?: string;
price?: number;
}
function hasPrice(product: Product): product is Product & Required<<, >> {
product. !== && product. > ;
}
priced = (products)
.(hasPrice)
.( p.)
.();
{
= ,
= ,
= ,
}
(): user is { : . } {
user. === .;
}
activeUsers = (allUsers)
.(isActive)
.( {
});
Inline Type Guards - Declare directly in lambda for simpler checks:
No need for separate functions when the guard is simple. Use inline syntax with (param): param is Type =>:
const emails = fluent(items)
.filter((item): item is { email: string } =>
typeof item === 'object' && item !== null && 'email' in item && typeof item.email === 'string'
)
.map(item => item.email)
.toArray();
const successData = fluent(responses)
.filter((r): r is { status: 'success'; data: string } => r.status === 'success')
.map(r => r.data)
.toArray();
const productsWithPrice = fluent(products)
.filter((p): p is typeof p & { price: number } => p.price !== undefined && p.price > 0)
.map(p => p.)
.();
= { : ; : ; : } | { : ; : };
updates = (actions)
.((action): action is { : ; : ; : } =>
action. ===
)
.( action.)
.();
When to use inline vs separate function:
fluent(items).filter((x): x is User => !!x.id && !!x.email)
function isValidUser(item: any): item is User {
return (
item !== null &&
typeof item === 'object' &&
'id' in item &&
'email' in item &&
/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(item.email) &&
item.id > 0
);
}
fluent(items).filter(isValidUser)
Why inline type guards are powerful:
- No boilerplate: One-liners for simple checks
- Self-documenting: Guard logic visible where it's used
- Still type-safe: Full compiler support same as separate functions
- Scoped: Guard only used where needed
Why type guards are powerful:**
- Type safety: Compiler catches errors after narrowing
- No runtime checks needed: Guard is purely TypeScript
- Reusable: Define once, use everywhere
- Propagates through chain: Every method after knows the narrowed type
- IDE autocomplete: IntelliSense shows only available properties
fluent(items)
.filter(item => item.type === 'user')
.map(item => item.name)
.forEach(name => console.log(name));
function isUser(item: any): item is { name: string; type: 'user' } {
return item.type === 'user' && typeof item.name === 'string';
}
fluent(items)
.filter(isUser)
.map(item => item.name)
.forEach(name => console.log(name));
Critical Rules
✅ DO:
-
Use fluent() for arrays and sync iterables
fluent([1, 2, 3]).map(...).filter(...)
-
Use fluentAsync() for AsyncIterables
fluentAsync(asyncGenerator()).filter(...).toSet()
-
Always await fluentAsync results
const result = await fluentAsync(...).toArray();
-
Use .forEachAsync() for async side effects
await fluent(items).forEachAsync(item => saveAsync(item));
-
Materialize only at the end
fluent(x).map(...).filter(...).toArray()
fluent(x).toArray().map(...).filter(...)
-
Use string selectors for cleaner code
fluent(users)
.()
.()
(users)
.( !!user.)
.( user.)
❌ DON'T:
-
Don't use fluent() with AsyncIterables
fluent(asyncGenerator())
fluentAsync(asyncGenerator())
-
Don't forget await on async operations
const result = fluentAsync(...).toArray();
const result = await fluentAsync(...).toArray();
-
Don't use async methods on fluentAsync()
fluentAsync(array).forEachAsync(...)
await fluent(array).forEachAsync(...)
await fluenAsync(data).forEach(...)
-
Don't chain multiple .toArray() calls
fluent(x).toArray().map(...).filter(...)
fluent(x).map(...).filter(...).toArray()
Real-World Examples
Example 1: Process Stream, Skip Already Done, Batch Send
const successfulUserIds = await fluentAsync(
this.messageRepository.findByAnnouncementIdStream(announcementId)
)
.filter(msg => msg.sendError === null)
.map(msg => msg.receiverId)
.toSet();
const usersToProcess = fluent(payload.receiverIds)
.filter(id => !successfulUserIds.has(id))
.toArray();
const batches = fluent(userTokens.entries())
.flatMap(([userId, tokens]) =>
tokens.map(token => ({ userId, token }))
)
.partition(100)
.map(batch => batch.toArray())
.toArray();
await fluent(batches)
.forEachAsync(batch =>
..(batch.( x.))
);
Example 2: Depaginate Multiple API Calls
const allData = await fluent(sellers)
.partition(500)
.map(chunk => ({
...request,
codes: chunk.toArray()
}))
.toArray()
.flatMapAsync(request =>
this.api.getData(request)
);
Example 3: Group and Count
const stats = fluent(users)
.reduce((acc, user) => {
const key = user.country;
acc.set(key, (acc.get(key) ?? 0) + 1);
return acc;
}, new Map<string, number>());
Advanced Patterns
Pattern A: Resilient Batch Processing with Error Handling
Use case: Process 5000+ items in batches while handling failures per batch (not failing entire operation)
const results = [];
for (const batch of fluent(items).partition(500)) {
try {
const tokenBatch = batch.toArray();
const result = await sendBatchToApi(tokenBatch);
results.push(result);
} catch (error) {
console.error(`Batch failed: ${error.message}`);
}
}
try {
for (const batch of fluent(items).partition(500)) {
await sendBatchToApi(batch.toArray());
}
} catch (error) {
}
Why this works:
- Failures are isolated per batch
- Successful batches still complete
- Perfect for distributed systems where partial success matters
- Accumulates results across all batches
Pattern B: O(1) Lookup Instead of O(n²)
Use case: Cross-reference 1000 items with 5000 recipients
const userMap = new Map();
for (const recipient of recipients) {
const user = items.find(i => i.id === recipient.userId);
userMap.set(recipient.userId, user);
}
const userLookup = fluent(items).toMap(i => i.id);
for (const recipient of recipients) {
const user = userLookup.get(recipient.userId);
}
const userLookup = fluent(items).toMap(i => i.id, i => i);
Why this works:
- Pre-computed Map provides O(1) lookups
- Reduces from n*m to n+m complexity
- Scales well with large datasets
- One-time cost of building map vs repeated .find() calls
Pattern C: Stream Processing to Stay Memory-Efficient
Use case: Process millions of database records without loading all into memory
const successCount = await fluentAsync(
this.database.streamLargeDataset()
)
.filter(record => record.status === 'active')
.map(record => record.id)
.count();
const allRecords = await fluentAsync(this.database.streamLargeDataset())
.toArray();
const successCount = fluent(allRecords)
.filter(r => r.status === 'active')
.count();
Why this works:
- Only holds current element in memory
- Filter/map applied lazily (per element)
- Perfect for pagination and large streams
- Async generators yield one item at a time
Pattern D: Inner Join Between Two Iterables
Use case: Match orders with customers by ID
const orders = [
{ id: 1, customerId: 10, amount: 100 },
{ id: 2, customerId: 20, amount: 200 },
];
const customers = [
{ id: 10, name: 'Alice' },
{ id: 20, name: 'Bob' },
];
const results = fluent(orders)
.combine(
customers,
(order) => order.customerId,
(customer) => customer.id
)
.map(([order, customer]) => ({
orderId: order.id,
customerName: customer.name,
amount: order.amount
}))
.toArray();
Why this works:
- Natural SQL-like join operation
- Avoids nested loops
- More readable than manual matching logic
Pattern E: Depagination with AsyncIterable
Use case: Fetch paginated API data until done
async function* depaginate<T>(
pageSize: number,
fetcher: (page: number) => Promise<T[]>
): AsyncIterable<T> {
let page = 1;
while (true) {
const items = await fetcher(page);
if (items.length === 0) break;
yield* items;
page++;
}
}
const allUsers = await fluentAsync(depaginate(100, page =>
fetchUserPage(page)
))
.filter(user => user.active)
.map(user => user.id)
.toArray();
Why this works:
- Lazy pagination (only fetches needed pages)
- No need to know total count upfront
- Stops automatically when empty page received
- Memory efficient (no buffering all pages)
Pattern F: Deduplication with Short-Circuit
Use case: Get unique values while being fast
const firstUniqueActive = fluent(items)
.distinct()
.filter(x => x.active)
.first();
const uniqueIds = fluent(items)
.map(item => item.id)
.toSet();
Why this works:
.distinct() is lazy (doesn't create set upfront)
.toSet() faster when you need complete set
- Choose based on whether you need all values or just some
Troubleshooting
Problem: "Cannot find property forEachAsync"
Cause: Using fluentAsync() instead of fluent()
fluentAsync(data).forEachAsync(...)
await fluent(data).forEachAsync(...)
await fluenAsync(data).forEach(...)
Problem: "Awaiting non-Promise"
Cause: Trying to await a sync fluent chain
const result = await fluent([1,2,3]).map(...);
const result = fluent([1,2,3]).map(...);
Problem: Memory keeps growing with streams
Cause: Calling .toArray() on entire stream before filtering
fluentAsync(stream).toArray().filter(...)
fluentAsync(stream).filter(...).toArray()
Problem: Operations run sequentially instead of parallel
Cause: Using .forEach() with async instead of .forEachAsync()
fluent(items).forEach(item => saveAsync(item));
await fluent(items).forEachAsync(item => saveAsync(item));
Performance Tips & Benchmarks
fluent-iterable Performance (from official benchmarks):
- ~494 ops/sec on large dataset iterations (Node 22)
- ~50% faster than native
for...of loops
- Faster than native iterator helpers (V8 engine)
- Similar to hand-written optimized loops
When to optimize:
- If processing 100k+ items frequently
- If chain has 5+ operations
- If you're on hot path (called thousands of times/sec)
General optimization order:
- Filter early (reduce data size)
- Use streaming when possible
- Avoid materialize-then-filter pattern
- Use
.count() instead of .toArray().length
- Use short-circuit methods (
.first(), .any()) instead of full iteration
Performance Tips
-
Filter early - Remove unwanted items before other operations
fluent(million_items)
.filter(x => x.active)
.map(x => x.value)
.toArray()
-
Use partition() for large batches
fluent(items)
.partition(100)
.map(chunk => processChunk(chunk))
-
Avoid nested loops - use flatMap instead
fluent(items).forEach(item => {
fluent(item.children).forEach(child => ...)
})
fluent(items)
.flatMap(item => item.children)
.forEach(child => ...)
-
For AsyncIterables, filter before reduce
await (stream)
.( (x))
.(aggregate, initial)
When NOT to Use fluent-iterable
✅ Use fluent-iterable when:
- Processing streams, async generators, or database cursors
- Building complex transformation chains (3+ operations)
- Working with large datasets (1000+ items)
- Need memory efficiency (streaming)
- Doing batch processing or pagination
- Performance matters (benchmarks show ~50% faster than for loops)
❌ Avoid fluent-iterable when:
- Processing small arrays (< 10 items) - overhead not worth it
- Need destructuring/spreading frequently -
.toArray() at start is simpler
- Only doing
.map() or .filter() once - just use native array methods
- Building a single value from scratch - use
.reduce() with plain JavaScript
fluent([1, 2, 3]).map(x => x * 2).toArray()
[1, 2, 3].map(x => x * 2)
await fluentAsync(largeStream)
.filter(x => x.active)
.map(x => x.id)
.group(x => x.category)
.toMap(g => g.key, g => g.values.count())
Comparison to Alternatives
vs RxJS Observables
| Aspect | fluent-iterable | RxJS |
|---|
| Purpose | Sequential iteration | Event streaming |
| Execution | Synchronous chain | Async/event-driven |
| Memory | O(1) with streams | Can accumulate events |
| Learning curve | Simple array-like API | Complex (Subjects, Operators) |
| Best for | Batch processing, data transforms | Real-time events, UI reactivity |
→ Use fluent-iterable for data pipelines, RxJS for event handling
vs Native Array Methods
| Aspect | fluent-iterable | .map().filter() |
|---|
| Iterations | 1 pass | Multiple passes per operation |
| Memory | Minimal (lazy) | Array created per operation |
| Performance | ~50% faster on 1k+ items | Faster on tiny arrays |
| Async support | ✅ Built-in | ❌ Need Promise.all() |
| Streaming | ✅ AsyncIterable | ❌ Must load all first |
→ Use fluent-iterable for large data, native methods for small arrays
See Also
Source: codibre/fluent-iterable — distributed by TomeVault.