memory-load-check

name	memory-load-check
description	Review PRs and diffs for unbounded memory loading, concurrency explosions, oversized payload materialization, and missing pagination or byte caps. Use when reviewing cleanup jobs, background jobs, data imports/exports, file parsing, API fan-out, workflow execution payloads, large arrays/files, or any change that reads many rows, files, responses, logs, or external API pages into process memory.

Memory Load Check

Use this skill when a PR or diff could load unbounded data into a Node/Bun process, especially in cron routes, background tasks, API routes, workflow execution, file parsing, cleanup jobs, migrations, import/export flows, and external API integrations.

Review Goal

Prove each changed path has explicit bounds for:

• rows held in memory
• bytes held in memory
• concurrent promises, DB queries, HTTP calls, storage operations, and jobs
• number of pages, batches, chunks, retries, and retained intermediate objects

If any bound depends only on current production size or "probably small" data, treat it as a finding.

References

Read these when doing a deeper pass:

• Node.js streams/backpressure: https://nodejs.org/learn/modules/backpressuring-in-streams
• Node.js stream usage: https://nodejs.org/en/learn/modules/how-to-use-streams
• Keyset/cursor pagination over offset scans: https://blog.sequinstream.com/keyset-cursors-not-offsets-for-postgres-pagination/
• Postgres pagination tradeoffs: https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/

Sim Helpers To Prefer

• apps/sim/lib/cleanup/batch-delete.ts
  • chunkedBatchDelete: bounded SELECT -> optional side effect -> DELETE loop.
  • batchDeleteByWorkspaceAndTimestamp: common workspace/timestamp cleanup wrapper.
  • selectRowsByIdChunks: chunks large ID sets and enforces an overall row cap.
  • chunkArray: use only after the input set itself is already bounded.
• apps/sim/lib/core/utils/stream-limits.ts
  • PayloadSizeLimitError
  • assertKnownSizeWithinLimit
  • assertContentLengthWithinLimit
  • readStreamToBufferWithLimit
  • readNodeStreamToBufferWithLimit
  • readResponseToBufferWithLimit
  • readResponseTextWithLimit
• Cleanup dispatcher pattern in apps/sim/lib/billing/cleanup-dispatcher.ts
  • page active workspaces with WHERE id > afterId ORDER BY id LIMIT N
  • dispatch concrete chunks (workspaceIds, retention, label) instead of one giant scope
  • prefer Trigger.dev queue/concurrency keys when available
  • execute inline fallback chunks sequentially, not with unbounded Promise.all
• File parse route pattern in apps/sim/app/api/files/parse/route.ts
  • cap downloads and parsed output separately
  • preserve partial results when a later item exceeds the cap
  • never read untrusted response bodies without a byte cap
• KB connector file downloads in apps/sim/connectors/utils.ts
  • CONNECTOR_MAX_FILE_BYTES: shared per-file cap (aligned with the manual KB upload limit)
  • readBodyWithLimit: stream a download body to a Buffer with a hard byte cap (null on overflow)
  • stubOrSkipBySize: listing-time skip when the reported size exceeds the cap
  • markSkipped / sizeLimitSkipReason: surface oversized files as failed (skipped) KB rows
  • ConnectorFileTooLargeError: thrown mid-download when the listing under-reported size
• Large workflow value payloads
  • prefer durable references/manifests over inlining large arrays or files
  • materialize refs only behind an explicit byte budget

KB Connector File Size Handling

The connector size pattern in apps/sim/connectors/utils.ts (CONNECTOR_MAX_FILE_BYTES + readBodyWithLimit + stubOrSkipBySize/markSkipped) exists for one risk: a knowledge-base connector downloading arbitrary, user-controlled file bytes that the source does not hard-cap. Apply it by that risk, not by the connector's name.

Use the pattern when the connector downloads file content via a stream/download_url where the user controls the size:

• file-storage connectors: Dropbox, OneDrive, SharePoint, Google Drive, S3, GitHub, GitLab, Azure DevOps
• any connector that fetches a file via a download URL even if it is not a "storage" service (e.g. the Zoom transcript .vtt)

For those, require all three:

• stream the body with readBodyWithLimit(resp, CONNECTOR_MAX_FILE_BYTES) — never raw response.text()/response.arrayBuffer()
• skip oversize at listing (stubOrSkipBySize with the reported size) and again at fetch time (overflow -> markSkipped), since the listing size can be missing or under-reported
• never drop/truncate silently — oversized files become content-less failed rows carrying skippedReason, so they stay visible in the KB UI instead of vanishing from the index

Skip the pattern when the source already bounds the payload:

• pure API/structured-data connectors (Jira, Linear, Notion, Confluence, Sentry, Slack, Zendesk, Gmail, ...) — paginated JSON/text; apply normal pagination + concurrency bounds instead of a per-file byte cap
• native-document connectors capped by the platform (Google Docs ~50 MB, Google Sheets via MAX_ROWS, Evernote ~25 MB/note) — a 100 MB cap can never fire, and wrapping a response.json()/Thrift parse in readBodyWithLimit is cargo-culting

Litmus test: "Can a user make this one fetch arbitrarily large, with nothing upstream stopping it?" Yes -> use the pattern. No (platform hard-cap, or already paginated) -> a per-file byte cap adds noise, not safety. Borderline: a user-configured/self-hosted endpoint with no platform cap (e.g. Obsidian) — bound it only if the content is genuinely unbounded.

Review Workflow

1. Identify every changed data source:
   • database queries
   • storage lists/downloads/uploads
   • external API pagination
   • file reads and HTTP responses
   • workflow logs, snapshots, payloads, arrays, and manifests
   • queues, cron routes, and background jobs
2. For each source, write down the maximum cardinality and maximum bytes. If the code does not enforce one, it is unbounded.
3. Trace whether data is processed incrementally or accumulated:
   • arrays from select, findMany, Promise.all, map, filter, flatMap
   • maps/sets keyed by all users, workspaces, executions, files, or rows
   • Buffer.concat, response.arrayBuffer(), response.text(), JSON.stringify, JSON.parse
   • queues of promises or job payloads built before dispatch
4. Check concurrency separately from memory:
   • no Promise.all(items.map(...)) unless items is already small and bounded
   • use chunks, sequential loops, queue concurrency, or a concurrency limiter
   • align concurrency with DB pool size, storage/API limits, and task queue semantics
5. Verify SQL shape:
   • every bulk query has LIMIT
   • large pagination uses cursor/keyset style (id > afterId, timestamps plus unique ID), not deep OFFSET
   • IN (...) lists are chunked
   • side-effect rows selected before delete have per-batch and per-run caps
6. Verify byte safety:
   • check Content-Length when available
   • stream with cumulative byte accounting
   • cap both input bytes and expanded output bytes
   • reject or reference oversized values before serializing large JSON responses
7. Confirm failure behavior:
   • exceeding a cap should stop before loading more data
   • partial successful work should be preserved when the API contract expects it
   • retries should not duplicate huge in-memory state
   • cleanup jobs should make progress over future runs instead of widening one run

Red Flags

• loads all active workspaces, users, executions, logs, files, messages, or subscriptions before filtering
• builds a full Map or Set for a platform-wide scope
• uses Promise.all over rows from an unbounded query
• fetches all pages from an external API before processing
• reads an entire file, HTTP response, or stream without a max byte budget
• checks size only after Buffer.concat, arrayBuffer, text, JSON.parse, or parse expansion
• a KB connector silently drops or truncates an oversized file instead of recording it as a failed (skipped) row
• chunks only after loading the complete dataset
• paginates with unbounded/deep OFFSET on a mutable or large table
• creates one queue job per row without batching or a queue-level concurrency key
• accumulates per-row errors/results with no maximum
• adds a cache, singleton, or module-level collection without eviction or size limits

Preferred Fixes

• Move filters into SQL/API requests and select only needed columns.
• Replace full-table loads with cursor/keyset pagination and a deterministic order.
• Process one page/batch at a time; do not keep previous pages unless needed.
• Add per-batch and per-run row caps so long backlogs drain across repeated jobs.
• Split large ID lists with selectRowsByIdChunks or chunkArray after bounding the source.
• Use chunkedBatchDelete for cleanup loops with row side effects.
• Use stream-limit helpers for file/HTTP/body reads.
• Store large workflow values as refs/manifests and materialize only within a caller budget.
• Replace unbounded Promise.all with sequential chunk loops, queue concurrency, or a small limiter.
• Include tests that prove caps stop work early and partial results or progress are preserved.

Findings Format

Lead with concrete findings, ordered by risk:

## Findings

- **P1 Unbounded workspace load in cleanup dispatch** (`path/to/file.ts`)
  The new path calls `select().from(workspace)` without a limit, then builds maps for every row before dispatch. In production this scales with all active workspaces and can exhaust the app process. Page by `workspace.id` with a fixed limit and dispatch bounded chunks.

## Good Signals

- Uses `readResponseToBufferWithLimit` for external downloads.
- Inline fallback processes chunks sequentially.

## Residual Risk

- The row cap is explicit, but no test currently proves the loop stops at the cap.

Only say "good to go" when every changed source has explicit row, byte, and concurrency bounds or the boundedness is proven by a stable invariant.