api-utils

## Overview Utility exports from `@cyanheads/mcp-ts-core/utils`. Utilities with complex APIs have dedicated reference files; simpler utilities are documented inline below. **Tier 3** = optional peer dependency. Install as needed (e.g., `bun add js-yaml`). All Tier 3 methods are **async** (lazy-load deps on first call). **Context parameters.** Every helper below that takes a `context` accepts the handler `Context` as well as a `RequestContext` bag — pass `ctx` straight through, no slicing. ## References | Reference | Path | Covers | |:----------|:-----|:-------| | Formatting | `references/formatting.md` | `markdown()`, `MarkdownBuilder`, `diffFormatter`, `tableFormatter`, `treeFormatter` — builder patterns, option types, style variants, usage examples | | Parsing | `references/parsing.md` | `yamlParser`, `xmlParser`, `csvParser`, `jsonParser`, `pdfParser`, `dateParser`, `frontmatterParser` — method signatures, option types, peer deps, `Allow` flags, PDF workflows | | Security | `references/security.md` | `sanitization`, `RateLimiter`, `IdGenerator` — config types, method details, sensitive fields, usage examples | --- ## `@cyanheads/mcp-ts-core/utils` — network | Export | API | Notes | |:-------|:----|:------| | `fetchWithTimeout` | `(url, timeoutMs, context, options?: FetchWithTimeoutOptions) -> Promise<Response>` | Wraps `fetch` with `AbortController` timeout. `timeoutMs` bounds the **whole exchange**: on a 2xx carrying a body the returned `Response` is a passthrough wrapper that keeps the deadline armed until the body closes, errors, or is cancelled, so a stalled stream rejects the caller's `.text()`/`.json()` with the same `Timeout` error the header phase raises. `status`, `statusText`, `headers`, `url`, `redirected`, and `type` carry across the wrapper; the original body is locked by it, and bodyless/null-body responses (HEAD, 204/205/304) come back untouched. `FetchWithTimeoutOptions` extends `RequestInit` (minus `signal`) and adds `rejectPrivateIPs?: boolean`, `expectedStatuses?: number[]` (listed non-2xx statuses logged at `debug` not `error`, still thrown), `errorBodyLimit?: number` (bytes of a non-2xx body kept, default `500`), `errorHeaders?: string[]` (response headers copied onto `error.data.headers` on a non-2xx — same selector as `httpErrorFromResponse` below; `location` is selectable under `redirect: 'manual'` but does **not** compose with `rejectPrivateIPs`, whose per-hop branch consumes the 3xx before the throw path sees it), and `signal?: AbortSignal` (external cancellation — an abort on it throws `RequestCancelled` (-32011), logged at `info` and outside `withRetry`'s transient set, since the caller is gone and no retry can reach them; an abort whose reason is a `TimeoutError` — `AbortSignal.timeout()`, or an `AbortSignal.any` whose timeout member fired — is a deadline instead, and throws `Timeout` (-32004) with `data.errorSource: 'FetchSignalTimeout'`, distinct from the helper's own `timeoutMs` expiry, `'FetchTimeout'`, and likewise outside `withRetry`'s default transient set, since every retry would reuse the fired signal). Validation rejections carry `data.reason` and a `recovery.hint`: `invalid_url` (not an absolute `http:`/`https:` URL, including a redirect target), `private_address_blocked` (the SSRF guard refused the host by name, literal IP, or DNS answer), `too_many_redirects` (past the 5-hop cap, with `data.maxRedirects`). On a non-2xx, `error.data` carries `status`/`body` plus the legacy `statusCode`/`responseBody` aliases (identical values; consolidating in a future major); a body over `errorBodyLimit` is captured from both ends — 40% head, 60% tail, joined by `…[N bytes elided]…` — so a diagnostic behind a boilerplate preamble survives the cap, while a body still streaming at the 16 KiB scan ceiling stays head-only with a trailing `…`. SSRF guard (best-effort, not hard isolation): blocks RFC 1918, loopback, link-local, CGNAT, cloud metadata. DNS validation on Node, Bun, and Cloudflare Workers under `nodejs_compat`; hostname-only fallback otherwise. **Both resolvers are queried** — `resolve4`/`resolve6` (c-ares) and `lookup` (the system resolver, which is what reads `/etc/hosts`, split DNS, and NSS modules) — and a non-global answer from either rejects. Runtimes differ in which resolver the connection uses (Bun 1.4 moved `net.connect()` on Linux to `getaddrinfo` while leaving `dns.resolve*()` on c-ares), so checking one alone leaves a name the other can see unguarded; each probe settles independently, so a resolver absent from the runtime is skipped rather than fatal. Manual redirect following (max 5) with per-hop SSRF check. **DNS rebinding / TOCTOU gap** — the validation lookup and `fetch`'s own resolution are independent; pair with egress controls or a DNS-pinning fetch proxy for strong isolation. **Error/log redaction:** URLs written into thrown errors and log lines are reduced to `origin + pathname` — the query string (where API keys commonly ride: `?api-key=…`, `?api_key=…`) never reaches the client or the logs. The actual request still uses the full URL. | | `withRetry` | `<T>(fn: (attempt: RetryAttempt) => Promise<T>, options?: RetryOptions) -> Promise<T>` | Executes `fn` with exponential backoff. Retries on transient errors (`ServiceUnavailable`, `Timeout`, `RateLimited`); non-transient errors fail immediately. Honors an upstream `Retry-After` on `data.retryAfter` (delta-seconds or HTTP-date) over exponential backoff, capped at `maxDelayMs`; a requested wait beyond the cap fails fast rather than sleeping. On exhaustion, enriches the final error with attempt count in message and `data.retryAttempts`. **Place the retry boundary around the full pipeline** (fetch + parse), not just the network call. `RetryOptions`: `maxRetries` (default `3`), `baseDelayMs` (default `1000`), `maxDelayMs` (default `30000`), `jitter` (default `0.25`), `operation` (log label), `context` (RequestContext), `signal` (AbortSignal), `isTransient` (custom predicate), `deadlineMs` (total wall-clock budget — see below). | | `RetryAttempt` | `{ readonly signal: AbortSignal; readonly remainingMs: number }` | What `fn` receives each attempt. `signal` is `AbortSignal.any` over the `deadlineMs` clock and `options.signal`; `remainingMs` is what is left of the total budget as the attempt starts, never negative and `Number.POSITIVE_INFINITY` when no deadline is set — so `Math.min(perAttemptMs, remainingMs)` is correct either way. A zero-argument `fn` stays assignable, so existing callers compile unchanged. | | `deadlineMs` | `RetryOptions` field | One wall-clock budget across every attempt, backoff, and honored `Retry-After` — the bound `maxRetries` plus a per-attempt timeout cannot express. Four 30s attempts outlast a client's 60s request timeout, so the caller gets a transport timeout instead of the server's classified error. **Thread `attempt.signal` into the attempt's I/O** (`fetchWithTimeout(url, Math.min(30_000, remainingMs), ctx, { signal })`) or the deadline overshoots by one in-flight request. Clock is `AbortController` + `setTimeout` (never `AbortSignal.timeout()`, per the Bun realm mismatch), cleared on return — no timer outlives the call. Expiry rejects with `Timeout` (-32004) carrying `data: { reason: 'retry_deadline_exceeded', deadlineMs, elapsedMs, retryAttempts }` and the last attempt's error as `cause`; **one shape for every expiry**, including the per-attempt `Timeout` (`errorSource: 'FetchSignalTimeout'`) the clock's abort raises inside `fetchWithTimeout` and the raw abort reason a mid-backoff expiry would otherwise surface. No `retryable` flag (a narrower call can still succeed) and no `attempt` index (`retryAttempts` carries it). A backoff that would outlast the remaining budget fails fast with the expiry instead of sleeping into a certain timeout; an honored `Retry-After` that would outlast it takes the `maxDelayMs` exit instead — the attempt's error unchanged, `data.retryAfter` intact, since "wait the window the upstream named" is still the caller's action. **Three clocks stay distinct:** a caller abort on `options.signal` keeps precedence — mid-attempt it rethrows the attempt's error unchanged, mid-backoff it rejects with `signal.reason` itself (an `AbortError` `DOMException` for a reason-less `abort()`), and the handler factory reports either as `RequestCancelled` when the request signal is the one that fired — a single attempt's timeout is `Timeout` with `errorSource: 'FetchTimeout'` and no `reason`, and the expiry is `Timeout` with the `reason`. Unset, behavior is identical to before — attempt counts, delays, log lines, and the exhausted-error shape untouched. Bounds **one** ladder: a tool making three upstream calls threads its own remaining budget into each. | | `defaultIsTransient` | `(error: unknown) -> boolean` | The predicate `withRetry` uses when `isTransient` is omitted: an `McpError` with a transient code (`ServiceUnavailable`, `Timeout`, `RateLimited`) unless it carries `data.retryable === false`, `data.reason === 'pacer_shed'`, or `data.errorSource === 'FetchSignalTimeout'` (a caller-side deadline that already fired); any non-`McpError` throw is assumed transient. Exported so `isTransient` — which **replaces** the default outright — can compose instead of mirroring the transient set, which drifts silently when the framework's classification changes: `isTransient: (error) => !isMyBudgetRefusal(error) && defaultIsTransient(error)`, or the inverse `defaultIsTransient(error) \|\| isMyRetryableShape(error)`. The transient code set itself stays private (a module-level `Set` an exported binding could be mutated into framework-wide retry behavior). | | `httpErrorFromResponse` | `(response: Response, options?: HttpErrorFromResponseOptions) -> Promise<McpError>` | Maps an HTTP `Response` to a properly classified `McpError` — full status table including 401/403/408/422/429/5xx, body capture (truncated), `retry-after` header, optional `cause`. `error.data` carries `status`/`body` plus the legacy `statusCode`/`responseBody` aliases (identical values), so a consumer can classify either helper's error without knowing which raised it. Use this instead of hand-rolling `if (status === 429) ...` ladders. Reads the response body — `clone()` first if you need it elsewhere. **`error.data` is client-facing** — the framework forwards it verbatim as `structuredContent.error.data` — so the full upstream URL is **omitted by default**: a request URL routinely carries user input, internal identifiers, or an API key in its query string. `includeUrl: true` opts into `data.url` carrying the full `response.url`; with an empty `response.url` no key is added either way, and the message still names the host. Response headers are opt-in on the same footing: `errorHeaders: ['x-ratelimit-remaining-usd', 'x-request-id']` copies the named headers onto `data.headers` under **lowercase** keys — selection is case-insensitive and entries differing only in case collapse to one key, presence follows `Headers.has()` (an empty value is captured as `''`, an absent header adds no key), and a multi-valued field is captured comma-joined as `Headers.get()` returns it. Omitted, empty, or matching nothing, no `headers` key is emitted. `set-cookie` is **never** captured whatever the selector says: it is credential-bearing and `Headers.get()` joins its values into a string that is not a valid reconstruction. Every selected value reaches the client, so never name a header that carries a credential — and a selected `Location` can itself carry a sensitive path, query, or token. `HttpErrorFromResponseOptions`: `service?` (logical name in message, e.g. `'NCBI'`), `captureBody?` (default `true`), `bodyLimit?` (default `500`), `includeUrl?` (default `false`), `errorHeaders?` (default none), `data?` (extra fields merged into `error.data`, overriding defaults on key collision — a caller's own `url` or `headers` still reaches the wire), `cause?`, `codeOverride?` (per-status mapping override). Pairs naturally with `withRetry` — both classify codes the same way. A 501 also carries `data.retryable: false`, so retry fails it fast instead of re-asking for a method the upstream does not implement. | | `createPacer` | `(options: PacerOptions) -> Pacer` | FIFO queue in front of one rate-limited upstream — the outbound counterpart to `RateLimiter` (`utils/security`), which is inbound, per-caller, and reject-only, so it cannot queue work against an upstream budget. `pacer.run(task, { signal?, maxWaitMs? })` holds `task` until every `limits` window, `minStartGapMs`, `maxConcurrent`, and the cooldown gate allow it, then calls it with the caller's signal. `PacerOptions`: `name` (author-set telemetry label), `limits` (`{ requests, perMs }[]` — each a sliding window over recorded **start** times, so a slow response never widens the rate the upstream sees; all must allow a start), `minStartGapMs` (**not** expressible through `limits`: `{ requests: 10, perMs: 1000 }` permits ten starts in the same millisecond), `maxConcurrent`, `maxQueueDepth` (absolute backpressure for callers passing no `maxWaitMs`; rejects without arming a timer; bounds **waiters only** — an arrival whose slot is open that instant starts without queueing, so `0` means "run when a slot is free, never wait"), `cooldown` (`{ baseMs, maxMs }`). **Shed:** `maxWaitMs` bounds queue time only, never the task. The projected wait is exact over the windows and the gap but a lower bound once `maxConcurrent` binds (a slot frees on an unknowable completion), so enqueue rejects only when that lower bound already exceeds `maxWaitMs` — no false sheds — and a still-queued entry rejects when `maxWaitMs` elapses, unless its slot opens that same instant. The shed error is `rateLimited` (-32003) with `data: { reason: 'pacer_shed', shedKind, retryAfter, queueDepth }`. `shedKind` (`PacerShedKind`) is `queue_full` (the call would wait behind `maxQueueDepth` waiters), `wait_projected` (the enqueue projection exceeds `maxWaitMs`), or `wait_elapsed` (`maxWaitMs` ran out while queued), and the message follows the kind — a `queue_full` shed names the full queue, not a wait budget. `retryAfter` is seconds until a caller joining behind every remaining waiter could start; while `maxConcurrent` is saturated — a release the projection cannot see — it is floored at the longest wait of any queued caller, the shed one included, minimum 1. `queueDepth` is the waiters still queued. **No `retryable: false`** — to the calling agent a shed is an ordinary rate limit (wait `retryAfter`, call again) and that flag would say the opposite; `defaultIsTransient` reads the `reason` instead, so an enclosing `withRetry` fails fast rather than sleeping past the deadline the shed enforces. **Cooldown gate:** a `RateLimited` thrown by the task closes the gate for every queued caller until an absolute instant, `min(max(baseMs · 2^(consecutive−1), retryAfter), maxMs)` — `maxMs` caps both the doubling and an honored `Retry-After`, so a pathological upstream value cannot park the queue. Absent or unparseable `retryAfter` leaves the doubling; any other error leaves the gate open and the count untouched, and a shed (`reason: 'pacer_shed'`) from a pacer nested inside the task is local backpressure, never a gate closure. The first success resets the count, and so does a gate that has stood open for `maxMs`: the next rate limit starts over at `baseMs`, while one arriving sooner — the gate still closed included — keeps doubling, so continuous demand under a sustained limit keeps its capped backoff. **`pacer.cooldown`** samples the gate as `PacerCooldownState` `{ remainingMs, consecutive }`: `remainingMs` is the shared gate, not one rate limit's own computation (rate limits landing together close one gate at the later instant), so a task's rejection handler can report it on the server's own error — the pacer never writes to the task's error. Both stay 0 without `cooldown`. **Composition:** `withRetry(({ signal }) => pacer.run(fn, { signal }), { signal, deadlineMs })` — retry outside, pacer inside, so each attempt re-queues and is re-paced. Because the gate is an absolute instant rather than a duration counted from dequeue, retry's `Retry-After` sleep and the gate overlap in wall-clock instead of summing: the window is waited once, not twice. **Lifecycle:** timers and `AbortSignal` only, process-local; the dispatch timer is `unref()`'d where supported; `dispose()` / `[Symbol.dispose]()` clears it and rejects queued waiters with `RequestCancelled` (in-flight tasks are left to finish) — wire it through `createApp({ teardown })`. On Workers state is per-isolate so the limits bind per isolate, OTel is off so the metrics are inert, and `createWorkerHandler` accepts no `teardown`. Metrics: `mcp.pacer.queue_depth`, `mcp.pacer.wait`, `mcp.pacer.sheds`, `mcp.pacer.cooldowns`, attributed by `mcp.pacer.name` only — see `api-telemetry`. | | `httpStatusToErrorCode` | `(status: number) -> JsonRpcErrorCode \| undefined` | Sync status → code lookup. Returns `undefined` for 1xx/2xx. A 3xx maps to `InvalidRequest` — it reaches error mapping under `redirect: 'manual'`, where the request as sent cannot be served at this URL, and that code is outside `withRetry`'s transient set since re-issuing returns the same redirect. Use when you need just the code without a `Response` object handy. No status maps to `InternalError` — that code means *this* server failed, which a remote status cannot establish; every 5xx is `ServiceUnavailable` (or `Timeout` for 504) and so picks up `withRetry`'s default transient policy. | --- ## `@cyanheads/mcp-ts-core/utils` — pagination | Export | API | Notes | |:-------|:----|:------| | `extractCursor` | `(params?) -> string \| undefined` | Extracts opaque cursor string from MCP request params. Checks `params.cursor` then `params._meta.cursor`. Returns `undefined` when no cursor is present. Does not decode. | | `paginateArray` | `<T>(items, cursorStr, defaultPageSize, maxPageSize, context: RequestContext) -> PaginatedResult<T>` | Decodes cursor, slices array, returns `{ items, nextCursor?, totalCount }`. `nextCursor` omitted on last page. Throws `McpError(InvalidParams)` on invalid cursor, inherited from `decodeCursor` (`data.reason: 'invalid_cursor'`). On a continued call the page size comes from the cursor, not `defaultPageSize` — a tool with a caller-facing `limit` input must slice on `decodeCursor(...).offset` itself to honor `limit` past page 1. |