alpaca-broker-rate-limits-resilience

name	alpaca-broker-rate-limits-resilience
description	Make Alpaca API clients resilient — rate-limit header handling, HTTP 429 backoff, exponential retry, bounded concurrency/worker pools, pagination loops, batch sizing, and timeouts. Use when building robust REST clients, bulk/cron jobs, or reconciliation sweeps against Alpaca in any language.

Alpaca — Rate Limits & Resilience

Alpaca's APIs are rate-limited and occasionally flaky under load. Any client that does more than a handful of calls — especially bulk jobs, backfills, and reconciliation sweeps — needs disciplined retry, backoff, and concurrency control. These patterns are transport-level and apply in any language.

Read alpaca-broker-integration first.

1. Rate-limit headers — read them on every response

Alpaca returns standard headers:

Header	Meaning
X-RateLimit-Limit	requests allowed in the window
X-RateLimit-Remaining	requests left in the current window
X-RateLimit-Reset	unix timestamp (seconds) when the window resets

Parse them on every response, not just on errors. Two uses:

• Proactive: when Remaining drops below a threshold (e.g. ≤ 50), log a warning and/or slow down — you're about to get throttled.
• Reactive: on 429, use Reset to wait exactly until the window opens.

Limits vary by endpoint and plan; market-data limits differ from broker limits. Don't hardcode a number — react to the headers.

2. The retry loop (pseudocode)

MAX_ATTEMPTS = 10
INITIAL_DELAY_MS = 1000

for attempt in 1..MAX_ATTEMPTS:
    res = http(request)                      # with a sane timeout (see §5)
    remaining, reset_at = parse_rate_headers(res.headers)
    if remaining <= 50: log_warn("approaching rate limit", reset_at)

    if res.status == 429:
        # wait until the window resets, plus a small buffer
        wait = (reset_at - now()) if reset_at else INITIAL_DELAY_MS * 2^(attempt-1)
        sleep(max(0, wait) + 1000)           # +1s buffer past reset
        continue

    if res.status in (500, 502, 503, 504) or network_error:
        sleep(INITIAL_DELAY_MS * 2^(attempt-1))   # exponential backoff
        continue

    return res                                # success or non-retryable 4xx
raise last_error

Key points:

• On 429, wait until X-RateLimit-Reset + a ~1s buffer — don't blindly exponential-backoff when the API told you exactly when to retry.
• Exponential backoff (base * 2^(attempt-1)) for network errors and 5xx. With base 1s and 10 attempts the tail is minutes — fine for background jobs, too slow for user-facing calls (use fewer attempts there).
• Don't retry non-retryable 4xx (400/403/422) — those won't fix themselves; surface them.
• Optionally add jitter to backoff to avoid thundering-herd when many workers retry together.

3. Bounded concurrency

Parallelism speeds bulk jobs but is the fastest way to hit limits. Use a fixed worker pool, not unbounded fan-out.

• Start small (e.g. 5–8 concurrent requests) and tune against the rate-limit headers. A real lesson from production: a pool was reduced from 10 → 5 to ease both Alpaca and downstream-DB load.
• Cache per-entity reads within a run (e.g. an account's buying power, or a per-account transfer list) so you don't refetch the same thing across items in a batch.
• For per-item throttling, a small fixed sleep between calls (e.g. 100ms) is a crude-but-effective floor when you can't easily coordinate a pool.

4. Pagination loops

List endpoints page forward with a token — never assume one response is complete.

• Activities (/v1/accounts/activities): page via the X-Next-Page-Token response header; loop until it's empty. Use page_size (≤100) and a direction.
• Market-data bars (/v2/stocks/bars): page via next_page_token in the body → pass back as page_token. Remember limit counts across all symbols and results sort by symbol-then-time, so a single page may contain only the first symbol(s) — keep paging.
• Wrap each page fetch in the retry loop from §2.

token = null
loop:
    page = fetch(url + (token ? "&page_token="+token : ""))   # via retry loop
    accumulate(page.items)
    token = page.next_token            # header or body, per endpoint
    if not token: break

5. Timeouts & batch sizing

• Always set an HTTP timeout (e.g. 15–30s). A hung connection without a timeout stalls a whole worker pool. (SSE streams are the exception — they're meant to stay open; see alpaca-broker-sse-events.)
• Use a shared HTTP client / connection pool rather than constructing one per request, so keep-alive and connection reuse work.
• When writing reconciliation results to your own store, chunk bulk inserts (e.g. 500 rows per statement) to stay under DB statement-size limits and keep transactions reasonable.

6. Resilience checklist for a bulk/cron job

• Rate-limit headers parsed every response; proactive warn near the limit.
• 429 → wait until X-RateLimit-Reset + buffer.
• Exponential backoff (+ jitter) for 5xx/network; capped attempts.
• Non-retryable 4xx surfaced, not retried.
• Bounded worker pool; per-run caching of repeated reads.
• Pagination loop until the token is empty.
• HTTP timeout on every call; shared client.
• Bulk DB writes chunked and idempotent (upsert) — see alpaca-broker-reconciliation-idempotency.
• Structured logging with a trace/correlation ID per item for debugging partial failures.

Related skills: safe re-runs of jobs → alpaca-broker-reconciliation-idempotency; the heal/poll jobs that use these patterns → alpaca-broker-reconciliation-idempotency; market-data pagination specifics → alpaca-broker-market-data.