rpc-selection-and-resilience

name	rpc-selection-and-resilience
description	Playbook for selecting Solana RPC providers and building resilient client access (fallbacks, timeouts, rate limits, caching, cost control). Use when designing infra or debugging RPC issues.

RPC Selection and Resilience

Role framing: You are a Solana infra engineer. Your goal is to choose the right RPC mix and make clients robust to rate limits, outages, and latency spikes.

Initial Assessment

• Traffic profile: reads vs writes, peak TPS, burstiness, geographic distribution.
• Critical paths: which features are user-facing vs background? Latency SLOs?
• Budget: monthly cap? willingness to pay for priority lanes?
• Data needs: logs/blocks historical depth, filters, WebSocket support, state compression?
• Clients: browser, server, bots? Using connection pooling? Using Jito? Alchemy/Helius/QuickNode/own node?
• Observability stack: how to measure RPC errors/latency; alert thresholds.

Core Principles

• Separate read/write: use high-quality paid endpoints for writes; cache-friendly endpoints for reads.
• Multi-provider strategy: primary + failover with health checks; avoid single vendor lock.
• Backpressure and rate limits: exponential backoff, jitter, and circuit breakers > blind retries.
• Timeouts tuned: short for UX-critical reads, longer for archival queries; prefer abortable fetch.
• Deterministic commitment: specify processed/confirmed/inalized per use case; avoid defaults.
• Security: pin RPC URLs; avoid leaking API keys to clients; prefer server-proxied writes.

Workflow

1. Provider comparison
   • Evaluate: reliability SLA, included features (webhooks, state compression, enhanced APIs), geo, price per million requests, burst limits.
   • Select primary + two fallbacks; note auth schemes.
2. Endpoint configuration
   • Define per-use-case endpoints: READ_PRIMARY, READ_CACHE, WRITE_TX, WEBSOCKET.
   • Store in env with rotation ability; never hardcode keys in client bundles.
3. Client resilience
   • Implement health checks (ping, slot distance, error rate); auto-failover when thresholds breached.
   • Use request hedging for latency-sensitive reads (send to two, pick fastest) with caps.
   • For writes: add priority fees and preflight; on BlockhashNotFound or NodeBehind, refresh blockhash and retry to another endpoint.
4. Performance controls
   • Batch RPC calls (getMultipleAccounts, getProgramAccounts with filters) instead of per-account fetches.
   • Cache layer (CDN/kv) for idempotent reads; invalidate on slot interval.
   • Use ALTs for tx with many accounts to reduce size + retries.
5. Cost management
   • Track request volume per method; throttle noisy endpoints; move polling to webhooks/WS where cheaper.
   • Compress/trim logs; prefer enhanced APIs when they reduce query count.
6. Monitoring & alerting
   • Metrics: p50/p95 latency, error codes (429, -32005), slot lag, WebSocket drop rate.
   • Alerts with runbooks: switch to fallback, raise priority fee, reduce polling.

Templates / Playbooks

• Health check thresholds: error rate >3% or slot lag >30 slots for 2 mins -> failover.
• Retry policy example: max 3 attempts; backoff 200ms * 2^n with jitter; switch provider after first rate-limit.
• Env layout:
  • RPC_READ_PRIMARY=https://...
  • RPC_READ_FALLBACK=https://...
  • RPC_WRITE=https://...
  • RPC_WS=wss://...
• Cost quick estimate: requests/day * price per 1M / 1,000,000.

Common Failure Modes + Debugging

• 429 / rate limit: reduce concurrency, add caching, switch endpoint tier.
• Stale blockhash causing tx drop: refresh via getLatestBlockhash with commitment; add priority fee.
• Slot lag on provider: monitor; switch read path; avoid mixing commitments across providers if lagging.
• WebSocket disconnect loops: implement heartbeat + auto-resubscribe with backoff.
• API key leaked in frontend: proxy writes through backend; rotate keys; monitor referrers.

Quality Bar / Validation

• Config includes at least primary + one fallback with health logic.
• Timeouts and retries defined per operation type; no infinite retries.
• Metrics/alerts wired with documented thresholds.
• Keys not exposed in client bundles; env documented.
• Load test or simulation shows graceful degradation under throttling.

Output Format

Return:

• Provider comparison table + chosen stack
• Endpoint map (read/write/ws) with commitments and timeouts
• Retry/failover policy description
• Monitoring plan with metrics + alert thresholds
• Cost estimate and knobs to stay within budget

Examples

• Simple: Frontend-only meme site
  • Read-only endpoint via public RPC; cached responses; writes proxied to single paid endpoint; fallback public RPC for reads; minimal telemetry.
• Complex: High-volume trading bot + dApp
  • Primary paid provider with priority fees; secondary provider for hedged reads; ALTs for large tx; WebSocket stream with heartbeats; caching layer reduces getAccountInfo spam; alerts on slot lag and 429s; monthly budget forecast with throttle switches.