openrouter-load-balancing

name	openrouter-load-balancing
description	Distribute OpenRouter requests across multiple keys and models for high throughput. Use when scaling beyond single-key rate limits or building high-availability systems. Triggers: 'openrouter load balance', 'openrouter scaling', 'distribute openrouter requests', 'multiple api keys'.
allowed-tools	Read, Write, Edit, Grep, Bash(python3:*)
version	1.20.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","openrouter","scaling","high-availability","load-balancing"]
compatibility	Designed for Claude Code

OpenRouter Load Balancing

Overview

A single OpenRouter API key has rate limits (requests/minute and tokens/minute). To scale beyond those limits, distribute requests across multiple keys. OpenRouter also provides server-side load balancing via provider routing and the :nitro variant for low-latency inference. This skill covers multi-key rotation, health-based routing, circuit breakers, and concurrent request patterns.

Prerequisites

• Two or more OpenRouter API keys exported as OPENROUTER_KEY_1, OPENROUTER_KEY_2, OPENROUTER_KEY_3 so the KeyPool has keys to rotate — see the openrouter-install-auth skill for creating and exporting keys
• OPENROUTER_API_KEY exported for the single-key concurrent-processing pattern
• Python 3.8+ with the OpenAI SDK and requests (pip install openai requests) — the concurrent example uses AsyncOpenAI from the same package
• Adequate credits on every key in the pool; per-key quota is visible via GET /api/v1/auth/key

Instructions

1. Export your pool keys and build the KeyPool from Multi-Key Round Robin — it round-robins across keys, trips a circuit breaker after 3 consecutive errors, and auto-recovers a key after a 60s cooldown.
2. Send traffic through balanced_completion(): on RateLimitError it calls pool.mark_error(key) and retries with the next healthy key.
3. For batch workloads, use parallel_completions() from Concurrent Request Processing — an asyncio.Semaphore (max_concurrent=3-5) caps in-flight requests against a single key.
4. Layer on server-side distribution per Provider-Level Load Balancing: pass extra_body={"provider": {"order": [...], "allow_fallbacks": True}} so OpenRouter spreads the same model across Anthropic, AWS Bedrock, and GCP Vertex.
5. Monitor quota per key with check_rate_limits() (GET /api/v1/auth/key) from Rate Limit Awareness, and when 429s hit all keys simultaneously, apply the fixes in Error Handling (more keys, request queuing).

Multi-Key Round Robin

import os, itertools, time, logging
from openai import OpenAI, RateLimitError
 dataclasses  dataclass, field

log = logging.getLogger()


 :
    
    keys: []
    _cycle: itertools.cycle = field(init=, =)
    _health: [, ] = field(init=, default_factory=)

     ():
        ._cycle = itertools.cycle(.keys)
        ._health = {k: {: , : , : }  k  .keys}

     () -> :
        
        attempts = 
         attempts < (.keys):
            key = (._cycle)
            h = ._health[key]
            
              h[]  time.time() - h[] > :
                h[] = 
                h[] = 
             h[]:
                 key
            attempts += 
        
         (._cycle)

     ():
        h = ._health[key]
        h[] += 
        h[] = time.time()
         h[] >= :  
            h[] = 
            log.warning()

     ():
        ._health[key][] = 
        ._health[key][] = 

pool = KeyPool(keys=[
    os.environ.get(, ),
    os.environ.get(, ),
    os.environ.get(, ),
])

 ():
    
    key = pool.next_key()
    client = OpenAI(
        base_url=,
        api_key=key,
        default_headers={: , : },
    )
    :
        response = client.chat.completions.create(
            model=model, messages=messages, **kwargs
        )
        pool.mark_success(key)
         response
     RateLimitError:
        pool.mark_error(key)
        
         balanced_completion(messages, model, **kwargs)