ai-gateway

name

ai-gateway

description

Route LLM API requests through a unified proxy with rate limiting, caching, and fallback. Use when you need to: call multiple LLM providers from one endpoint, add caching to reduce API costs, implement fallback between providers, enforce rate limits per team or application, or log all LLM usage centrally. Triggers include "LLM proxy", "unified API", "provider fallback", "LLM rate limiting", "AI gateway", "route to OpenAI", "cache LLM responses", or any task calling multiple LLM providers.

ai-gateway

Unified LLM proxy with rate limiting, caching, logging, and automatic fallback. Drop-in replacement for direct provider calls.

When to use

• Calling multiple LLM providers from one endpoint
• Adding response caching to reduce API costs
• Implementing automatic fallback when a provider fails
• Enforcing per-team or per-application rate limits
• Centralizing LLM usage logs and cost tracking
• A/B testing different models or providers
• Any application calling OpenAI, Anthropic, or Cohere APIs

Prerequisites

1. A running ai-gateway instance (local or Docker)
2. A gateway API key (gw_ prefix)
3. At least one provider configured in the gateway admin

Quick Start

# Start the gateway
gateway start

# Create a gateway API key
gateway key create --label myapp

# Point your OpenAI client at the gateway
# (change the base URL, keep your application logic unchanged)

Usage in Code

Node.js with OpenAI SDK

import OpenAI from 'openai'

const client = new OpenAI({
  apiKey: 'gw_your_gateway_key',
  baseURL: 'http://localhost:4080/v1',
})

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello' }],
})

Python with OpenAI SDK

import openai

client = openai.OpenAI(
    api_key="gw_your_gateway_key",
    base_url="http://localhost:4080/v1",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Direct HTTP

curl http://localhost:4080/v1/chat/completions \
  -H "Authorization: Bearer gw_your_gateway_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Model Routing

The gateway routes requests to providers based on the model name:

Model pattern	Default provider
gpt-4*	OpenAI
claude-*	Anthropic
command-*	Cohere
unknown model	Highest-priority provider that lists it

Custom routing rules can override defaults. Set them in the admin dashboard or API.

Caching

Responses are cached automatically when temperature is 0 or not set. The cache key is derived from the model, messages, and parameters. On a cache hit, the stored response is returned without calling the provider.

Cache TTL defaults to 1 hour. Configure in settings:

GATEWAY_CACHE_TTL=3600   # seconds
GATEWAY_CACHE_ENABLED=1  # 0 to disable

Rate Limiting

Each gateway key has a requests-per-minute limit. The default is 60 rpm (global setting). Override per key:

gateway key create --label myapp --rate-limit 100

On limit exceeded, the gateway returns:

HTTP 429 Too Many Requests
Retry-After: 15

Fallback Chains

When a provider fails (timeout, 5xx, or provider-side 429), the gateway automatically tries the next provider in the configured fallback chain. The response includes standard OpenAI format regardless of which provider served it.

Configure fallback in the admin dashboard under Routes.

CLI Reference

Command	Description
gateway start	Start the gateway
gateway key create --label <name>	Create a gateway API key
gateway key create --label <name> --rate-limit <rpm>	Create key with custom rate limit
gateway key list	List all gateway keys
gateway key revoke <id>	Revoke a key
gateway provider add	Add a provider interactively
gateway provider list	List providers
gateway provider health	Check all provider health
gateway --help	Show help
gateway --version	Show version

Environment Variables

Variable	Description	Default
GATEWAY_PORT	Proxy port	4080
GATEWAY_ADMIN_PORT	Admin API + dashboard port	4081
GATEWAY_ENCRYPTION_KEY	32-byte hex key for provider key encryption	required
GATEWAY_DATA_DIR	Data directory for SQLite and logs	~/.ai-gateway
GATEWAY_REDIS_URL	Redis URL for distributed rate limiting	(none)
GATEWAY_CACHE_ENABLED	Enable response cache (1 or 0)	1
GATEWAY_CACHE_TTL	Cache TTL in seconds	3600
GATEWAY_RATE_LIMIT_RPM	Default rate limit per key	60
GATEWAY_LOG_RETENTION	Request log retention in days	30
GATEWAY_DEV	Dev mode: verbose logging, no auth (1 or 0)	0

Troubleshooting

"invalid gateway key"

Key is wrong, revoked, or missing gw_ prefix. Check with gateway key list.

Request goes to wrong provider

Check routing rules in the admin dashboard. Model name must match the glob pattern. Default routing uses provider priority order.

Cache never hits

Ensure temperature is 0 or absent. Cache only stores deterministic responses. Check GATEWAY_CACHE_ENABLED=1.

429 from gateway vs 429 from provider

Gateway 429 means your key hit the gateway rate limit. Provider 429 (wrapped in a 502) means the provider rejected the request. Check the error response body for the source.