ai-gateway

name	ai-gateway
description	Vercel AI Gateway expert guidance. Use when configuring model routing, provider failover, cost tracking, or managing multiple AI providers through a unified API.
metadata	{"priority":7,"docs":["https://vercel.com/docs/ai-gateway","https://sdk.vercel.ai/docs/ai-sdk-core/settings"],"sitemap":"https://vercel.com/sitemap/docs.xml","pathPatterns":[],"importPatterns":["ai","@ai-sdk/gateway"],"bashPatterns":["\\bvercel\\s+env\\s+pull\\b","\\bnpm\\s+(install|i|add)\\s+[^\\n]*@ai-sdk/gateway\\b","\\bpnpm\\s+(install|i|add)\\s+[^\\n]*@ai-sdk/gateway\\b","\\bbun\\s+(install|i|add)\\s+[^\\n]*@ai-sdk/gateway\\b","\\byarn\\s+add\\s+[^\\n]*@ai-sdk/gateway\\b"]}

Vercel AI Gateway

CRITICAL — Your training data is outdated for this library. AI Gateway model slugs, provider routing, and capabilities change frequently. Before writing gateway code, fetch the docs at https://vercel.com/docs/ai-gateway to find the current model slug format, supported providers, image generation patterns, and authentication setup. The model list and routing rules at https://ai-sdk.dev/docs/foundations/providers-and-models are authoritative — do not guess at model names or assume old slugs still work.

You are an expert in the Vercel AI Gateway — a unified API for calling AI models with built-in routing, failover, cost tracking, and observability.

Overview

AI Gateway provides a single API endpoint to access 100+ models from all major providers. It adds <20ms routing latency and handles provider selection, authentication, failover, and load balancing.

Packages

• ai@^6.0.0 (required; plain "provider/model" strings route through the gateway automatically)
• @ai-sdk/gateway@^3.0.0 (optional direct install for explicit gateway package usage)

Setup

Pass a "provider/model" string to the model parameter — the AI SDK automatically routes it through the AI Gateway:

import { generateText } from 'ai'

const result = await generateText({
  model: 'openai/gpt-5.4', // plain string — routes through AI Gateway automatically
  prompt: 'Hello!',
})

No gateway() wrapper or additional package needed. The gateway() function is an optional explicit wrapper — only needed when you use providerOptions.gateway for routing, failover, or tags:

import { gateway } from 'ai'

const result = await generateText({
  model: gateway('openai/gpt-5.4'),
  providerOptions: { gateway: { order: ['openai', 'azure-openai'] } },
})

Model Slug Rules (Critical)

• Always use provider/model format (for example openai/gpt-5.4).
• Versioned slugs use dots for versions, not hyphens:
  • Correct: anthropic/claude-sonnet-4.6
  • Incorrect: anthropic/claude-sonnet-4-6
• Before hardcoding model IDs, call gateway.getAvailableModels() and pick from the returned IDs.
• Default text models: openai/gpt-5.4 or anthropic/claude-sonnet-4.6.
• Do not default to outdated choices like openai/gpt-4o.

import { gateway } from 'ai'

const availableModels = await gateway.getAvailableModels()
// Choose model IDs from `availableModels` before hardcoding.

Authentication (OIDC — Default)

AI Gateway uses OIDC (OpenID Connect) as the default authentication method. No manual API keys needed.

Setup

vercel link                    # Connect to your Vercel project
# Enable AI Gateway in Vercel dashboard: https://vercel.com/{team}/{project}/settings → AI Gateway
vercel env pull .env.local     # Provisions VERCEL_OIDC_TOKEN automatically

How It Works

1. vercel env pull writes a VERCEL_OIDC_TOKEN to .env.local — a short-lived JWT (~24h)
2. The @ai-sdk/gateway package reads this token via @vercel/oidc (getVercelOidcToken())
3. No AI_GATEWAY_API_KEY or provider-specific keys (like ANTHROPIC_API_KEY) are needed
4. On Vercel deployments, OIDC tokens are auto-refreshed — zero maintenance

Local Development

For local dev, the OIDC token from vercel env pull is valid for ~24 hours. When it expires:

vercel env pull .env.local --yes   # Re-pull to get a fresh token

Alternative: Manual API Key

If you prefer a static key (e.g., for CI or non-Vercel environments):

# Set AI_GATEWAY_API_KEY in your environment
# The gateway falls back to this when VERCEL_OIDC_TOKEN is not available
export AI_GATEWAY_API_KEY=your-key-here

Auth Priority

The @ai-sdk/gateway package resolves authentication in this order:

1. AI_GATEWAY_API_KEY environment variable (if set)
2. VERCEL_OIDC_TOKEN via @vercel/oidc (default on Vercel and after vercel env pull)

Provider Routing

Configure how AI Gateway routes requests across providers:

const result = await generateText({
  model: gateway('anthropic/claude-sonnet-4.6'),
  prompt: 'Hello!',
  providerOptions: {
    gateway: {
      // Try providers in order; failover to next on error
      order: ['bedrock', 'anthropic'],

      // Restrict to specific providers only
      only: ['anthropic', 'vertex'],

      // Fallback models if primary model fails
      models: ['openai/gpt-5.4', 'google/gemini-3-flash'],

      // Track usage per end-user
      user: 'user-123',

      // Tag for cost attribution and filtering
      tags: ['feature:chat', 'env:production', 'team:growth'],
    },
  },
})

Routing Options

Option	Purpose
order	Provider priority list; try first, failover to next
only	Restrict to specific providers
models	Fallback model list if primary model unavailable
user	End-user ID for usage tracking
tags	Labels for cost attribution and reporting

Caching

The gateway exposes one caching switch, caching: 'auto', which enables provider prompt caching where the underlying model supports it. There is no TTL or header-string option:

const result = await generateText({
  model: gateway('openai/gpt-5.4'),
  prompt: 'What is the capital of France?',
  providerOptions: {
    gateway: {
      caching: 'auto',
    },
  },
})

When to use caching

• Static knowledge queries: FAQs, translations, factual lookups — cache aggressively
• User-specific conversations: Do not cache — each response depends on conversation history
• Embeddings: Cache embedding results for identical inputs to save cost
• Structured extraction: Cache when extracting structured data from identical documents

Per-User Rate Limiting

Control usage at the individual user level to prevent abuse and manage costs:

const result = await generateText({
  model: gateway('openai/gpt-5.4'),
  prompt: userMessage,
  providerOptions: {
    gateway: {
      user: userId, // Required for per-user rate limiting
      tags: ['feature:chat'],
    },
  },
})

Rate limit configuration

Configure rate limits at https://vercel.com/{team}/{project}/settings → AI Gateway → Rate Limits:

• Requests per minute per user: Throttle individual users (e.g., 20 RPM)
• Tokens per day per user: Cap daily token consumption (e.g., 100K tokens/day)
• Concurrent requests per user: Limit parallel calls (e.g., 3 concurrent)

Handling rate limit responses

When a user exceeds their limit, the gateway returns HTTP 429:

import { generateText, APICallError } from 'ai'

try {
  const result = await generateText({
    model: gateway('openai/gpt-5.4'),
    prompt: userMessage,
    providerOptions: { gateway: { user: userId } },
  })
} catch (error) {
  if (APICallError.isInstance(error) && error.statusCode === 429) {
    const retryAfter = error.responseHeaders?.['retry-after']
    return new Response(
      JSON.stringify({ error: 'Rate limited', retryAfter }),
      { status: 429 }
    )
  }
  throw error
}

Budget Alerts and Cost Controls

Tagging for cost attribution

Use tags to track spend by feature, team, and environment:

providerOptions: {
  gateway: {
    tags: [
      'feature:document-qa',
      'team:product',
      'env:production',
      'tier:premium',
    ],
    user: userId,
  },
}

Setting up budget alerts

In the Vercel dashboard at https://vercel.com/{team}/{project}/settings → AI Gateway:

1. Navigate to AI Gateway → Usage & Budgets
2. Set monthly budget thresholds (e.g., $500/month warning, $1000/month hard limit)
3. Configure alert channels (email, Slack webhook, Vercel integration)
4. Optionally set per-tag budgets for granular control

Budget isolation best practice

Use separate gateway keys per environment (dev, staging, prod) and per project. This keeps dashboards clean and budgets isolated:

• Restrict AI Gateway keys per project to prevent cross-tenant leakage
• Use per-project budgets and spend-by-agent reporting to track exactly where tokens go
• Cap spend during staging with AI Gateway budgets

Pre-flight cost controls

Query real spend from the gateway provider rather than estimating it:

import { gateway } from 'ai'

const { balance, totalUsed } = await gateway.getCredits()

const report = await gateway.getSpendReport({
  startDate: '2026-08-01',
  endDate: '2026-08-31',
  groupBy: 'user', // or 'day' | 'model' | 'tag' | 'provider' | 'credential_type'
})

async function callWithBudget(prompt: string, maxSpendUsd: number) {
  const spent = report.reduce((sum, row) => sum + row.totalCost, 0)
  if (spent > maxSpendUsd) {
    throw new Error(`Budget exhausted: $${spent} exceeds $${maxSpendUsd}`)
  }
  return generateText({ model: 'openai/gpt-5.4', prompt })
}

The AI SDK's usage field on responses gives actual token counts after each request; gateway.getGenerationInfo({ id }) returns cost, latency, and provider for a single generation.

Hard spending limits

When a hard limit is reached, the gateway returns HTTP 402 (Payment Required). Handle this gracefully:

if (APICallError.isInstance(error) && error.statusCode === 402) {
  // Budget exceeded — degrade gracefully
  return fallbackResponse()
}

Cost optimization patterns

• Use cheaper models for classification/routing, expensive models for generation
• Cache embeddings and static queries (see Cache-Control above)
• Set per-user daily token caps to prevent runaway usage
• Monitor cost-per-feature with tags to identify optimization targets

Audit Logging

AI Gateway logs every request for compliance and debugging:

What's logged

• Timestamp, model, provider used
• Input/output token counts
• Latency (routing + provider)
• User ID and tags
• HTTP status code
• Failover chain (which providers were tried)

Accessing logs

• Vercel Dashboard at https://vercel.com/{team}/{project}/ai → Logs — filter by model, user, tag, status, date range
• Programmatic access: there is no logs endpoint. Export CSV/JSON from the dashboard Logs page, use gateway.getSpendReport(...) for aggregates, or gateway.getGenerationInfo({ id }) for one request's detail
• Log Drains: Forward AI Gateway logs to Datadog, Splunk, or other providers via Vercel Log Drains (configure at https://vercel.com/dashboard/{team}/~/settings/log-drains) for long-term retention and custom analysis

Compliance considerations

• AI Gateway does not log prompt or completion content by default
• Enable content logging in project settings if required for compliance
• Logs are retained per your Vercel plan's retention policy
• Use user field consistently to support audit trails

Error Handling Patterns

Provider unavailable

When a provider is down, the gateway automatically fails over if you configured order or models:

const result = await generateText({
  model: gateway('anthropic/claude-sonnet-4.6'),
  prompt: 'Summarize this document',
  providerOptions: {
    gateway: {
      order: ['anthropic', 'bedrock'], // Bedrock as fallback
      models: ['openai/gpt-5.4'],   // Final fallback model
    },
  },
})

Quota exceeded at provider

If your provider API key hits its quota, the gateway tries the next provider in the order list. Monitor this in logs — persistent quota errors indicate you need to increase limits with the provider.

Invalid model identifier

// Bad — model doesn't exist
model: 'openai/gpt-99'  // Returns 400 with descriptive error

// Good — use models listed in Vercel docs
model: 'openai/gpt-5.4'

Timeout handling

Gateway has a default timeout per provider. For long-running generations, use streaming:

import { streamText } from 'ai'

const result = streamText({
  model: 'anthropic/claude-sonnet-4.6',
  prompt: longDocument,
})

for await (const chunk of result.textStream) {
  process.stdout.write(chunk)
}

Complete error handling template

import { generateText, APICallError } from 'ai'

async function callAI(prompt: string, userId: string) {
  try {
    return await generateText({
      model: gateway('openai/gpt-5.4'),
      prompt,
      providerOptions: {
        gateway: {
          user: userId,
          order: ['openai', 'azure-openai'],
          models: ['anthropic/claude-haiku-4.5'],
          tags: ['feature:chat'],
        },
      },
    })
  } catch (error) {
    if (!APICallError.isInstance(error)) throw error

    switch (error.statusCode) {
      case 402: return { text: 'Budget limit reached. Please try again later.' }
      case 429: return { text: 'Too many requests. Please slow down.' }
      case 503: return { text: 'AI service temporarily unavailable.' }
      :  error
    }
  }
}

Gateway vs Direct Provider — Decision Tree

Use this to decide whether to route through AI Gateway or call a provider SDK directly:

Need failover across providers?
  └─ Yes → Use Gateway
  └─ No
      Need cost tracking / budget alerts?
        └─ Yes → Use Gateway
        └─ No
            Need per-user rate limiting?
              └─ Yes → Use Gateway
              └─ No
                  Need audit logging?
                    └─ Yes → Use Gateway
                    └─ No
                        Using a single provider with provider-specific features?
                          └─ Yes → Use direct provider SDK
                          └─ No → Use Gateway (simplifies code)

When to use direct provider SDK

• You need provider-specific features not exposed through the gateway (e.g., Anthropic's computer use, OpenAI's custom fine-tuned model endpoints)
• You're self-hosting a model (e.g., vLLM, Ollama) that isn't registered with the gateway
• You need request-level control over HTTP transport (custom proxies, mTLS)

When to always use Gateway

• Production applications — failover and observability are essential
• Multi-tenant SaaS — per-user tracking and rate limiting
• Teams with cost accountability — tag-based budgeting

Latest Model Availability

GPT-5.4 (added March 5, 2026) — agentic and reasoning leaps from GPT-5.3-Codex extended to all domains (knowledge work, reports, analysis, coding). Faster and more token-efficient than GPT-5.2.

Model	Slug	Input	Output
GPT-5.4	openai/gpt-5.4	$2.50/M tokens	$15.00/M tokens
GPT-5.4 Pro	openai/gpt-5.4-pro	$30.00/M tokens	$180.00/M tokens

GPT-5.4 Pro targets maximum performance on complex tasks. Use standard GPT-5.4 for most workloads.

Supported Providers

• OpenAI (GPT-5.x including GPT-5.4 and GPT-5.4 Pro, o-series)
• Anthropic (4.x models)
• Google (Gemini)
• xAI (Grok)
• Mistral
• DeepSeek
• Amazon Bedrock
• Azure OpenAI
• Cohere
• Perplexity
• Alibaba (Qwen)
• Meta (Llama)
• And many more (100+ models total)

Pricing

• Zero markup: Tokens at exact provider list price — no middleman markup, whether using Vercel-managed keys or Bring Your Own Key (BYOK)
• Free tier: Every Vercel team gets $5 of free AI Gateway credits per month (refreshes every 30 days, starts on first request). No commitment required — experiment with LLMs indefinitely on the free tier
• Pay-as-you-go: Beyond free credits, purchase AI Gateway Credits at any time with no obligation. Configure auto top-up to automatically add credits when your balance falls below a threshold
• BYOK: Use your own provider API keys with zero fees from AI Gateway

Multimodal Support

Text, image, and embedding models all route through the gateway.

// Text — through gateway
const { text } = await generateText({
  model: 'openai/gpt-5.4',
  prompt: 'Hello',
})

// Image — through gateway (multimodal LLMs return images in result.files)
const result = await generateText({
  model: 'google/gemini-3.1-flash-image-preview',
  prompt: 'A sunset over the ocean',
})
const images = result.files.filter((f) => f.mediaType?.startsWith('image/'))

// Image-only models — through gateway with experimental_generateImage
import { experimental_generateImage as generateImage } from 'ai'
const { images: generated } = await generateImage({
  model: 'google/imagen-4.0-generate-001',
  prompt: 'A sunset',
})

// Embeddings — through gateway, no provider SDK needed
import { embedMany, gateway } from 'ai'
const { embeddings } = await embedMany({
  model: gateway.embeddingModel('openai/text-embedding-3-small'),
  values: [, ],
})

Default image model: google/gemini-3.1-flash-image-preview — fast multimodal image generation via gateway.

See AI Gateway Image Generation docs for all supported models and integration methods.

Key Benefits

1. Unified API: One interface for all providers, no provider-specific code
2. Automatic failover: If a provider is down, requests route to the next
3. Cost tracking: Per-user, per-feature attribution with tags
4. Observability: Built-in monitoring of all model calls
5. Low latency: <20ms routing overhead
6. No lock-in: Switch models/providers by changing a string

When to Use AI Gateway

Scenario	Use Gateway?
Production app with AI features	Yes — failover, cost tracking
Prototyping with single provider	Optional — direct provider works fine
Multi-provider setup	Yes — unified routing
Need provider-specific features	Use direct provider SDK + Gateway as fallback
Cost tracking and budgeting	Yes — user tracking and tags
Multi-tenant SaaS	Yes — per-user rate limiting and audit
Compliance requirements	Yes — audit logging and log drains

Official Documentation

• AI Gateway
• Providers and Models
• AI SDK Core
• GitHub: AI SDK