api-gateway-patterns

# API Gateway Patterns ## When to Use **Use this skill when:** - A user is designing or refactoring the entry point for a microservices architecture and needs to decide which gateway topology to use (single gateway, Backend for Frontend, multi-tier, etc.) - A user asks how to implement cross-cutting concerns -- authentication, rate limiting, request transformation, observability -- without duplicating that logic across dozens of services - A user needs to decide between a self-hosted API gateway (Kong, Envoy, NGINX), a cloud-managed gateway (AWS API Gateway, Azure API Management, Google Cloud Apigee), or a service mesh sidecar pattern - A user is experiencing reliability issues at the edge -- thundering herds, cascading failures, or uneven traffic distribution across backend replicas -- and needs gateway-level mitigation strategies - A user is building a mobile or single-page application that aggregates data from multiple microservices and wants to reduce chatty network round trips via request aggregation or BFF pattern - A user needs to enforce API versioning, deprecation, or traffic shaping (canary releases, blue/green at the edge) for a production API surface - A user is evaluating whether to add a service mesh (Istio, Linkerd) alongside or instead of a traditional API gateway and needs a clear decision framework - A user is migrating a monolith to microservices and needs the strangler fig pattern at the routing layer **Do NOT use this skill when:** - The user needs help designing internal service-to-service communication contracts without an external-facing edge component -- use an internal RPC/gRPC design skill instead - The user is asking about REST API design principles (resource naming, HTTP status codes, HATEOAS) without a gateway component -- use an API design skill - The user needs help with database schema design or query optimization -- this skill does not cover data access patterns - The user is asking about frontend routing, React Router, or SPA navigation -- this is a client-side concern unrelated to gateway infrastructure - The user wants a general microservices decomposition strategy (how to split a domain into services) without a gateway question -- use a microservices decomposition skill - The user is asking about Kubernetes networking primitives (ClusterIP, NodePort, LoadBalancer) without a gateway -- use a Kubernetes networking skill - The user needs CI/CD pipeline design for deploying gateways -- use a deployment pipeline skill, though this skill's output can inform deployment strategy --- ## Process ### Step 1: Establish the Traffic Topology and Entry Point Requirements Before selecting any pattern, characterize the traffic entering the system. - **Identify client types.** Distinguish between browser SPA clients, native mobile apps (iOS/Android), third-party API consumers, internal microservice-to-microservice calls, and IoT devices. Each client type has different payload size expectations, connection persistence requirements (WebSockets vs. HTTP/1.1 vs. HTTP/2), and retry behavior. - **Quantify expected load.** Determine peak requests per second (RPS), expected p99 latency budget (a common production target is p99 < 200ms at the gateway), and payload sizes. A gateway handling 50,000 RPS has fundamentally different configuration requirements than one handling 500 RPS. - **Map the upstream service inventory.** List every backend service the gateway will route to, including their protocols (HTTP/REST, gRPC, WebSocket, GraphQL), expected response times, and failure rates. This mapping determines whether aggregation or simple proxying is the right approach. - **Identify cross-cutting requirements.** Make an explicit list of capabilities that must exist at the edge: authentication (OAuth2/JWT validation, API key check), authorization (RBAC, scope enforcement), rate limiting (per-user, per-plan, per-IP), TLS termination, request/response transformation, logging and tracing, CORS handling, and caching. These requirements heavily influence pattern selection. - **Determine latency budget allocation.** A total API latency budget of, say, 300ms means the gateway layer should consume no more than 10--20ms for routing and policy enforcement, leaving 280ms for upstream services. Lua-based plugins in Kong or WASM filters in Envoy can add 1--5ms per plugin; a chain of 10 plugins at 3ms each consumes 30ms of budget. ### Step 2: Select the Gateway Topology Pattern Apply the decision framework below to choose between the four primary topologies. - **Single Gateway (Monolithic Gateway):** Use when all clients are of the same type, the service count is below ~20, the team is smaller than 10 engineers, and operational simplicity is the priority. A single NGINX Plus or Kong gateway in front of all services is the correct starting point. Drawback: the gateway becomes a bottleneck for team velocity -- every team's routing change goes through the same configuration. - **Backend for Frontend (BFF):** Use when client types diverge significantly in data shape requirements. Deploy one gateway instance per client surface: one BFF for the mobile app (optimized for bandwidth -- smaller payloads, aggressive caching), one for the web SPA (richer payloads, longer cache TTLs), one for partner API consumers (versioned, schema-validated). Each BFF is owned by the team that owns the client. Netflix, SoundCloud, and Spotify pioneered this pattern at scale. - **Multi-Tier Gateway (Edge + Internal):** Use when you have a public-facing edge concern (DDoS protection, TLS termination, global CDN routing) separate from internal API management concerns (service discovery, circuit breaking, auth). The edge tier is a global load balancer or CDN with basic routing rules. The internal tier is a full-featured API gateway handling policy enforcement. AWS CloudFront + AWS API Gateway is the canonical managed implementation. - **Service Mesh with Gateway Ingress:** Use when you need east-west (service-to-service) traffic management in addition to north-south (client-to-gateway) management. The gateway handles external traffic; the service mesh (Istio Ingress Gateway or Linkerd Gateway) handles internal routing, mTLS, and retries. Adds significant operational complexity -- only appropriate for organizations with platform engineering teams of 3+ dedicated engineers. ### Step 3: Design the Authentication and Authorization Pattern Authentication at the gateway is one of the most consequential architectural decisions. - **JWT validation at the gateway:** The gateway validates the JWT signature (RS256 or ES256 -- avoid HS256 for distributed systems because it requires sharing the secret), checks expiration, and forwards the decoded claims as HTTP headers (e.g., `X-User-Id`, `X-User-Roles`) to upstream services. Upstream services trust these headers, eliminating the need for each service to perform its own token validation. This is the dominant pattern for stateless microservices. Configure the gateway to reject tokens with `exp` within 0 seconds (no clock skew tolerance beyond 30--60 seconds). - **OAuth2 token introspection:** For opaque tokens (not JWTs), the gateway calls an introspection endpoint on the Authorization Server for each request. This is expensive -- introspection calls add 5--50ms per request. Mitigate with short-TTL caching of introspection results (30--120 seconds TTL, keyed on token hash). Only use introspection when token revocation must be near-real-time. - **API key management:** For third-party developer access, issue API keys stored in the gateway's key store (Kong's key-auth plugin, AWS API Gateway usage plans). Associate keys with rate limit tiers: free tier (100 req/day), standard tier (10,000 req/day), premium tier (unlimited with fair-use SLA). Never pass raw API keys to upstream services. - **mTLS for machine-to-machine:** Service-to-service calls that pass through the gateway should use mutual TLS with client certificates managed by a certificate authority (SPIFFE/SPIRE is the standard for short-lived, automatically rotated certificates). Configure the gateway to require and verify client certificates for designated machine client routes. ### Step 4: Design Rate Limiting and Traffic Shaping Rate limiting without proper design causes legitimate users to be throttled while adversaries adapt. - **Choose the rate limiting algorithm.** Token bucket allows short bursts above the sustained rate (appropriate for interactive APIs). Fixed window is simple but creates thundering herds at window boundaries (avoid for high-traffic APIs). Sliding window log is accurate but memory-intensive. Sliding window counter is a practical compromise -- Kong's rate-limiting-advanced plugin implements this. Use token bucket as the default. - **Define rate limit dimensions.** Apply limits in layers: global rate limit (protects all upstream services from gateway overload, e.g., 100,000 RPS total), per-route limit (protects specific services, e.g., `/search` endpoint at 5,000 RPS), per-consumer limit (prevents API key abuse, e.g., 1,000 req/min per key), per-IP limit (prevents anonymous abuse, e.g., 100 req/min per IP). - **Use a distributed counter store.** Single-instance rate limiting does not work when the gateway is horizontally scaled. Use Redis with the `INCR` + `EXPIRE` pattern or Redis Lua scripts for atomic counter operations. Set Redis connection pool size to at least 20 connections per gateway instance. Configure a fallback policy (allow or deny) when Redis is unavailable -- defaulting to allow is safer for user experience but riskier for abuse; choose based on abuse profile. - **Return standard rate limit headers.** Always include `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` (Unix timestamp), and `Retry-After` on 429 responses. Clients that respect these headers will back off gracefully instead of hammering the gateway. - **Circuit breaking at the gateway.** Configure a circuit breaker per upstream service: open after 5 consecutive 5xx responses or when error rate exceeds 50% over a 10-second window, half-open after 30 seconds, close after 2 consecutive successes. Return `503 Service Unavailable` with a `Retry-After: 30` header when the circuit is open. ### Step 5: Design Request/Response Transformation and Aggregation Transformation logic belongs at the gateway only when it is client-specific shaping, not business logic. - **Header manipulation:** Strip internal headers before forwarding upstream (e.g., `X-Internal-Debug`, `X-Forwarded-For` normalization). Inject correlation IDs (`X-Request-Id` or `X-Correlation-Id`) using UUID v4 at the gateway if not already present. Propagate trace context (`traceparent` per W3C Trace Context spec, or `X-B3-TraceId` for Zipkin-compatible systems). - **Protocol translation:** Use the gateway to translate between REST/HTTP and gRPC (Envoy's gRPC-JSON transcoding or Kong's grpc-gateway plugin). Translate between WebSocket and HTTP long-polling for legacy clients. Never put business logic in protocol translation -- it should be purely structural transformation. - **Request aggregation (fan-out):** When a single client request requires data from 3+ microservices, implement aggregation at the BFF layer using parallel upstream calls with a timeout. Set a hard deadline of total_timeout = max_upstream_latency + 50ms buffer. Use the GraphQL federation pattern (Apollo Router as the gateway) when aggregation relationships are complex and evolve frequently. For simpler cases, a custom BFF with parallel HTTP calls suffices. - **Response caching:** Cache GET responses at the gateway for endpoints where freshness tolerance exceeds 1 second. Use Varnish or built-in Kong proxy-cache plugin. Set cache keys based on URI + relevant query params + `Authorization` header hash (never cache per raw auth token). Configure `Cache-Control: max-age` upstream to drive gateway TTL, or override with gateway-level TTL policies. Never cache responses to requests with `Authorization` headers at a shared cache layer without per-user cache isolation. - **Content negotiation:** Handle `Accept: application/json` vs `Accept: application/xml` at the gateway for legacy clients by transforming JSON responses from modern upstream services. Use XSLT or jq-based transformations. Document the overhead -- JSON-to-XML transformation adds 2--15ms depending on payload size. ### Step 6: Implement Observability at the Gateway Layer The gateway is the single best place to collect API-level telemetry because all traffic passes through it. - **Structured access logging.** Log every request with: timestamp (ISO8601), method, path (stripped of PII query params), status code, upstream service name, upstream latency (ms), total gateway latency (ms), client IP (hashed if GDPR-applicable), user ID (from decoded JWT claims), request size (bytes), response size (bytes), and request ID. Output as structured JSON to stdout for log aggregation (Fluentd, Logstash). Log at INFO level for 2xx/3xx, WARN for 4xx, ERROR for 5xx. - **Metrics emission.** Emit the following Prometheus metrics from the gateway: `gateway_requests_total` (counter, labels: method, route, status_class, upstream_service), `gateway_request_duration_seconds` (histogram, buckets: 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s), `gateway_upstream_errors_total` (counter, labels: upstream_service, error_type), `gateway_rate_limit_hits_total` (counter, labels: consumer, route). Set scrape interval to 15 seconds. - **Distributed tracing.** Inject or propagate trace context on every request. Configure sampling at 1--5% for high-traffic APIs (10,000+ RPS), 100% for low-traffic or debugging scenarios. Send spans to a collector (Jaeger, Zipkin, or OpenTelemetry Collector). Tag gateway spans with: upstream service name, route ID, consumer ID, plugin execution times. - **Alerting thresholds.** Alert on: p99 latency > 500ms sustained for 5 minutes, error rate (5xx) > 1% over 5 minutes, circuit breaker open for any upstream, rate limit hit rate > 10% of total requests, Redis connection pool saturation > 80%. ### Step 7: Design for High Availability and Deployment A gateway with a single point of failure is an unacceptable production architecture. - **Horizontal scaling.** Run a minimum of 3 gateway instances across 3 availability zones (for cloud deployments). Use a load balancer (AWS ALB, GCP Load Balancing) in front of the gateway tier with health checks every 10 seconds. Gateway instances must be stateless -- all shared state (rate limit counters, session data) lives in external Redis or etcd. - **Configuration management.** Store gateway configuration in version control (GitOps). Use Kong's declarative configuration (deck tool) or Envoy's xDS API driven by a control plane (Istio Pilot). Never make manual configuration changes to production gateways -- all changes go through pull request + CI validation + automated deployment. Configuration drift is a leading cause of gateway outages. - **Canary and blue/green at the gateway.** Implement traffic splitting at the gateway routing layer: route 5% of traffic to the new service version, 95% to the stable version. Use weighted round-robin upstream load balancing. Monitor error rates for both versions for 15--30 minutes before promoting the canary. This eliminates the need for DNS-based blue/green switching for most scenarios. - **Graceful degradation.** Define fallback responses for each critical upstream: return a cached stale response (if available and acceptable), return a predetermined static response, or return a 503 with a descriptive error body (including `X-Error-Service` header identifying which upstream failed). Never let an upstream failure propagate as a generic 500 from the gateway. - **TLS configuration.** Terminate TLS at the gateway. Use TLS 1.2 minimum (TLS 1.3 preferred). Configure cipher suites to exclude RC4, DES, 3DES, and export ciphers. Use HSTS with `max-age=31536000; includeSubDomains; preload`. Rotate certificates automatically with cert-manager (Kubernetes) or AWS Certificate Manager. Set certificate expiry alerts at 30 days and 7 days. ### Step 8: Validate, Test, and Document the Gateway Configuration Gateway configuration bugs cause production outages that are difficult to diagnose without proper testing. - **Contract testing.** Write consumer-driven contract tests (using Pact or similar) that validate the gateway's routing rules produce the expected upstream calls. Run these in CI on every configuration change. - **Load testing.** Before any significant traffic increase, load test the gateway with k6 or Locust at 2x expected peak RPS. Validate that p99 latency stays within budget, rate limiting fires at the correct thresholds, and circuit breakers open and recover correctly. - **Chaos engineering.** Inject upstream failures (using Toxiproxy or Chaos Monkey) and verify: circuit breakers open at the correct thresholds, fallback responses are returned correctly, alerts fire within expected time windows, and the gateway continues serving other routes unaffected. - **Configuration validation.** Use `deck validate` for Kong, `envoy --mode validate` for Envoy, or `kic --dry-run` for Kubernetes Ingress Controller configurations before deploying. Prevent syntactically invalid configurations from ever reaching production. - **Document with OpenAPI + gateway overlay.** Maintain an OpenAPI 3.x spec for the public API surface exposed by the gateway. Add gateway-specific extensions (`x-rate-limit`, `x-auth-required`, `x-cache-ttl`) as vendor extensions to document gateway behavior alongside API contracts. Publish this documentation to a developer portal (Backstage, Redoc). --- ## Output Format When providing API gateway pattern guidance, structure the response as follows: ``` ## API Gateway Pattern Analysis ### Traffic Profile | Dimension | Value | Notes | |------------------------|-------------------------------|----------------------------------------| | Client Types | [list] | Drives BFF vs. single gateway decision | | Peak RPS | [number] | Sizing and rate limit baseline | | p99 Latency Budget | [ms] | Gateway overhead target: <10% of total | | Protocol(s) | [REST/gRPC/WebSocket/GraphQL] | Drives protocol translation needs | | Upstream Service Count | [number] | Affects routing complexity | ### Selected Pattern **Pattern:** [Single Gateway | BFF | Multi-Tier | Service Mesh Ingress] **Rationale:** [2-4 sentences explaining why this pattern fits the specific profile above] ### Cross-Cutting Concern Configuration | Concern | Mechanism | Configuration | |-----------------|----------------------------------------|----------------------------------------------------| | Authentication | [JWT Validation / API Key / mTLS] | [Algorithm, cache TTL, header forwarding spec] | | Rate Limiting | [Token Bucket / Sliding Window] | [Per-consumer limit, per-route limit, Redis config]| | Circuit Breaker | [Error threshold / timeout config] | [Open threshold, half-open delay, fallback] | | Caching | [Cache-Control / gateway-level TTL] | [TTL value, cache key spec, invalidation strategy] | | Observability | [Prometheus metrics / trace context] | [Sampling rate, log fields, alert thresholds] | | TLS | [Termination config] | [Min version, cipher policy, cert rotation] | ### Routing Rules | Route Pattern | Upstream Service | Auth Required | Rate Limit | Circuit Breaker | |-----------------------|---------------------|---------------|-------------------|-----------------| | GET /api/v1/[resource]| [service-name]:[port]| JWT (scope) | 1000 req/min/user | 5xx > 50%/10s | | POST /api/v1/[action] | [service-name]:[port]| JWT + RBAC | 100 req/min/user | 5xx > 20%/10s | ### Implementation Artifacts #### Gateway Configuration Snippet ```yaml # [Kong / Envoy / NGINX config for the primary route] [concrete configuration block] ``` #### Redis Rate Limit Configuration ``` [Connection pool settings, key schema, fallback policy] ``` #### Observability Setup ``` [Prometheus scrape config, alert rule examples] ``` ### Trade-off Summary | Decision | Chosen Approach | Alternative Considered | Reason for Choice | |--------------------|-----------------------|------------------------|---------------------------------| | [decision point] | [chosen] | [alternative] | [specific technical rationale] | ### Operational Runbook (Brief) - **Scale up:** [specific trigger metric and action] - **Circuit breaker open:** [diagnosis steps and recovery procedure]