Groq Reference Architecture
Overview
Production architecture for applications built on Groq's LPU inference API. It
covers four concerns that every serious Groq integration needs: routing requests
to the right model by latency/capability/cost, a middleware band (cache, metrics,
retry), a multi-provider fallback chain, and a streaming pipeline. The service
layer built here is reusable across a chat UI, an API backend, a batch processor,
or an agent.
The full layer diagram and how the pieces interact lives in
references/architecture.md; the complete,
copy-ready TypeScript for every layer is in
references/implementation.md.
Prerequisites
- Groq API key — create one at console.groq.com
and export it as
GROQ_API_KEY. The Groq SDK reads it from the environment;
the client is constructed as new Groq({ apiKey: process.env.GROQ_API_KEY }).
Never hardcode the key.
- Runtime: Node.js 18+ (for
performance.now() and native fetch).
- Packages:
groq-sdk and lru-cache (npm install groq-sdk lru-cache).
- Optional backup provider: an OpenAI-compatible key if you extend the
fallback chain beyond Groq's own models.
Instructions
Build the service layer in five ordered steps. Each step is one file under
src/groq/. The router depends on the registry; the middleware and fallback
depend on the client; the streaming pipeline stands alone. Full source for every
step (verbatim) is in references/implementation.md.
-
Model Registry (models.ts) — declare a ModelSpec for each model with
its tier, context window, speed, cost, and capabilities. Skeleton:
export const MODELS: Record<string, ModelSpec> = {
"llama-3.1-8b-instant": { tier: "speed", },
"llama-3.3-70b-versatile": { tier: "quality", },
"meta-llama/llama-4-scout-17b-16e-instruct": { tier: "vision" },
"whisper-large-v3-turbo": { tier: "audio" },
};
-
Model Router (router.ts) — selectModel(req) maps requirements
(maxLatencyMs, needsVision, needsTools, costSensitive) to the cheapest
model that satisfies them. Callers pass requirements, never hardcoded ids.
-
Middleware (middleware.ts) — completionWithMiddleware() wraps each call
with an LRU cache (deterministic requests only, temperature === 0), latency +
token metrics, and a pluggable metrics sink.
-
Fallback Chain (fallback.ts) — completionWithFallback() tries the
primary model, drops to a model in a different rate-limit pool on 429/5xx, then
returns a graceful-degradation payload instead of throwing.
-
Streaming Pipeline (streaming.ts) — streamCompletion() is an async
generator yielding { type: "token" | "done" | "error" } for real-time SSE UIs.
When applying this to an existing repo, Read the current src/ layout and
Grep for direct groq.chat.completions.create calls to find code that should
route through the middleware and fallback wrappers instead.
Integration Patterns
| Pattern | When to Use | Groq Feature |
|---|
| Direct completion | Simple request/response | chat.completions.create |
| Streaming SSE | Real-time chat UI | stream: true |
| Tool calling | Agent with function execution | tools parameter |
| JSON extraction | Structured data from text | response_format: json_object |
| Batch processing | High-volume document processing | Queue + rate limiting |
| Audio transcription | Voice input | audio.transcriptions.create |
| Vision analysis | Image understanding | Llama 4 Scout/Maverick |
Output
Applying this skill produces a src/groq/ service layer with six files
(client.ts, models.ts, router.ts, middleware.ts, fallback.ts,
streaming.ts) plus the service and API layers that consume it. At runtime you get:
- Routed completions —
selectModel() returns a ModelSpec; callers never
hardcode a model id, so cost/latency policy lives in one place.
- Cached deterministic responses — repeated
temperature: 0 calls return from
the LRU cache instead of re-billing the API.
- Resilient calls —
completionWithFallback() returns a valid completion shape
even when Groq is rate-limited, never surfacing a raw 429 to the user.
- Streamed tokens —
streamCompletion() yields { type, content } events for
SSE, with a terminal done or error event.
- Metrics — every call emits
{ model, latencyMs, tokens, cached } to your
metrics sink (Prometheus, Datadog, or console.log by default).
Error Handling
| Issue | Cause | Solution |
|---|
| 429 on primary model | RPM/TPM exceeded | Fall back to different model |
| High latency | Wrong model tier | Route to 8b-instant for latency-critical paths |
| Context overflow | Input > 128K tokens | Truncate or chunk input |
| Vision errors | Wrong model for images | Use Llama 4 Scout full model path |
GROQ_API_KEY undefined | Env var not exported | Export the key before starting the process |
Examples
A latency-critical chat turn routes to the speed tier and returns one completion:
const model = selectModel({ maxLatencyMs: 80, costSensitive: true });
const res = await completionWithMiddleware(groq, model.id, messages);
Streaming a UI consumes the async generator token-by-token:
for await (const event of streamCompletion(groq, messages)) {
if (event.type === "token") process.stdout.write(event.content!);
}
Four fully worked examples — latency-critical, quality-with-fallback, streaming,
and vision routing — are in references/examples.md.
Resources
Next Steps
For multi-environment deployment, see the groq-multi-env-setup skill, which
extends this service layer with per-environment configuration and secrets handling.
