| name | groq-reference-architecture |
| description | Implement Groq reference architecture with model routing, streaming pipelines, and fallbacks.
Use when designing new Groq integrations, reviewing project structure,
or establishing architecture standards for Groq applications.
Trigger with phrases like "groq architecture", "groq best practices",
"groq project structure", "how to organize groq", "groq design".
|
| allowed-tools | Read, Grep |
| version | 1.11.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
| tags | ["saas","groq","groq-reference"] |
| compatibility | Designed for Claude Code, also compatible with Codex and OpenClaw |
Groq Reference Architecture
Overview
Production architecture for applications built on Groq's LPU inference API. It
covers four concerns that every serious Groq integration needs: routing requests
to the right model by latency/capability/cost, a middleware band (cache, metrics,
retry), a multi-provider fallback chain, and a streaming pipeline. The service
layer built here is reusable across a chat UI, an API backend, a batch processor,
or an agent.
The full layer diagram and how the pieces interact lives in
references/architecture.md; the complete,
copy-ready TypeScript for every layer is in
references/implementation.md.
Prerequisites
- Groq API key — create one at console.groq.com
and export it as
GROQ_API_KEY. The Groq SDK reads it from the environment;
the client is constructed as new Groq({ apiKey: process.env.GROQ_API_KEY }).
Never hardcode the key.
- Runtime: Node.js 18+ (for
performance.now() and native fetch).
- Packages:
groq-sdk and lru-cache (npm install groq-sdk lru-cache).
- Optional backup provider: an OpenAI-compatible key if you extend the
fallback chain beyond Groq's own models.
Instructions
Build the service layer in five ordered steps. Each step is one file under
src/groq/. The router depends on the registry; the middleware and fallback
depend on the client; the streaming pipeline stands alone. Full source for every
step (verbatim) is in references/implementation.md.
-
Model Registry (models.ts) — declare a ModelSpec for each model with
its tier, context window, speed, cost, and capabilities. Skeleton:
export const MODELS: Record<string, ModelSpec> = {
"llama-3.1-8b-instant": { tier: "speed", },
"llama-3.3-70b-versatile": { : , },
: { : },
: { : },
};
