groq-reference-architecture

name	groq-reference-architecture
description	Implement Groq reference architecture with model routing, streaming pipelines, and fallbacks. Use when designing new Groq integrations, reviewing project structure, or establishing architecture standards for Groq applications. Trigger with phrases like "groq architecture", "groq best practices", "groq project structure", "how to organize groq", "groq design".
allowed-tools	Read, Grep
version	1.11.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","groq","groq-reference"]
compatibility	Designed for Claude Code, also compatible with Codex and OpenClaw

Groq Reference Architecture

Overview

Production architecture for applications built on Groq's LPU inference API. It covers four concerns that every serious Groq integration needs: routing requests to the right model by latency/capability/cost, a middleware band (cache, metrics, retry), a multi-provider fallback chain, and a streaming pipeline. The service layer built here is reusable across a chat UI, an API backend, a batch processor, or an agent.

The full layer diagram and how the pieces interact lives in references/architecture.md; the complete, copy-ready TypeScript for every layer is in references/implementation.md.

Prerequisites

• Groq API key — create one at console.groq.com and export it as GROQ_API_KEY. The Groq SDK reads it from the environment; the client is constructed as new Groq({ apiKey: process.env.GROQ_API_KEY }). Never hardcode the key.
• Runtime: Node.js 18+ (for performance.now() and native fetch).
• Packages: groq-sdk and lru-cache (npm install groq-sdk lru-cache).
• Optional backup provider: an OpenAI-compatible key if you extend the fallback chain beyond Groq's own models.

Instructions

Build the service layer in five ordered steps. Each step is one file under src/groq/. The router depends on the registry; the middleware and fallback depend on the client; the streaming pipeline stands alone. Full source for every step (verbatim) is in references/implementation.md.

1. Model Registry (models.ts) — declare a ModelSpec for each model with its tier, context window, speed, cost, and capabilities. Skeleton:

   export const MODELS: Record<string, ModelSpec> = {
     "llama-3.1-8b-instant":     { tier: "speed",   /* fast, cheap */ },
     "llama-3.3-70b-versatile":  { : ,  },
     : { :  },
     :   { :  },
   };