| name | groq-performance-tuning |
| description | Optimize Groq API performance with model selection, caching, streaming, and parallel requests.
Use when experiencing slow responses, implementing caching strategies,
or optimizing request throughput for Groq integrations.
Trigger with phrases like "groq performance", "optimize groq",
"groq latency", "groq caching", "groq slow", "groq speed".
|
| allowed-tools | Read, Write, Edit |
| version | 1.11.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
| tags | ["saas","groq","api","performance"] |
| compatibility | Designed for Claude Code, also compatible with Codex and OpenClaw |
Groq Performance Tuning
Overview
Maximize Groq's LPU inference speed advantage. Groq already delivers extreme throughput (280-560 tok/s) and low latency (<200ms TTFT), but client-side optimization -- model selection, prompt size, streaming, caching, and parallelism -- determines whether your application fully exploits that speed.
This skill walks through six tuning levers at a high level; the complete, copy-pasteable code for each lives in references/implementation.md, and end-to-end worked scenarios live in references/examples.md.
Prerequisites
- Groq API key — set
GROQ_API_KEY in the environment. The groq-sdk client (new Groq()) reads it automatically; never hardcode the key.
- Node.js 18+ with the
groq-sdk package installed (npm install groq-sdk).
- Optional packages for the caching and parallelism steps:
lru-cache and p-queue (npm install lru-cache p-queue).
- A baseline latency measurement of your current integration so you can confirm the tuning actually helps.
Groq Speed Benchmarks
| Model | TTFT | Throughput | Context |
|---|
llama-3.1-8b-instant | ~50ms | ~560 tok/s | 128K |
llama-3.3-70b-versatile | ~150ms | ~280 tok/s | 128K |
llama-3.3-70b-specdec | ~100ms | ~400 tok/s | 128K |
meta-llama/llama-4-scout-17b-16e-instruct | ~80ms | ~460 tok/s | 128K |
TTFT = Time to First Token. Actual values depend on prompt size and server load.
Instructions
Apply these six levers in order. Each is a small, independent change — start with the ones that match your bottleneck (model choice and caching give the biggest wins on most workloads). The full code for every step is in references/implementation.md.
- Choose the right model for speed. Map each call site to a speed tier:
llama-3.1-8b-instant for latency-critical paths, for quality-sensitive paths, for 70b quality at higher throughput. Set so responses are deterministic (and cacheable).
