Groq Observability
Overview
Monitor Groq LPU inference for latency, token throughput, rate limit utilization, and cost. Groq's defining advantage is speed (280-560 tok/s), so latency degradation is the highest-priority signal. The API returns rich timing metadata (queue_time, prompt_time, completion_time) and rate limit headers on every response.
Prerequisites
- A Groq account with an API key exported as the
GROQ_API_KEY environment variable — the groq-sdk client reads it automatically (new Groq()).
- Node.js with
groq-sdk and prom-client installed (npm install groq-sdk prom-client).
- A Prometheus scrape target and (optionally) Grafana for the dashboard panels.
Key Metrics to Track
| Metric | Type | Source | Why |
|---|
| TTFT (time to first token) | Histogram | Client-side timing | Groq's main value prop |
| Tokens/second | Gauge | usage.completion_time | Throughput degradation |
| Total latency | Histogram | Client-side timing | End-to-end performance |
| Rate limit remaining | Gauge | x-ratelimit-remaining-* headers | Prevent 429s |
| Token usage | Counter | usage.total_tokens | Cost attribution |
| Error rate by code | Counter | Error handler | Availability |
| Estimated cost | Counter | Tokens * model price | Budget tracking |
Instructions
Apply these six steps in order. Steps 1-2 are the core instrumentation loop —
wrap the client, then feed a Prometheus instrument set from each call. Steps 3-6
add rate-limit tracking, alerting, structured logs, and dashboards on top. The
lean client skeleton is below; the full code for every step lives in
references/implementation.md.
- Instrumented client — wrap
groq.chat.completions.create so latency, tokens, queue time, and estimated cost are captured on the same path as the request (trackedCompletion).
- Prometheus metrics — register a histogram (latency), counters (tokens, cost, errors), and gauges (throughput, rate-limit remaining), then feed them from
emitMetrics.
- Rate limit header tracking — parse
x-ratelimit-remaining-* off every response into a gauge so you alert before a 429, not after.
- Prometheus alert rules — ship latency/rate-limit/throughput/error/cost alerts tuned to Groq's sub-200ms, 280+ tok/s baseline.
- Structured request logging — emit one JSON line per request for log aggregation, preserving per-request detail metrics roll up.
- Dashboard panels — TTFT distribution, tokens/sec, rate-limit utilization, request volume, error rate, cost, and queue time.
import Groq from "groq-sdk";
const groq = new Groq();
async function trackedCompletion(model: string, messages: any[]) {
const start = performance.now();
const result = await groq.chat.completions.create({ model, messages });
const latencyMs = performance.now() - start;
const usage = result.usage!;
const metrics = {
model,
latencyMs: Math.round(latencyMs),
tokensPerSec: Math.round(usage.completion_tokens / ((usage as any).completion_time || latencyMs / 1000)),
totalTokens: usage.total_tokens,
};
emitMetrics(metrics);
return { result, metrics };
}
See references/implementation.md for the complete
GroqMetrics shape, pricing table, Prometheus instruments, rate-limit tracking,
alert rules, structured logging, and dashboard panel list.
Output
Applying the workflow produces:
- A
trackedCompletion wrapper that returns { result, metrics }, where metrics is a GroqMetrics object (latency, TTFT, tokens/sec, token counts, queue time, estimated cost).
- A Prometheus metric set —
groq_latency_ms (histogram), groq_tokens_total / groq_cost_usd / groq_errors_total (counters), and groq_tokens_per_second / groq_ratelimit_remaining (gauges).
- Five alert rules (
GroqLatencyHigh, GroqRateLimitCritical, GroqThroughputDrop, GroqErrorRateHigh, GroqCostSpike).
- A structured JSON log line per request and a 7-panel dashboard spec.
Examples
Instrument a single completion and emit a structured log line:
const { result, metrics } = await trackedCompletion(
"llama-3.3-70b-versatile",
[{ role: "user", content: "Summarize this incident report in two sentences." }]
);
logGroqRequest(metrics, result.id);
For a 429-guard using rate-limit headers and a dashboard health-reading table,
see references/examples.md.
Error Handling
| Issue | Cause | Solution |
|---|
| 429 with high retry-after | RPM or TPM exhausted | Implement request queuing |
| Latency spike > 2s | Model overloaded or large prompt | Reduce prompt size or switch to lighter model |
| 503 Service Unavailable | Groq capacity issue | Enable fallback to alternative provider |
| Tokens/sec drop | Streaming disabled or large prompts | Enable streaming for better perceived performance |
Resources
