Cohere Observability
Overview
Set up production observability for Cohere API v2 with Prometheus metrics, OpenTelemetry tracing, and AlertManager rules. Tracks per-endpoint latency, token usage, error rates, and costs.
Prerequisites
- Prometheus or compatible metrics backend
- OpenTelemetry SDK installed
cohere-ai SDK v7+
Instructions
Step 1: Metrics Collection
import { Registry, Counter, Histogram, Gauge } from 'prom-client';
const registry = new Registry();
const requestCounter = new Counter({
name: 'cohere_requests_total',
help: 'Total Cohere API requests',
labelNames: ['endpoint', 'model', 'status'],
registers: [registry],
});
const requestDuration = new Histogram({
name: 'cohere_request_duration_seconds',
help: 'Cohere request duration',
labelNames: ['endpoint', 'model'],
buckets: [0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30],
registers: [registry],
});
const tokenCounter = new Counter({
name: 'cohere_tokens_total',
help: 'Total tokens consumed',
labelNames: ['endpoint', 'model', 'direction'],
registers: [registry],
});
const errorCounter = new Counter({
name: 'cohere_errors_total',
help: 'Cohere errors by status code',
labelNames: ['endpoint', 'status_code'],
registers: [registry],
});
const rateLimitGauge = new Gauge({
name: 'cohere_rate_limit_remaining',
help: 'Remaining rate limit capacity',
labelNames: ['endpoint'],
registers: [registry],
});
Step 2: Instrumented Client Wrapper
import { CohereClientV2, CohereError, CohereTimeoutError } from 'cohere-ai';
const cohere = new CohereClientV2();
async function instrumentedCall<T>(
endpoint: string,
model: string,
operation: () => Promise<T>
): Promise<T> {
const timer = requestDuration.startTimer({ endpoint, model });
try {
const result = await operation();
requestCounter.inc({ endpoint, model, status: 'success' });
timer();
const usage = (result as any)?.usage?.billedUnits;
if (usage) {
if (usage.inputTokens) {
tokenCounter.inc({ endpoint, model, direction: 'input' }, usage.inputTokens);
}
if (usage.outputTokens) {
tokenCounter.inc({ endpoint, model, direction: 'output' }, usage.outputTokens);
}
}
result;
} (err) {
requestCounter.({ endpoint, model, : });
();
(err ) {
errorCounter.({ endpoint, : (err.) });
} (err ) {
errorCounter.({ endpoint, : });
}
err;
}
}
response = (, ,
cohere.({
: ,
: [{ : , : query }],
})
);
Step 3: OpenTelemetry Tracing
import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api';
const tracer = trace.getTracer('cohere-client', '1.0.0');
async function tracedCohereCall<T>(
endpoint: string,
model: string,
operation: () => Promise<T>
): Promise<T> {
return tracer.startActiveSpan(
`cohere.${endpoint}`,
{ kind: SpanKind.CLIENT },
async (span) => {
span.setAttribute('cohere.model', model);
span.setAttribute('cohere.endpoint', endpoint);
try {
const result = await operation();
const usage = (result as any)?.usage?.billedUnits;
if (usage) {
span.setAttribute('cohere.tokens.input', usage.inputTokens ?? 0);
span.setAttribute('cohere.tokens.output', usage. ?? );
}
span.({ : . });
result;
} (: ) {
span.({ : ., : err. });
span.(err);
(err ) {
span.(, err. ?? );
}
err;
} {
span.();
}
}
);
}
Step 4: Structured Logging
import pino from 'pino';
const logger = pino({ name: 'cohere', level: process.env.LOG_LEVEL ?? 'info' });
function logCohereCall(
endpoint: string,
model: string,
durationMs: number,
status: 'success' | 'error',
meta?: Record<string, unknown>
) {
loggerstatus === 'error' ? 'error' : 'info';
}
async function observedCall<T>(
endpoint: string,
model: string,
fn: () => Promise<T>
): Promise<T> {
return tracedCohereCall(endpoint, model, () =>
instrumentedCall(endpoint, model, async () => {
const start = Date.now();
try {
const result = await fn();
(endpoint, model, .() - start, , {
: (result )?.?.,
});
result;
} (err) {
(endpoint, model, .() - start, , {
: err ? err. : ,
});
err;
}
})
);
}
Step 5: Alert Rules
groups:
- name: cohere
rules:
- alert: CohereHighErrorRate
expr: |
rate(cohere_errors_total[5m]) /
rate(cohere_requests_total[5m]) > 0.05
for: 5m
labels:
severity: warning
annotations:
summary: "Cohere error rate > 5%"
description: "{{ $labels.endpoint }} error rate: {{ $value | humanizePercentage }}"
- alert: CohereRateLimited
expr: rate(cohere_errors_total{status_code="429"}[5m]) > 0.1
for: 2m
labels:
severity: warning
annotations:
summary: "Cohere rate limiting detected"
- alert: CohereHighLatency
expr: |
histogram_quantile(0.95,
rate(cohere_request_duration_seconds_bucket[5m])
) > 10
for: 5m
labels:
severity: warning
annotations:
Step 6: Metrics Endpoint
import express from 'express';
const app = express();
app.get('/metrics', async (req, res) => {
res.set('Content-Type', registry.contentType);
res.send(await registry.metrics());
});
Dashboard Panels (Grafana)
| Panel | Query | Type |
|---|
| Request Rate | rate(cohere_requests_total[5m]) | Time series |
| Error Rate | rate(cohere_errors_total[5m]) / rate(cohere_requests_total[5m]) | Stat |
| P50/P95 Latency | histogram_quantile(0.95, rate(cohere_request_duration_seconds_bucket[5m])) | Time series |
| Token Usage | rate(cohere_tokens_total[1h]) | Bar chart |
| Errors by Code | sum by (status_code)(rate(cohere_errors_total[5m])) | Pie chart |
Output
- Prometheus metrics for requests, latency, tokens, and errors
- OpenTelemetry traces with Cohere-specific attributes
- Structured JSON logging with pino
- AlertManager rules for error rate, latency, auth, and cost
Error Handling
| Issue | Cause | Solution |
|---|
| Missing token metrics | Usage not in response | Check response.usage.billedUnits |
| High cardinality | Too many model labels | Use model family, not exact version |
| Alert storm | Threshold too low | Tune thresholds for your traffic |
| Trace gaps | Missing context propagation | Ensure OTel context flows through async |
Resources
Next Steps
For incident response, see cohere-incident-runbook.
