Deepgram Observability
Overview
Full observability stack for Deepgram: Prometheus metrics (request counts, latency histograms, audio processed, cost tracking), OpenTelemetry distributed tracing, structured JSON logging with Pino, Grafana dashboard JSON, and AlertManager rules.
Four Pillars
| Pillar | Tool | What It Tracks |
|---|
| Metrics | Prometheus | Request rate, latency, error rate, audio minutes, estimated cost |
| Traces | OpenTelemetry | End-to-end request flow, Deepgram API span timing |
| Logs | Pino (JSON) | Request details, errors, audit trail |
| Alerts | AlertManager | Error rate >5%, P95 latency >10s, rate limit hits |
Instructions
Step 1: Prometheus Metrics Definition
import { Counter, Histogram, Gauge, Registry, collectDefaultMetrics } from 'prom-client';
const registry = new Registry();
collectDefaultMetrics({ register: registry });
const requestsTotal = new Counter({
name: 'deepgram_requests_total',
help: 'Total Deepgram API requests',
labelNames: ['method', 'model', 'status'] as const,
registers: [registry],
});
const latencyHistogram = new Histogram({
name: 'deepgram_request_duration_seconds',
help: 'Deepgram API request duration',
labelNames: ['method', 'model'] as const,
buckets: [0.1, 0.5, 1, 2, 5, 10, 30, 60],
registers: [registry],
});
const audioProcessedSeconds = new Counter({
name: 'deepgram_audio_processed_seconds_total',
help: 'Total audio seconds processed',
labelNames: ['model'] as const,
registers: [registry],
});
const estimatedCostDollars = new Counter({
name: 'deepgram_estimated_cost_dollars_total',
help: 'Estimated cost in USD',
labelNames: ['model', 'method'] as const,
registers: [registry],
});
const activeConnections = new Gauge({
name: 'deepgram_active_websocket_connections',
help: 'Currently active WebSocket connections',
registers: [registry],
});
const rateLimitHits = new Counter({
name: 'deepgram_rate_limit_hits_total',
help: 'Number of 429 rate limit responses',
registers: [registry],
});
export { registry, requestsTotal, latencyHistogram, audioProcessedSeconds,
estimatedCostDollars, activeConnections, rateLimitHits };
Step 2: Instrumented Deepgram Client
import { createClient, DeepgramClient } from '@deepgram/sdk';
class InstrumentedDeepgram {
private client: DeepgramClient;
private costPerMinute: Record<string, number> = {
'nova-3': 0.0043, 'nova-2': 0.0043, 'base': 0.0048, 'whisper-large': 0.0048,
};
constructor(apiKey: string) {
this.client = createClient(apiKey);
}
async transcribeUrl(url: string, options: Record<string, any> = {}) {
const model = options.model ?? 'nova-3';
const timer = latencyHistogram.startTimer({ method: 'prerecorded', model });
try {
const { result, error } = await this.client.listen.prerecorded.(
{ url }, { model, : , ...options }
);
status = error ? : ;
();
requestsTotal.({ : , model, status });
(error) {
((error ). === ) rateLimitHits.();
error;
}
duration = result..;
audioProcessedSeconds.({ model }, duration);
estimatedCostDollars.(
{ model, : },
(duration / ) * (.[model] ?? )
);
result;
} (err) {
();
requestsTotal.({ : , model, : });
err;
}
}
() {
model = options. ?? ;
activeConnections.();
connection = ...(options);
originalFinish = connection..(connection);
connection. = {
activeConnections.();
();
};
connection;
}
}
Step 3: OpenTelemetry Tracing
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
import { trace } from '@opentelemetry/api';
const sdk = new NodeSDK({
resource: new Resource({
[SEMRESATTRS_SERVICE_NAME]: 'deepgram-service',
'deployment.environment': process.env.NODE_ENV ?? 'development',
}),
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/traces',
}),
instrumentations: [
getNodeAutoInstrumentations({
'@opentelemetry/instrumentation-http': {
ignoreIncomingPaths: ['/health', '/metrics'],
},
}),
],
});
sdk.();
tracer = trace.();
() {
tracer.(, (span) => {
span.(, model);
span.(, url.(, ));
{
instrumented = (process..!);
result = instrumented.(url, { model });
span.(, result..);
span.(, result..);
span.(,
result..[].[].);
result;
} (: ) {
span.(err);
span.({ : , : err. });
err;
} {
span.();
}
});
}
Step 4: Structured Logging with Pino
import pino from 'pino';
const logger = pino({
level: process.env.LOG_LEVEL ?? 'info',
formatters: {
level: (label) => ({ level: label }),
},
timestamp: pino.stdTimeFunctions.isoTime,
base: {
service: 'deepgram-integration',
env: process.env.NODE_ENV,
},
});
const transcriptionLog = logger.child({ component: 'transcription' });
const metricsLog = logger.child({ component: 'metrics' });
transcriptionLog.info({
action: 'transcribe',
model: 'nova-3',
audioUrl: url.substring(0, 100),
requestId: result.metadata.request_id,
duration: result.metadata.duration,
confidence: result.results.channels[0].[].,
}, );
transcriptionLog.({
: ,
: ,
: err.,
: err.,
}, );
Step 5: Grafana Dashboard Panels
{
"title": "Deepgram Observability",
"panels": [
{
"title": "Request Rate",
"type": "timeseries",
"targets": [{ "expr": "rate(deepgram_requests_total[5m])" }]
},
{
"title": "P95 Latency",
"type": "gauge",
"targets": [{ "expr": "histogram_quantile(0.95, rate(deepgram_request_duration_seconds_bucket[5m]))" }]
},
{
"title": "Error Rate %",
"type": "stat"
Step 6: AlertManager Rules
groups:
- name: deepgram-alerts
rules:
- alert: DeepgramHighErrorRate
expr: >
rate(deepgram_requests_total{status="error"}[5m])
/ rate(deepgram_requests_total[5m]) > 0.05
for: 5m
labels: { severity: critical }
annotations:
summary: "Deepgram error rate > 5% for 5 minutes"
- alert: DeepgramHighLatency
expr: >
histogram_quantile(0.95,
rate(deepgram_request_duration_seconds_bucket[5m])
) > 10
for: 5m
labels: { severity: warning }
annotations:
summary: "Deepgram P95 latency > 10 seconds"
- alert: DeepgramRateLimited
expr: rate(deepgram_rate_limit_hits_total[1h]) > 10
for: 10m
labels: { severity: warning }
annotations:
summary: "Deepgram rate limit hits > 10/hour"
- alert: DeepgramCostSpike
{ }
{ }
Metrics Endpoint
import express from 'express';
const app = express();
app.get('/metrics', async (req, res) => {
res.set('Content-Type', registry.contentType);
res.send(await registry.metrics());
});
Output
- Prometheus metrics (6 metrics covering requests, latency, usage, cost)
- Instrumented Deepgram client with auto-tracking
- OpenTelemetry distributed tracing with custom spans
- Structured JSON logging (Pino)
- Grafana dashboard panel definitions
- AlertManager rules (5 alerts)
Error Handling
| Issue | Cause | Solution |
|---|
| Metrics not appearing | Registry not exported | Check /metrics endpoint |
| High cardinality | Too many label values | Limit labels to known set |
| Alert storms | Thresholds too sensitive | Add for: duration, tune values |
| Missing traces | OTEL exporter not configured | Set OTEL_EXPORTER_OTLP_ENDPOINT |
Resources
