otel-dashboards

name	otel-dashboards
description	Building and editing Grafana dashboards for this stack — the Claude Code / Codex telemetry schema (Prometheus metric names, Loki event/attribute names), PromQL/LogQL query patterns, the Loki-labels gotcha, and panel/color conventions. Use when adding or editing panels in dashboards/*.json.

OTel dashboard development

Grafana reads Prometheus (metrics) and Loki (logs/events). Dashboards live in dashboards/*.json and hot-reload (no restart). Validate with make dashboard-validate.

Telemetry schema

OTel dot-notation → Prometheus underscores, _total suffix on counters (claude_code.cost.usage → claude_code_cost_usage_USD_total).

Prometheus counters: claude_code_session_count_total, claude_code_lines_of_code_count_total (label type=added/removed), claude_code_pull_request_count_total, claude_code_commit_count_total, claude_code_cost_usage_USD_total (label model), claude_code_token_usage_tokens_total (labels type,model), claude_code_code_edit_tool_decision_total (labels tool,decision).

Loki events (log body carries the full claude_code.<event> name; the event_name label carries the short form): user_prompt (prompt_length), tool_result (tool_name, success, duration_ms, error), api_request (model, cost_usd, duration_ms, input/output/cache_read_tokens), api_error (model, error, status_code, attempt), tool_decision (tool_name, decision, source).

Standard attributes on all data: session.id, app.version, organization.id, user.account_uuid, operator (who was at keyboard), project, hostname, branch, linear.key, catalyst.orchestration, catalyst.role, catalyst.orchestrator. Cardinality controls and full env-var list: see CLAUDE_OBSERVABILITY.md.

Codex (OpenAI) telemetry — same pipeline, no extra config

Codex CLI/desktop rides the same collector → Prometheus/Loki path; no Codex-specific config exists. Native codex_* metrics pass straight through; log events land in Loki under service_name="codex_exec" (widen selectors to service_name=~"codex.*") via the service-agnostic transform/metrics_normalize (event.name → event_name). Surfaced by dashboards/codex-usage.json. No cost telemetry — ChatGPT-subscription auth emits no codex_*_cost_* series, so no cost panels.

Prometheus: codex_conversation_turn_count_total (model, originator, host_name) · codex_tool_call_total (tool, success, originator) · codex_turn_token_usage_sum/_count/_bucket (token_type, model) · codex_turn_ttft_seconds_bucket / _ttfm_ / _e2e_ latency histograms (model) · codex_websocket_request_total.

Loki events (body |= "codex.tool_result" under {service_name=~"codex.*"}; attrs event_name/originator/model/tool/success are labels): codex.api_request, codex.tool_result, codex.tool_decision, codex.conversation_starts, codex.user_prompt, codex.turn_ttft.

Codex Usage dashboard conventions (codex-usage.json, UID codex-usage; gated by scripts/validate-dashboard.sh + make dashboard-validate):

• Template vars model/originator/hostname/bucket; selectors on codex_* (Prom) and service_name=~"codex.*" (Loki).
• Sparse-event KPIs (Turns, Tool Calls, Threads, event logs) use Loki count_over_time, not Prometheus increase() — native codex_* counters expire at metric_expiration:15m, so a short/idle session keeps only its first sample and increase() reads zero. Turns = codex.turn_ttft; Tool Calls = codex.tool_result; Threads = codex.conversation_starts.
• The Tokens KPI has no per-event Loki source, so it uses the Prometheus codex_turn_token_usage_sum histogram, filtered by model only (it carries token_type+model, never originator/host_name).

Loki labels — critical gotcha

OTel log-record attributes become Loki stream labels (service_name, event_name, tool_name, success, status_code, model, duration_ms, …). The event_name label is the short name (tool_result), never the claude_code.-prefixed form (that's in the body).

• Filter by body substring (preferred, matches existing panels): {service_name=~"claude-code.*"} |= "claude_code.tool_result"
• Filter/aggregate by label: | success = "true", sum by (tool_name) (...), | line_format "{{.tool_name}}"
• Never | json — the body is a plain string, not JSON.

Query patterns

PromQL:

sum by (model) (rate(claude_code_cost_usage_USD_total{job="otel-collector"}[5m]))
sum(increase(claude_code_session_count_total{job="otel-collector"}[1h]))
sum by (model) (changes(claude_code_cost_usage_USD_total[5m]))

LogQL:

sum by (tool_name) (count_over_time({service_name=~"claude-code.*"} |= "claude_code.tool_result" [$__range]))
sort_desc(topk(10, sum by (tool_name) (count_over_time({service_name=~"claude-code.*"} |= "claude_code.tool_result" [$__range]))))

Sparse-event counts (occurrences, idle/short sessions): use Loki count_over_time, not Prometheus increase() — native counters expire at metric_expiration:15m, so increase() reads zero for a just-born/idle counter.

Conventions

• Stat/KPI: lastNotNull, thresholds green→yellow→red, background color mode. Guard sparse LogQL with or vector(0) so empty reads show 0 not "No data".
• Timeseries: table legend with max/mean/sum; axis units; per-series color overrides.
• Logs: line_format for readability, sort descending.
• histogram_quantile: keep le inside sum by (le, ...).
• Template vars: includeAll:true, allValue:".*", multi:true. allValue:".*" also matches series lacking the label.
• Colors: green=success/added · yellow=warning · red=error/removed/Bash · blue=Read/Haiku/cache-read · purple=Sonnet/cache-create · orange=input/Grep.

Adding a panel: pick source (Prometheus=counters, Loki=events) → write query with the patterns above → choose viz → set gridPos (h,w,x,y) → thresholds/colors → test against live data → make dashboard-validate.