grafana

# Grafana Integration ## Connection - **Base URL**: `$GRAFANA_URL` (e.g. `https://your-org.grafana.net`) - **Auth**: `Authorization: Bearer $GRAFANA_API_TOKEN` (service account token) - **Credentials**: `GRAFANA_URL`, `GRAFANA_API_TOKEN` from Settings → Data sources - **Caching**: 10-minute cache for metadata; query results are NOT cached (time-sensitive) - **Key datasource**: Prometheus UID `grafanacloud-prom` ## Server Lib & API Routes - **File**: `server/lib/grafana.ts` ### Exported Functions | Function | Description | | --------------------------------------------- | --------------------------------------- | | `listDashboards(query?)` | Search dashboards by query | | `getDashboard(uid)` | Full dashboard JSON with panels | | `getDatasources()` | List all datasources | | `getAlertRules()` | All alert rules (flattened from groups) | | `getAlertInstances()` | Currently firing alert instances | | `queryDatasource(uid, queries[], from?, to?)` | Proxy to Grafana's `/api/ds/query` | ### API Routes | Route | Description | | ------------------------------------ | ----------------------------------------- | | `GET /api/grafana/dashboards` | Search dashboards | | `GET /api/grafana/dashboard?uid=...` | Full dashboard JSON | | `GET /api/grafana/datasources` | List datasources | | `GET /api/grafana/alerts` | Alert rules and firing instances | | `POST /api/grafana/query` | Query datasource (Prometheus, Loki, etc.) | ### Agent Action Use `grafana` for agent-facing Grafana work. Do not call `/api/grafana/*` directly from the agent. | Mode | Args | Description | | ------------- | ---------------------------------------- | -------------------------------- | | `dashboards` | `search` | Search dashboards | | `dashboard` | `uid` | Full dashboard JSON | | `datasources` | | List datasources | | `alerts` | | Alert rules and firing instances | | `query` | `datasourceUid`, `queries`, `from`, `to` | Query a datasource | ### Dashboard - `/adhoc/engineering` — Engineering dashboard (mirrors a Grafana dashboard by UID) ## Grafana Response Frame Format ``` { results: { [refId]: { frames: [{ schema: { fields }, data: { values } }] } } } ``` - `values[0]` = timestamps (epoch ms), `values[1+]` = metric values - Series labels from `schema.fields[i].labels` (e.g. `{ model: "claude-3.5" }`) - Transform to Recharts: `{ time: number, [seriesName]: number }[]` ## Template Variables - `$CodegenMode` → default `quality-v4` - `$AIModel` → regex pattern, default `.*` (all) - `$environment` → `cloud` or `cloud-v2`, default `cloud` - Variables interpolated via string replace before querying ## Key Prometheus Metrics - **Codegen**: `vcpcodegen_completion_total`, `vcpcodegen_completion_latency_bucket`, `vcpcodegen_feedback_total`, `vcpcodegen_error_total` - **LLM**: `llm_completion_cost_total`, `llm_input_tokens_total`, `llm_output_tokens_total`, `llm_latency_bucket`, `llm_failures_total`, `llm_completions_total` - **Projects**: `projects_proposed_config_total`, `projects_status_total`, `projects_remote_machine_*`, `projects_start_duration_bucket` - **API/Runtime**: `api_request_total`, `with_span_duration_bucket`, `memory_heap_usage_percent_bucket` - **Fly.io**: `fly_endpoint_total`, `fly_machine_*` - **GitHub**: `builderbot_pr_created_total`, `builderbot_pr_closed_total` ## Key Patterns & Gotchas - `queryDatasource` constructs body with datasource UID in each query target; timestamps as string ms - `getAlertRules` flattens nested rule groups from unified alerting API - `getAlertInstances` handles both array and wrapped object response shapes - POST helper caches by default (fine for idempotent endpoints) but `queryDatasource` skips cache - Heatmap panels render as multi-series line/area (Recharts has no native heatmap; PromQL `histogram_quantile()` computes quantiles) ## Incident Investigation Pattern For production issues, query Grafana/Prometheus FIRST: 1. LLM latency by model (`llm_latency_bucket`) 2. Request rates (`api_request_total`) 3. Error rates (`llm_failures_total`) 4. Instance counts (via Cloud Monitoring) Then check Agent-Native Analytics Monitoring -> Errors for captured application issues, external Sentry when connected, and Cloud Logging for raw logs.