grafana-best-practice

name

grafana-best-practice

description

Review, improve, organize, deploy, and verify Grafana dashboards, provisioned alert rules, and the Telegraf→InfluxDB→Grafana monitoring stack. Use when working on Grafana dashboards, folders, tags, legends, panel readability, Flux/InfluxDB query correctness or performance, Telegraf JSON/HTTP scraping, anonymous access, dashboard/alert provisioning, Grafana Docker deployments, high CPU or memory from dashboard queries, InfluxDB cardinality and tag-vs-field modelling, Flux alert conditions, contact points and Telegram delivery, or requests like "review this dashboard", "optimize Grafana panels", "why is this dashboard slow", "fix legends", "move dashboards into a folder", "set up Grafana alerting", "alert won't fire / won't deliver", "fix Telegram alerts", "route alerts to a Telegram topic (message_thread_id)", "bump the Grafana version", "deploy and verify dashboards", "InfluxDB is OOM-killed / restart-looping", "series cardinality explosion", "should this be a tag or a field", or "Grafana best practices".

Grafana Best Practice

Use this skill to make Grafana dashboards readable, correctly aggregated, navigable, and safely deployable. Treat the rendered dashboard, Grafana API state, datasource query results, and container health as the source of truth.

Core Workflow

1. Inspect the current dashboard JSON and provisioning files before editing:
   • Dashboard JSON: uid, title, tags, templating.list, links, panels.
   • Provisioning: provisioning/dashboards/*.yaml, especially folder, provider path, and allowUiUpdates.
   • Runtime state when available: /api/search, /api/folders, /api/dashboards/uid/<uid>, /api/ds/query, /api/health.
2. Audit dashboards for:
   • Filters: default values, multi/all behavior, variable labels, and whether All views preserve required dimensions.
   • Aggregation dimensions: each metric's real labels — a per-instance/shard id, per-event labels (e.g. symbol, reason, side in a trading app), container_name for containers, and host-level metrics that carry none of these.
   • Legend placement, values, sorting, and display names.
   • Long labels leaking into stat cards or legends, such as environment, project, url, or raw Prometheus family names.
   • Bar chart sorting and limits.
   • Table column headers — no field-wide defaults.displayName (it overrides every column); columns renamed via the organize transform.
   • Multi-bucket / union() queries — every filter() pushed inside each from() (predicate pushdown), never after union().
   • Query cost: panel count, target count, default time range, refresh interval, high-cardinality tags, expensive reshape operators, and whether current-value panels scan the whole dashboard range.
   • Tag-vs-field modelling on any dimension the dashboard filters or groups by — a churning value used as a tag is a latent datasource outage, not a query-tuning nit (see Tags Versus Fields).
   • Anonymous/read-only access if the dashboard is public.
3. Patch dashboard JSON narrowly and preserve existing panel intent. Row nesting differs by state: a collapsed row (collapsed: true) holds its child panels in the row's own panels[], but an expanded row (collapsed: false) keeps them as TOP-LEVEL siblings after the row object — not nested. A programmatic patch that assumes nesting silently skips every panel under an expanded row; index panels by walking both top-level entries and each row's panels[].
4. Validate locally with JSON parsing, repository tests, and static checks.
5. Validate queries against the real datasource before deployment when possible.
6. Deploy through the repository's normal provisioning path.
7. Verify post-deploy using live Grafana API and datasource query results. Do not rely only on file diffs.
8. Then verify what the page actually RENDERS, because step 7 cannot. Transformations and the display processor run client-side, so the API answers a different question than "what does the reader see". Open the deployed panel (/d-solo/<uid>/?panelId=N isolates one) and read its cells out of the DOM. Do this whenever you touched units, transforms, displayName, value mappings, overrides, or anything that returns more than one frame.

Hard Rules

These fail SILENTLY — the panel renders, the query returns, the alert stays green, and the result is wrong. You will not know to go read the reference file, so they live here. Each points at the file with the worked example.

• Never make a churning value a tag (status, "best route", container name on a shared host, interface name). It is a latent datasource outage, not a tuning nit — see below.
• schema.tagValues() / schema.fieldKeys() do not tell you what is being written now. They read the index and report the old shape after a collector change. Query real points.
• The InfluxDB Flux datasource IGNORES legendFormat. Series names come from group-key labels + displayName. A bare set() that is not in the group key is not a label. → flux.md
• Never group() before a pivot that mixes a string field with numeric ones — schema collision: column "_value" is both of type string and float. Pivot first, group after. (For an all-numeric pivot the opposite holds: group first, or the rowKey tags stay frame labels and pollute every column header.) → flux.md, panels.md
• aggregateWindow(fn: max|mean) cannot aggregate a string — use fn: last.
• Aggregators drop every column that is not a group key or the aggregated value. A column built with map() before spread() vanishes. Map after, or make it a group key. → flux.md
• Push every filter() INSIDE each from() before union(). A filter after union() is not pushed down: each branch full-scans the whole bucket into memory and can wedge a shared host. → flux.md
• Never run an unfiltered or heavy ad-hoc Flux query against a shared production datasource. Filter by measurement, keep the range tight, validate with count() first.
• Never set fieldConfig.defaults.displayName on a table — it overrides EVERY column header. Rename in the organize transform. → panels.md

Tags Versus Fields (Cardinality At The Source)

This is a collector-config decision (telegraf.conf tags vs included_keys), not a dashboard one, but it is the single thing most likely to take the datasource down — so audit it whenever you touch a dashboard that filters or groups by a status-like dimension.

series = distinct tag-sets × field keys. A tag whose value changes for a fixed entity over time — a "churning tag" — multiplies cardinality by every value it has ever taken inside the retention window. Status flags, a "best route"/"cheapest provider" pick, a container name on a shared host, a network interface name, a request id: all churn.

Three properties make this worse than an ordinary sizing mistake:

1. It does not self-heal. InfluxDB's series file is append-only. Retention expiry drops shard data but never reclaims series entries, so cardinality only ever goes up until someone rebuilds the index.
2. Uptime hides it. The cost is paid on shard-index load, i.e. only on a COLD start. A container can sit for weeks pegged at 95-100% of its memory cap and look perfectly healthy, then fail to boot forever after the first restart — a deploy, a host reboot, one heavy query. Cold-start capacity is not exercised by uptime; nothing warns you.
3. The estimate in the comment is probably wrong. Measured example (2026-08): a comment claiming "20-25k series worst case" sat above a measurement holding 166,773 — 7× off, and that staleness is why nobody looked. Another: six route-venue tags added on the belief that "cardinality is bounded by asset × a small venue-pair" minted 897,212 dead series, 79% of the whole bucket, and the resulting boot loop was a full outage.

Rules:

• Keep as tags only what genuinely identifies the entity (venue, asset, instance, container_name for containers). Put churning values in included_keys as string fields and pivot them back in at query time.
• State the arithmetic (rows × field keys) BEFORE changing a tag set, and re-measure AFTER deploying. Against a stopped instance: influxd inspect report-db --db-path <data>/<bucket-id> --rollup m. Every cardinality comment must carry a measured number and its date.
• Demoting a tag is a COORDINATED change — collector config, every dashboard panel that filters/groups on it, every alert rule, and the downsample task's string guard all move in one commit. A partial change breaks silently, not loudly.
• Never trust schema.tagValues() / schema.fieldKeys() to tell you what is being written NOW. They read the index, which still lists every tag value and field key whose series exists in an overlapping shard, so right after a collector change they report the OLD shape for the new window — a -3m query can return 30 days of dead values. To answer "what is actually landing", query real points: range(start: -90s) |> filter(...) |> keep(columns: ["<tag>"]) |> group() |> distinct(column: "<tag>").

Folder And Tag Rules

Prefer a named Grafana folder when permissions are known to be handled. For provisioned Grafana OSS dashboards with anonymous access, verify folder permissions explicitly because named folders can 403 even when dashboards exist.

Safe folder checklist:

• provisioning/dashboards/default.yaml uses the intended folder, for example folder: Production.
• Startup or deployment code grants Viewer read on every provisioned folder and dashboard when Grafana state is ephemeral.
• /api/folders shows the folder.
• /api/search?type=dash-db&tag=<base-tag> shows every dashboard with the expected folderTitle.
• Anonymous probes return 200 for both /d/<uid>/... and /dashboards.

Use one stable base tag for dashboard links and discovery, then add semantic tags. Keep tags low-cardinality and queryable.

Recommended tag shape — one stable base tag (the service slug) plus semantic axes. The values below are an illustrative set; substitute your own facets:

<service>                     # stable base tag, e.g. `myservice`
area:<facet>                  # e.g. area:overview | area:app | area:risk | area:infra
audience:<role>               # e.g. audience:operator | audience:oncall
plane:<data-plane>            # e.g. plane:prometheus | plane:app | plane:host

Keep dashboard dropdown links on the stable base tag (the service slug, e.g. myservice) so adding semantic tags does not break navigation.

Variables And Filters

Use explicit labels that explain important defaults:

{
  "name": "instance",
  "label": "Instance (All by default)",
  "multi": true,
  "includeAll": true,
  "allValue": ".*"
}

For high-level fleet dashboards, defaulting instance to All is good only if every panel handles All safely. When All is selected:

• Stat panels should either show one compact value per instance or deliberately aggregate fleet-wide.
• Time series panels should not collapse multiple instances into one unlabeled series.
• Legends and display names should stay short.

Detailed References

Load the file that matches what you are doing:

• Writing or fixing a Flux query — dimension handling, series naming, hidden-cardinality collapse, ratio math, reading a string field back, dual-bucket union() and downsampling: references/flux.md
• Reviewing or improving a dashboard's readability — information architecture, the answer-first summary row, and stat / timeseries / bar / table panel rules: references/panels.md
• Something is slow, timing out, or pegging the datasource — query-surface audit, bounded lookbacks, cheaper producer-side fields, and how to prove a performance fix: references/performance.md
• Alert rules, contact points, notification delivery — Flux alert conditions and reducer errors, noDataState choice, Telegram chat_id / supergroup / topic message_thread_id, parse-mode escaping, message-template composition (group-label-in-header DRY, no-text-color emphasis, resolved-only context), live rule-health and real delivery verification: references/alerting.md
• Deploying, verifying, upgrading, or recovering the stack — provisioning paths, folder permissions, Grafana major-version bumps, the minimum post-deploy verification curls, compose + Telegraf monitoring-stack gotchas, and the InfluxDB OOM boot-loop recovery runbook: references/deployment.md

Review Output

When reviewing or finishing work, report evidence, not just intent:

• Files changed.
• Dashboard count and panel query count verified.
• Folder and tag API evidence.
• Anonymous access probes.
• Datasource query result summary. After any schema-shaped change (a tag demoted, a measurement/field renamed, a bucket purged), extract EVERY panel target from the deployed dashboards, substitute the Grafana variables, and execute them all against the datasource — not just the ones you edited. It is cheap, it is scriptable, and it is the only thing that catches a panel you did not realise depended on the old shape. Report the tally (e.g. "70 queries: 69 returned data, 0 errors, 1 empty") and explain every empty one; an empty result is only acceptable once you have shown the query is sound by loosening its filter to a value that does exist.
• Cardinality before/after when a tag set changed, measured (not estimated) with influxd inspect report-db, plus proof the new shape is what is actually landing (a distinct() over real points from the last 90s — NOT schema.tagValues()).
• Query performance evidence when performance changed: target count, expensive operators removed, bounded time windows added, old/new query result or timing comparison, and live dashboard JSON proof after deploy.
• First-viewport summary row evidence for operational dashboards: key stat cards render, use range-correct semantics, and are not blank when expected data exists.
• Rendered-cell evidence whenever formatting, transforms or frame count changed — the actual DOM text of the affected panel, quoted. "The datasource returns the right value" is not evidence that the panel shows it; those are different layers and only the second one is what the reader gets. A before/after pair is best: it is the only thing that distinguishes a fixed panel from one whose bug you never saw.
• Alert rule health (/api/.../rules all ok) and a real delivery test, when alerting changed.
• Container/service health.
• Any risks intentionally left unchanged.

If a validation failure appears, fix the dashboard and rerun the relevant static, API, and datasource checks before declaring completion.