# Grafana > [Applies to: **/*] This guide defines definitive best practices for developing, structuring, and maintaining Grafana dashboards, alerts, and data sources as code, ensuring readability, performance, and scalability across teams. - Skill: `tryboy869/grafana` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tryboy869/grafana` - Raw SKILL.md: https://api.skillmd.com/api/skills/tryboy869/grafana/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: Tryboy869 (https://skillmd.com/u/tryboy869) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/tryboy869/grafana --- # Grafana Best Practices Our team treats Grafana dashboards, alerts, and data sources as first-class code artifacts. This means applying software engineering principles: version control, peer review, automated testing, and CI/CD. Adhere strictly to these guidelines to ensure our observability platform remains robust, maintainable, and actionable. ## 1. Code Organization and Structure Define all Grafana resources declaratively using Observability as Code (OaC). This enables Git-based workflows, automated provisioning, and consistent deployments. ### 1.1. Declarative Configuration (OaC) **Always** define dashboards, alerts, and data sources in JSON/YAML files. Store these in Git and provision them via Grafana's API. Avoid manual UI configuration for anything beyond initial prototyping. ❌ **BAD**: Manual UI changes, no version history. ✅ **GOOD**: Git-versioned JSON/YAML files, automated provisioning. ```json // my-service-dashboard.json { "apiVersion": 1, "title": "My Service Overview", "uid": "my-service-overview", "panels": [ // ... panel definitions ... ], "templating": { "list": [ // ... variables ... ] } } ``` ### 1.2. Utilize Foundation SDKs For complex or dynamically generated dashboards, **prefer** using Grafana Foundation SDKs (Go, Python, TypeScript) to programmatically define resources. This provides strong typing, validation, and better code reusability than raw JSON manipulation. ```typescript // dashboard.ts (using TypeScript SDK) import { Dashboard, Row, Panel, GraphPanel } from '@grafana/sdk'; const dashboard = new Dashboard({ title: 'Service Health', uid: 'service-health', panels: [ new Row({ panels: [ new GraphPanel({ title: 'CPU Utilization', targets: [{ expr: 'sum(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)' }], yAxis: { min: 0 } }) ] }) ] }); console.log(JSON.stringify(dashboard.toGrafanaJson(), null, 2)); ``` ### 1.3. Templated Variables **Mandate** the use of templated variables for dynamic filtering and reusability across environments or instances. This prevents dashboard duplication. ❌ **BAD**: Hardcoded instance names or environment-specific queries. ```json // Duplicated dashboard for 'prod' and 'dev' { "title": "Prod CPU Usage", "targets": [{ "expr": "node_cpu_seconds_total{instance='prod-app-01'}" }] } { "title": "Dev CPU Usage", "targets": [{ "expr": "node_cpu_seconds_total{instance='dev-app-01'}" }] } ``` ✅ **GOOD**: Single dashboard with variables. ```json { "title": "Service CPU Usage", "templating": { "list": [ { "name": "instance", "type": "query", "query": "label_values(node_cpu_seconds_total, instance)" }, { "name": "environment", "type": "custom", "options": ["prod", "dev"], "current": { "value": "prod" } } ] }, "panels": [ { "title": "CPU Utilization on $instance ($environment)", "targets": [{ "expr": "node_cpu_seconds_total{instance=\"$instance\", env=\"$environment\"}" }] } ] } ``` ## 2. Common Patterns and Anti-patterns Design dashboards for immediate diagnostic value and clarity. ### 2.1. Diagnostic Frameworks (USE & Four Golden Signals) **Structure dashboards** around the USE method (Utilization, Saturation, Errors) for resource-centric views and the Four Golden Signals (Rate, Errors, Duration, Latency) for service-centric views. Each row should focus on a single resource or signal. * **USE Method**: Left column for Utilization, right for Saturation/Errors. * **Four Golden Signals**: Panels ordered logically (e.g., Rate, Errors, Latency). ### 2.2. Visual Consistency **Enforce** these visual standards for immediate comprehension: * **Y-axis**: **Always** zero-based. * **Graph Type**: Prefer thin line graphs (`fill: 0`) for time-series. Use bar charts for quantities per interval (e.g., rates, CPU usage). * **Metrics per Panel**: Limit to **max 4 lines** per graph panel. More than this indicates the panel is trying to answer too many questions. * **Titles & Legends**: Clear, concise panel titles. Informative legends using `alias` to describe metric transformations. ❌ **BAD**: Misleading Y-axis, cluttered panels. ```json { "title": "Service Latency", "type": "graph", "yAxis": { "min": null }, // Y-axis auto-scales, can hide small changes "targets": [ { "expr": "service_latency_p99", "legendFormat": "P99" }, { "expr": "service_latency_p95", "legendFormat": "P95" }, { "expr": "service_latency_p75", "legendFormat": "P75" }, { "expr": "service_latency_mean", "legendFormat": "Mean" }, { "expr": "service_latency_median", "legendFormat": "Median" } // Too many lines ] } ``` ✅ **GOOD**: Zero-based Y-axis, focused metrics, clear aliases. ```json { "title": "Service Latency (P99 & P95)", "type": "graph", "yAxis": { "min": 0 }, // Always zero-based "options": { "legend": { "displayMode": "table", "calcs": ["mean", "max"] } }, "fieldConfig": { "defaults": { "custom": { "drawStyle": "line", "lineInterpolation": "linear", "fillOpacity": 0 } } }, "targets": [ { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "P99 Latency" }, { "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "P95 Latency" } ] } ``` ### 2.3. Metric Selection * **Counters**: **Always** use `.rate()` for StatsD counters from Graphite or `rate()`/`irate()` for Prometheus counters. Never `count()` or `sum()`. * **Timing**: **Prioritize** `max`, `p99`, or `p75` for timing metrics. Avoid `mean` or `median` as they hide outliers. ❌ **BAD**: Using `sum` for a counter. ```json { "targets": [{ "expr": "sum(http_requests_total)" }] } // Shows cumulative total, not rate ``` ✅ **GOOD**: Using `rate` for a counter. ```json { "targets": [{ "expr": "sum(rate(http_requests_total[5m]))" }] } // Shows requests per second ``` ## 3. Performance Considerations Efficient dashboards minimize load on data sources and Grafana itself. ### 3.1. Query Optimization **Keep queries minimal and efficient**. Avoid excessive aggregations or long time ranges in individual panels unless explicitly needed for historical context. Use `group by` sparingly. ### 3.2. Auto-Refresh Interval **Set auto-refresh to sensible defaults** (e.g., 5min or 15min). Avoid smaller intervals (<1min) for non-alerting dashboards to prevent unnecessary load on the metric database. For immediate notification, use alerts. ```json { "refresh": "5m", // Recommended default "time": { "from": "now-6h", "to": "now" } } ``` ## 4. Common Pitfalls and Gotchas Avoid these common mistakes that lead to confusing or inefficient dashboards. ### 4.1. Misleading Y-axis **Never** allow the Y-axis to auto-scale or start above zero for percentage or count metrics. This distorts trends and makes small changes appear significant. ### 4.2. Overloaded Panels A panel with too many metrics or complex queries becomes unreadable and slow. If you can't interpret a panel at a glance, it's overloaded. Break it down. ### 4.3. Lack of Context Every dashboard **must** include a "Legend" text panel at the top. This panel should describe: * The dashboard's purpose. * The data flow and any significant transformations. * Links to relevant documentation or source repositories. ```json { "title": "Legend", "type": "text", "gridPos": { "x": 0, "y": 0, "w": 24, "h": 2 }, "options": { "content": "This dashboard monitors the **Payment Service** health. Metrics are collected via Prometheus exporters, aggregated every 15s. [Service Docs](https://internal.wiki/payment-service) | [Source Code](https://github.com/org/payment-service)", "mode": "markdown" } } ``` ### 4.4. Inconsistent Annotations **Standardize annotations** for deployments and critical events. Configure dashboards to query common tags (e.g., `deploy`, `incident`). ```json { "annotations": { "list": [ { "name": "Deployments", "datasource": "Grafana", "iconColor": "rgba(255, 96, 96, 1)", "enable": true, "query": { "tags": "deploy", "matchAny": true } } ] } } ``` ## 5. Testing Approaches Treat Grafana configurations as production code, subject to automated testing and review. ### 5.1. Dashboard Linter Integration **Integrate `grafana/dashboard-linter`** into your CI/CD pipeline. This tool automatically scans JSON definitions for common anti-patterns (e.g., missing titles, non-zero-based axes, excessive series). ```bash # Example CI/CD step - name: Lint Grafana Dashboards run: | go install github.com/grafana/dashboard-linter@latest dashboard-linter lint dashboards/*.json ``` ### 5.2. Peer Review **Require peer review** for all changes to Grafana configuration files (dashboards, alerts, data sources). This ensures adherence to standards and catches logical errors before deployment. ### 5.3. Automated Provisioning & Git Sync **Leverage Grafana's provisioning API** for automated deployment. For Grafana 12+, **explore and adopt Git Sync** to manage dashboards directly from the UI with Git workflows (branches, PRs). This bridges the gap between UI-driven iteration and code-driven version control.