Grafana

Grafana Best Practices

Our team treats Grafana dashboards, alerts, and data sources as first-class code artifacts. This means applying software engineering principles: version control, peer review, automated testing, and CI/CD. Adhere strictly to these guidelines to ensure our observability platform remains robust, maintainable, and actionable.

1. Code Organization and Structure

Define all Grafana resources declaratively using Observability as Code (OaC). This enables Git-based workflows, automated provisioning, and consistent deployments.

1.1. Declarative Configuration (OaC)

Always define dashboards, alerts, and data sources in JSON/YAML files. Store these in Git and provision them via Grafana's API. Avoid manual UI configuration for anything beyond initial prototyping.

❌ BAD: Manual UI changes, no version history. ✅ GOOD: Git-versioned JSON/YAML files, automated provisioning.

// my-service-dashboard.json
{
  "apiVersion": 1,
  "title": "My Service Overview",
  "uid": "my-service-overview",
  "panels": [
    // ... panel definitions ...
  ],
  "templating": {
    "list": [
      // ... variables ...
    ]
  }
}

1.2. Utilize Foundation SDKs

For complex or dynamically generated dashboards, prefer using Grafana Foundation SDKs (Go, Python, TypeScript) to programmatically define resources. This provides strong typing, validation, and better code reusability than raw JSON manipulation.

// dashboard.ts (using TypeScript SDK)
import { Dashboard, Row, Panel, GraphPanel } from '@grafana/sdk';

const dashboard = new Dashboard({
  title: 'Service Health',
  uid: 'service-health',
  panels: [
    new Row({
      panels: [
        new GraphPanel({
          title: 'CPU Utilization',
          targets: [{ expr: 'sum(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)' }],
          yAxis: { min: 0 }
        })
      ]
    })
  ]
});

console.log(JSON.stringify(dashboard.toGrafanaJson(), null, 2));

1.3. Templated Variables

Mandate the use of templated variables for dynamic filtering and reusability across environments or instances. This prevents dashboard duplication.

❌ BAD: Hardcoded instance names or environment-specific queries.

// Duplicated dashboard for 'prod' and 'dev'
{ "title": "Prod CPU Usage", "targets": [{ "expr": "node_cpu_seconds_total{instance='prod-app-01'}" }] }
{ "title": "Dev CPU Usage", "targets": [{ "expr": "node_cpu_seconds_total{instance='dev-app-01'}" }] }

✅ GOOD: Single dashboard with variables.

{
  "title": "Service CPU Usage",
  "templating": {
    "list": [
      { "name": "instance", "type": "query", "query": "label_values(node_cpu_seconds_total, instance)" },
      { "name": "environment", "type": "custom", "options": ["prod", "dev"], "current": { "value": "prod" } }
    ]
  },
  "panels": [
    {
      "title": "CPU Utilization on $instance ($environment)",
      "targets": [{ "expr": "node_cpu_seconds_total{instance=\"$instance\", env=\"$environment\"}" }]
    }
  ]
}

2. Common Patterns and Anti-patterns

Design dashboards for immediate diagnostic value and clarity.

2.1. Diagnostic Frameworks (USE & Four Golden Signals)

Structure dashboards around the USE method (Utilization, Saturation, Errors) for resource-centric views and the Four Golden Signals (Rate, Errors, Duration, Latency) for service-centric views. Each row should focus on a single resource or signal.

• USE Method: Left column for Utilization, right for Saturation/Errors.
• Four Golden Signals: Panels ordered logically (e.g., Rate, Errors, Latency).

2.2. Visual Consistency

Enforce these visual standards for immediate comprehension:

• Y-axis: Always zero-based.
• Graph Type: Prefer thin line graphs (fill: 0) for time-series. Use bar charts for quantities per interval (e.g., rates, CPU usage).
• Metrics per Panel: Limit to max 4 lines per graph panel. More than this indicates the panel is trying to answer too many questions.
• Titles & Legends: Clear, concise panel titles. Informative legends using alias to describe metric transformations.

❌ BAD: Misleading Y-axis, cluttered panels.

{
  "title": "Service Latency",
  "type": "graph",
  "yAxis": { "min": null }, // Y-axis auto-scales, can hide small changes
  "targets": [
    { "expr": "service_latency_p99", "legendFormat": "P99" },
    { "expr": "service_latency_p95", "legendFormat": "P95" },
    { "expr": "service_latency_p75", "legendFormat": "P75" },
    { "expr": "service_latency_mean", "legendFormat": "Mean" },
    { "expr": "service_latency_median", "legendFormat": "Median" } // Too many lines
  ]
}

✅ GOOD: Zero-based Y-axis, focused metrics, clear aliases.

{
  "title": "Service Latency (P99 & P95)",
  "type": "graph",
  "yAxis": { "min": 0 }, // Always zero-based
  "options": { "legend": { "displayMode": "table", "calcs": ["mean", "max"] } },
  "fieldConfig": { "defaults": { "custom": { "drawStyle": "line", "lineInterpolation": "linear", "fillOpacity": 0 } } },
  "targets": [
    { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "P99 Latency" },
    { "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "P95 Latency" }
  ]
}

2.3. Metric Selection

• Counters: Always use .rate() for StatsD counters from Graphite or rate()/irate() for Prometheus counters. Never count() or sum().
• Timing: Prioritize max, p99, or p75 for timing metrics. Avoid mean or median as they hide outliers.

❌ BAD: Using sum for a counter.

{ "targets": [{ "expr": "sum(http_requests_total)" }] } // Shows cumulative total, not rate

✅ GOOD: Using rate for a counter.

{ "targets": [{ "expr": "sum(rate(http_requests_total[5m]))" }] } // Shows requests per second

3. Performance Considerations

Efficient dashboards minimize load on data sources and Grafana itself.

3.1. Query Optimization

Keep queries minimal and efficient. Avoid excessive aggregations or long time ranges in individual panels unless explicitly needed for historical context. Use group by sparingly.

3.2. Auto-Refresh Interval

Set auto-refresh to sensible defaults (e.g., 5min or 15min). Avoid smaller intervals (<1min) for non-alerting dashboards to prevent unnecessary load on the metric database. For immediate notification, use alerts.

{
  "refresh": "5m", // Recommended default
  "time": { "from": "now-6h", "to": "now" }
}

4. Common Pitfalls and Gotchas

Avoid these common mistakes that lead to confusing or inefficient dashboards.

4.1. Misleading Y-axis

Never allow the Y-axis to auto-scale or start above zero for percentage or count metrics. This distorts trends and makes small changes appear significant.

4.2. Overloaded Panels

A panel with too many metrics or complex queries becomes unreadable and slow. If you can't interpret a panel at a glance, it's overloaded. Break it down.

4.3. Lack of Context

Every dashboard must include a "Legend" text panel at the top. This panel should describe:

• The dashboard's purpose.
• The data flow and any significant transformations.
• Links to relevant documentation or source repositories.

{
  "title": "Legend",
  "type": "text",
  "gridPos": { "x": 0, "y": 0, "w": 24, "h": 2 },
  "options": {
    "content": "This dashboard monitors the **Payment Service** health. Metrics are collected via Prometheus exporters, aggregated every 15s. [Service Docs](https://internal.wiki/payment-service) | [Source Code](https://github.com/org/payment-service)",
    "mode": "markdown"
  }
}

4.4. Inconsistent Annotations

Standardize annotations for deployments and critical events. Configure dashboards to query common tags (e.g., deploy, incident).

{
  "annotations": {
    "list": [
      {
        "name": "Deployments",
        "datasource": "Grafana",
        "iconColor": "rgba(255, 96, 96, 1)",
        "enable": true,
        "query": { "tags": "deploy", "matchAny": true }
      }
    ]
  }
}

5. Testing Approaches

Treat Grafana configurations as production code, subject to automated testing and review.

5.1. Dashboard Linter Integration

Integrate grafana/dashboard-linter into your CI/CD pipeline. This tool automatically scans JSON definitions for common anti-patterns (e.g., missing titles, non-zero-based axes, excessive series).

# Example CI/CD step
- name: Lint Grafana Dashboards
  run: |
    go install github.com/grafana/dashboard-linter@latest
    dashboard-linter lint dashboards/*.json

5.2. Peer Review

Require peer review for all changes to Grafana configuration files (dashboards, alerts, data sources). This ensures adherence to standards and catches logical errors before deployment.

5.3. Automated Provisioning & Git Sync

Leverage Grafana's provisioning API for automated deployment. For Grafana 12+, explore and adopt Git Sync to manage dashboards directly from the UI with Git workflows (branches, PRs). This bridges the gap between UI-driven iteration and code-driven version control.