Grafana Dashboard Review
When to Use
- Building dashboards for a data platform (Airflow, Kafka, Spark, dbt)
- Reviewing existing dashboards for usability and performance issues
- Setting up dashboard-as-code (version-controlled JSON or Grafonnet)
- Adding deployment annotations to correlate incidents with releases
- Creating standardized dashboard templates for multiple environments
Data Platform Overview Dashboard
Panel Structure
Row 1: Pipeline Health (last 24h)
├── Stat: DAG success rate %
├── Stat: Active task instances
├── Stat: Failed DAG runs (last 1h)
└── Stat: SLA misses today
Row 2: Pipeline Throughput
├── Timeseries: Task completions per minute by DAG
├── Timeseries: Task duration P50/P95 by task_id
└── Heatmap: Task duration distribution
Row 3: Infrastructure
├── Timeseries: Pool slot utilization (default_pool)
├── Timeseries: Scheduler task queue depth
└── Gauge: Worker CPU/memory utilization
Variable Templating
{
"templating": {
"list": [
{
"name": "datasource",
"type": "datasource",
"query": "prometheus",
"label": "Datasource"
},
{
"name": "namespace",
"type": "query",
"datasource": "${datasource}",
"query": "label_values(kube_namespace_labels, namespace)",
"label": "Namespace",
"multi": true,
"includeAll": true,
"allValue": ".*"
},
{
"name": "dag_id",
"type": "query",
"datasource": "${datasource}",
"query": "label_values(airflow_task_success_total{namespace=~\"$namespace\"}, dag_id)",
"label": "DAG",
"multi": true,
"includeAll": true
},
{
"name": "environment",
"type": "custom",
"options": [
{"text": "Production", "value": "prod"},
{"text": "Staging", "value": "staging"}
],
"label": "Environment"
}
]
}
}
Key Panel Configurations
Pipeline Success Rate (Stat Panel)
{
"type": "stat",
"title": "Pipeline Success Rate (24h)",
"targets": [{
"expr": "100 * sum(increase(airflow_task_success_total{dag_id=~\"$dag_id\"}[24h])) / (sum(increase(airflow_task_success_total{dag_id=~\"$dag_id\"}[24h])) + sum(increase(airflow_task_failures_total{dag_id=~\"$dag_id\"}[24h])))",
"legendFormat": "Success Rate"
}],
"options": {
"reduceOptions": {"calcs": ["lastNotNull"]},
"colorMode": "background",
"thresholds": {
"steps": [
{"color": "red", "value": 0},
{"color": "yellow", "value": 95},
{"color": "green", "value": 99}
]
}
}
}
Task Duration Heatmap
{
"type": "heatmap",
"title": "Task Duration Distribution",
"targets": [{
"expr": "sum(rate(airflow_task_duration_seconds_bucket{dag_id=~\"$dag_id\"}[5m])) by (le)",
"format": "heatmap",
"legendFormat": "{{le}}"
}],
"options": {
"calculate": false,
"yAxis": {
"unit": "s",
"decimals": 0
}
}
}
Kafka Lag Table
{
"type": "table",
"title": "Kafka Consumer Lag",
"targets": [{
"expr": "sum(kafka_consumergroup_lag) by (consumergroup, topic)",
"legendFormat": "",
"instant": true,
"format": "table"
}],
"transformations": [
{"id": "organize", "options": {
"renameByName": {
"consumergroup": "Consumer Group",
"topic": "Topic",
"Value": "Lag"
}
}},
{"id": "sortBy", "options": {
"fields": [{"displayName": "Lag", "desc": true}]
}}
],
"fieldConfig": {
"overrides": [{
"matcher": {"id": "byName", "options": "Lag"},
"properties": [{
"id": "thresholds",
"value": {
"steps": [
{"color": "green", "value": 0},
{"color": "yellow", "value": 1000},
{"color": "red", "value": 10000}
]
}
}, {
"id": "custom.displayMode",
"value": "color-background"
}]
}]
}
}
Deployment Annotations
# Annotate Grafana with deployment events (via API)
GRAFANA_URL="http://grafana.monitoring.svc:3000"
GRAFANA_TOKEN="${GRAFANA_SA_TOKEN}"
annotate_deployment() {
local dag_id=$1
local version=$2
curl -s -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${GRAFANA_TOKEN}" \
-d "{
\"dashboardUID\": \"data-platform\",
\"time\": $(date +%s000),
\"text\": \"Deploy: ${dag_id} v${version}\",
\"tags\": [\"deploy\", \"${dag_id}\"]
}" \
"${GRAFANA_URL}/api/annotations"
}
# Call from CI/CD after successful deploy
annotate_deployment "etl_orders" "$IMAGE_TAG"
# GitHub Actions step: annotate on deploy
- name: Annotate Grafana
run: |
curl -s -X POST \
-H "Authorization: Bearer ${{ secrets.GRAFANA_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"text": "Deploy: ${{ github.repository }}@${{ github.sha }}", "tags": ["deploy"]}' \
${{ secrets.GRAFANA_URL }}/api/annotations
Dashboard-as-Code (Terraform)
# Manage Grafana dashboards via Terraform
resource "grafana_dashboard" "data_platform" {
config_json = file("${path.module}/dashboards/data-platform.json")
folder = grafana_folder.data_engineering.id
overwrite = true
}
resource "grafana_folder" "data_engineering" {
title = "Data Engineering"
uid = "data-engineering"
}
# Alert rule as code
resource "grafana_rule_group" "pipeline_alerts" {
name = "Pipeline Alerts"
folder_uid = grafana_folder.data_engineering.uid
interval_seconds = 60
rule {
name = "Pipeline Freshness"
condition = "C"
for = "10m"
labels = { severity = "warning", team = "data-engineering" }
annotations = {
summary = "Pipeline {{ $labels.dag_id }} data is stale"
runbook = "https://wiki.my-org.com/runbooks/pipeline-freshness"
}
data {
ref_id = "A"
query_type = "range"
datasource_uid = grafana_data_source.prometheus.uid
model = jsonencode({
expr = "time() - max(airflow_dagrun_end_date{state='success'}) by (dag_id)"
})
}
data {
ref_id = "C"
model = jsonencode({ type = "threshold", conditions = [{ evaluator = { type = "gt", params = [7200] } }] })
}
}
}
Dashboard Performance Optimization
// Use recording rules instead of complex PromQL in panels
// ❌ Slow: complex query re-evaluated per panel load
{
"expr": "sum(rate(airflow_task_failures_total[5m])) by (dag_id) / (sum(rate(airflow_task_success_total[5m])) by (dag_id) + sum(rate(airflow_task_failures_total[5m])) by (dag_id))"
}
// ✅ Fast: precomputed recording rule
{
"expr": "job_dag:pipeline_error_rate:ratio5m"
}
// Limit default time range to avoid full-scan queries
{
"time": { "from": "now-6h", "to": "now" },
"timepicker": {
"refresh_intervals": ["30s", "1m", "5m"],
"time_options": ["1h", "3h", "6h", "12h", "24h", "2d", "7d"]
}
}
Dashboard Review Checklist
[ ] All variable dropdowns use label_values() — not hardcoded
[ ] Time range defaults to 6h or 24h (not 90d)
[ ] Panels use recording rules for complex queries
[ ] Stat panels have color thresholds (green/yellow/red)
[ ] Table panels have cell coloring on key columns
[ ] Deployment annotations configured in CI/CD
[ ] Dashboard UID set (not auto-generated) for stable links
[ ] Folder structure: Data Engineering / Airflow / Kafka / Spark
[ ] Dashboard JSON committed to git (Terraform or ConfigMap)
[ ] Links between dashboards: overview → detail drilldown
[ ] Description field populated on dashboard and key panels
Anti-Patterns
- No variable templating — hardcoded namespace/dag_id means dashboard only works for one environment; use
label_values() variables.
- Complex PromQL in every panel — 20 panels × complex query = slow dashboard and high Prometheus load; precompute with recording rules.
- Default time range of 30 days — loads millions of data points on open; set default to 6h or 24h.
- Dashboard managed only in Grafana UI — no version history, no peer review; manage dashboards as JSON in git.
- No color thresholds on key metrics — a success rate of 60% looks the same as 99.9% without colors; always add threshold steps.
References
- Grafana panel types:
grafana.com/docs/grafana/latest/panels-visualizations/
- Terraform Grafana provider:
registry.terraform.io/providers/grafana/grafana/latest/docs
- Grafana annotations API:
grafana.com/docs/grafana/latest/http_api/annotations/
- Related skills:
[[infra-observability-stack-review]], [[infra-prometheus-optimization]], [[dataops-airflow-observability]]
ivanshamaev