Dashboards As Code

Dashboards as Code

This skill is the entry point for the Materialize dashboards-as-code project. Stable conventions live in the repo docsite under docs/content/reference/internal/dashboard/ — this file is intentionally slim and links into the docsite at heading-level granularity. The non-link content below is the state snapshot: what currently exists, what's in flight, and what's queued for cleanup.

Audience reminder

The dashboards themselves target Materialize end users: database-literate operators with basic graph-reading fluency but minimal cloud / Kubernetes / observability expertise. SQL is fair game; jargon like "differential dataflow's arrangement" needs a one-liner explanation. Panel descriptions, titles, and cluster names should respect that baseline.

The docsite reference pages target repo contributors (SRE, Field Engineering, CloudOps, Database Engineers) and AI agents reading this skill.

Where to find what

Frequently needed deep links into the Style Guidelines:

• Tab-level theming
• Multi-select variables in regex contexts
• Sparkline stats
• Partitioned sparkline stats
• Writing panel descriptions
• Filtering by cluster / replica
• Materialize metric label families
• Logs dashboard conventions — Loki-discovered pickers, all_value rules, the non-empty-matcher anchor, and how exclusion switches are wired
• Time-range guards on expensive rows — the paired show/hide rows that keep volume panels off a month-wide range
• Kubernetes events in Loki — labels vs structured metadata, and why an event's namespace is the involved object's
• Deployment generations (blue/green) — how a generation reaches a query at all, and why the event filter needs its or arm
• orchestratord reconciliation metrics — the outcome vocabulary, and why abandoned is not a failure signal
• Known metric quirks and gotchas
• PromQL recipes
• Shared constants and helpers

And into Generating:

• PUT body shape — required Kubernetes-style envelope when pushing v2 dashboards via grafana_api_request
• Service account permissions — decoding 403s

Schema reference files

When uncertain about the exact shape Grafana expects, read the cog-generated JSON Schema (draft-07) documents vendored at packages/mzmon-lib/schemas/grafana/. The three dashboard documents there are:

• dashboard.jsonschema.json — v1
• dashboardv2beta1.jsonschema.json — v2beta1
• dashboardv2.jsonschema.json — v2

The other 52 documents in the same directory cover the panel, datasource, and common packages — that is where a panel's options and fieldConfig shapes live, not in the dashboard documents.

packages.json beside them maps each document to its Grafana plugin id — the value that goes in VizConfigKind.group or DataQueryKind.group. Read it rather than assuming the document name: annotationslist publishes as annolist.

The whole set is vendored from the grafana/grafana-foundation-sdk release tag v0.0.18 (June 12, 2026), generated there by cog v0.1.20. The v2 and v2beta1 documents track Grafana v13.0.2; v1 tracks Grafana v11.6.0. Re-vendor with bin/fetch-grafana-schemas.sh; Renovate maintains the tag pinned in that script, and PROVENANCE.md records the current pin.

Rust types are generated from these schemas into packages/mzmon-lib/src/grafana/generated/ by bin/gen-grafana-models.sh. See Rust models for why the layout is one module per document, and for the schema quirks that leak into any code built on them.

Current Dashboard State

This section captures the live state of the dashboards in this repo so the next session has something concrete to start from. Update it when state changes meaningfully (new dashboard, new tab, retired panel, theme reassignment).

Dashboard inventory

Each is rendered to charts/…/pre-rendered/dashboards/grafana/<stem>.yaml (chart) and docs/assets/dashboards/grafana/<stem>.json (docsite). One file per dashboard — there was a second, gcp- prefixed set until the clouds stopped differing in panel content, which left it recording nothing but its own name. The cloud render option, the --cloud / --prefix flags and the target-cloud annotation went with it.

env-upgrade is installed by default, because dashboards.selected defaults to ["env-*"] and the stem matches. While the operator-side instrumentation is unreleased it degrades unevenly, and the split is worth knowing: Generations works fully (every panel reads metrics that predate the change, and the blue/green split comes from pod names), Events keeps its Kubernetes Activity row, and Reconciliation is empty apart from its two pre-existing gauges. MIN_MZ_VERSION in env_upgrade/mod.rs is v26.41.0 and must stay in step with the Materialize row of docs/content/reference/compatibility.md. Narrow dashboards.selected to ["env-top"] to hold it back.

The live UID diverged from the codified one before mz-mon-env-top became authoritative — see UID selection and behavior.

env-top tabs

Six tabs, in declared order. Per-tab shades live in env_top/theme.rs — the source of truth, and deliberately the only place they are written down:

The Summary tab's CPU/memory capacity panels borrow the Kubernetes shade, and its Currently Hydrating panel is the same definition the Compute tab uses (env_top/mod.rs), with the shade as the only parameter.

Tab-by-tab row structure

Generated from the rendered artifact; regenerate rather than hand-editing when the layout changes.

Summary

1. Environment Health — Environment Status, Environment Availability (Select Time Range), Last Restart Time, Currently Hydrating, Max Lag (Select Time Range), Current CPU Usage (5 min), Current Memory Usage
2. Environment Info — Materialize Version, Total CPU Capacity, Total Memory

Kubernetes Workloads

1. Resources Summary (header hidden) — Total CPU Capacity, Total Memory
2. Workload Readiness (header hidden) — Pod Readiness, StatefulSet Readiness, Deployment Readiness
3. Pod Metrics — Pod CPU Usage, Pod Memory Usage
4. Pod Networking — Pod Network Rx, Pod Network Tx, Pod Network Errors, Pod Network Packet Drops

Connections / Activity

1. Connection Summary (header hidden) — Active Sessions, Active Queries, SQL Control Plane Command Rate
2. Queries — Query Distribution (by statement_type), Query Rate (by statement_type / session_type), Peek Latency (p50), Peek Latency (p90), Peek Latency (p99)
3. SQL Control Plane Commands — SQL Control Plane Commands by Application (one column: a wide table needs the room)

Cluster Objects / Replicas

1. Cluster Summary (header hidden) — Cluster Count, Replica Count
2. Replication / Availability — Replica Sizes
3. Cluster Information — Cluster Information

Compute Objects

1. Compute Objects Summary (header hidden) — Active Materialized Views, Active Indexes, Active Views, Active Subscribes, Index Relationship Types
2. Freshness — Freshness Lag by Cluster, Total Lag by Cluster, Most-Lagged Collections
3. Hydration — Currently Hydrating, Hydration Queue Size, Slowest Hydrating Collections
4. Dataflows — Dataflow Count, Dataflow Count (per worker), Dataflow Elapsed Rate
5. Arrangements — Arrangement Maintenance Rate, Arrangement Maintenance Rate (per worker), System / User / Transient Collections — Record Counts

Sources and Sinks

1. Storage Objects Summary (header hidden) — Active Sources, Active Sinks, Active Tables
2. Sources — Source Types, Sources, Source Bytes Received (rate), Source Ingestion by Replica, Source Upstream Errors
3. Sinks — Sink Types, Sink Throughput (committed), Sink Lag (staged minus committed)
4. Iceberg Sinks (collapsed) — Iceberg Commit Latency (p50 / p90 / p99), Iceberg Commit Failures & Conflicts, Iceberg File & Snapshot Rate
5. Kafka Sinks (collapsed) — Kafka TX Error Rate, Kafka Output Buffer (messages), Kafka Connect / Disconnect Rate

Replica AZs are intentionally unwired: materialize_cloud_availability_zone is cloud-only, and AZ semantics confuse the target audience.

env-upgrade tabs

Three tabs, ordered by descending altitude: what happened, which side of the rollout is ready, is the operator itself healthy. Shades come from env_upgrade/theme.rs.

This is the repo's first mixed-datasource dashboard. Events is Loki, Reconciliation is Thanos, and the two are separate tabs partly because they are scoped differently — see the namespace note below.

Events — the first tab in this repo built on Loki rather than Thanos. Rows narrow from verdict to cause:

1. Event Summary (header hidden) — Warning Events, Reconciliation Failures, Lifecycle Transitions
2. Rollout — Lifecycle Transitions (timeseries), Lifecycle Events (logs)
3. Operator Health — Reconciliation Failures (timeseries), Reconciliation Failure Events (logs)
4. Kubernetes Activity — Event Rate by Reason (timeseries), Warning Events (logs)
5. All Events (collapsed) — All Events (logs)

Each rate panel sits beside the feed it summarizes, in the same row: the chart says when, the feed says what.

Generations — the two sides of a blue/green rollout, split apart:

1. Rollout Status (header hidden) — Active Generations, Currently Hydrating, Worst-Case Lag, Pods
2. Versions — Version by Generation (table)
3. Hydration — Hydrating Collections by Generation, Collections by Generation
4. Freshness — Worst-Case Lag by Generation, Total Lag by Generation, Total Lag by Generation and Cluster
5. Footprint — CPU by Generation, Memory by Generation

Version by Generation is the row that says what the rollout is for. It reads the mz_version label off compute_cluster_status, which each generation's own environmentd reports, so the two sides genuinely disagree during a rollout — over a window spanning one, the table reads gen 2 → v26.38.2 beside gen 3 → v26.40.0-rc.1. A table rather than a stat because the value is a string and a stat cannot show two of those legibly. Two rows with the same version means a forced rollout rather than an upgrade, which is worth confirming before it costs a rehydration.

Reconciliation — the operator's control loop, as counters and histograms:

1. Operator Status (header hidden) — Reconciling Replicas, Environments Needing Update, Reconciliation Rate, Failed Passes (Select Time Range)
2. Reconciliation Passes — Pass Outcomes, Failed Passes by Controller
3. Duration — Pass Duration (p50/p90/p99), Step Duration (p99)
4. Steps — Step Activity, Step Failures and Abandonments

env-logs tabs

Two tabs, shaded from env_logs/theme.rs. Events deliberately reuses the shade it carries on env-upgrade, since it is the same kind of content — an operator moving between the two is not told otherwise.

Logs — Volume (Log Rate, Warning Rate, Log Rate by App, Log Rate by Level), Warnings (feed), All Logs (feed). Events — Activity (rate by reason, rate by namespace), Warnings (feed), All Events (feed).

infra-logs tabs

The first of the infra-* family — scoped to the cluster rather than to an environment.

Logs — Volume (rate by component, rate by namespace, warning rate), Warnings feed, All Logs feed. Nodes — Journal Volume (rate by unit), Node Warnings feed, Node Journal feed. Events — Activity (by reason, by namespace), Warnings feed, All Events feed.

Why it is a second dashboard rather than a wider env-logs

Two things env-logs cannot reach however its pickers are set:

• The node journal. Journal lines carry unit, component, job, level and service_name and no namespace, app or container — they come from the node, not a pod. Every env-logs selector requires a namespace, so those lines are excluded by construction. unit is their anchor, with all_value .+, standing in for the namespace matcher container-log selectors lean on.
• Sub-components. component splits loki into eight processes (canary, querier, ingester, query-frontend, index-gateway, compactor, distributor, ruler) and thanos into three. A Materialize environment has none, so adding the picker there would be a control that does nothing.

A third, smaller reason: container is the only picker that reaches workloads with no app label, which on a representative install is the whole of kube-system (14 containers, app empty). It sits in the controls menu.

What the two dashboards share

The variable names and the Kubernetes-event queries. materialize.events.cluster.* carries no Materialize-specific filter and is scoped by the same $logNamespaceList both dashboards define, so the events half is one set of definitions serving both. log_namespaces(opens_on) takes the opening selection as an argument — the Materialize pattern for env-logs, .+ for infra-logs — which is the only intended difference.

The container-log queries are not shared: infra.logs.* carries the component and container filters, and adding those to materialize.logs.* would oblige env-logs to define pickers it has no use for.

infra-nodes tabs

One node at a time — the picker is single-select with no "All", because a machine's measurements averaged across a fleet make every panel ambiguous. Fleet views are a separate dashboard and a different question (which node).

Seven tabs against a seven-colour palette — this dashboard uses the whole of it, and an eighth would have to repeat.

Summary — Node Info (identity and capacity as info cells), Utilization (CPU / memory / swap / rx / tx sparklines), Allocation (radial gauges for requested CPU, memory and pod slots), Scheduling and Conditions (Ready, cordon state, pressure conditions, taints, pods by namespace). The audience is an operator who cannot run kubectl describe node, deciding whether to escalate — so the panel descriptions say which readings are worth handing to an infrastructure team.

Pods — Pod Health (by phase, not-ready table, container restarts), Requests and Limits (one table: CPU and memory, requests beside limits, one row per pod). Requests are what the scheduler set aside; limits are what the kernel enforces, and the comparison that matters is a pod's request against its own limit — across four separate tables that means holding pod names in your head. A pod that sets no limit leaves those cells empty rather than zero, which is the honest reading.

Everything on that tab except the restart graph is instant: true. These are current facts; evaluated over a range each pod repeats once per scrape and the table becomes unreadable. That is the same trap the env-upgrade version table documents from the other side, where a plain instant query would have hidden a torn-down generation.

Logs & Events leads with events, not the journal: the events are few, dated, and each one a decision, so they are usually what the reader came for. The journal is where the explanation is once an event has said where to look.

Two names for one machine

The identifier join this dashboard rests on — and the vetting status of the node query families — is a convention rather than state, so it lives in the style guide.

Notes on the trickier panels

• Freshness reads mz_dataflow_wallclock_lag_seconds. Collections with no established frontier report a u64::MAX (~1.8e19) sentinel, filtered with < 1e9; the metric is a summary carrying quantile 0/1 only, so take 1 for worst-case. Those excluded collections are what Currently Hydrating counts. Max and total answer different questions and both are drawn. The max is pinned to whichever single collection is furthest behind, so it barely moves while the rest of a rehydrating cluster converges; the sum falls with every collection that catches up, which is the descent an operator watches before promoting a generation. Any sum of this metric has to dedupe replicas first — sum by (…) (max by (…, collection_id) (…)) — because a collection served by two replicas reports twice, and two replicas is the normal shape mid-rollout.
• Source Ingestion by Replica is a divergence detector: replicas read upstream independently, so one flat at 0 while its siblings ingest has lost its connection — the aggregate throughput panel hides that.
• Source Upstream Errors pairs a commit-failure rate with an offset_committed > offset_known disconnect indicator, because the broker-unreachable case never reaches the commit step.

Self-managed metric migration (done)

Migrated off the cloud-only v2_mz_* family and materialize_cloud_organization_id onto self-managed mz_* metrics and materialize_cloud_organization_name, with SQL-derived metrics converged behind $sqlMetricPrefix. The roadmap records it as shipped; every rule that came out of it — the wallclock-lag sentinel behind Currently Hydrating, duplicate-job dedup on the shared :6878 endpoint, the datasource timeInterval that empties rate() panels, and the prefix rules themselves — lives in the style guide, which is where to look rather than here.

Local push: gcx context local-mzmon → http://localhost:13000. Render with mz-monitoring-build gen-dashboards --format json, then carry the live resourceVersion + folder annotation forward on the PUT — see PUT body shape. The Grafana MCP is wired to the same local instance for query verification.

Reference environments

Materialize developers may have access to an internal shared Grafana with multiple test environments. It can be useful to look at queries in live environments when building dashboards. Do not use environments without explicit permission.

When testing against a cloud shared env, scope queries to one environment so you don't mix data across tenants. The dashboards target self-managed Materialize, where the scoping label is materialize_cloud_organization_name="..." (cloud's hex materialize_cloud_organization_id does not exist on self-managed, and neither does the v2_mz_* metric family). Always verify which labels/metrics actually exist on the instance you're querying with list_prometheus_label_names / list_prometheus_metric_names before assuming — see Deployment target: self-managed vs cloud .

Cleanup / refactor candidates

Tracked items that are working but could be tidier:

• Hydration is SQL-only on self-managed. No Prometheus metric exposes per-collection hydration state or time (v2_mz_compute_hydration_time_seconds is cloud-only; mz_compute_controller_hydration_queue_size is just the controller queue and reads 0 even with many objects mid-hydration). Slowest Hydrating Collections stays backed by the cloud metric (blank here) with a description pointing at mz_internal.mz_hydration_statuses / mz_compute_hydration_times; the live metric-side proxy is the Freshness row (wallclock_lag).
• The mz-mon- UID prefix is not validated. One dashboard, consistent today. Worth a check if more land.
• packages/ref-alloy-pipelines/ Python is dead. It imports py_mzmon_lib.alloy.config_dsl, a module that no longer exists; the .alloy files beside it are still the behavioral porting reference. Unrelated to the dashboards, but it is the last Python in packages/.

Resolved by the Rust port, listed so they are not re-filed: the duplicated ENV_SCOPED_NOTE and long-form cluster filter (prose lives on the registry query, filters are parameters), and the orphaned dataflows.py.