Building Dashboards

Building Dashboards

You design dashboards that help humans make decisions quickly. Dashboards are products: audience, questions, and actions matter more than chart count.

Philosophy

1. Decisions first. Every panel answers a question that leads to an action.
2. Overview → drilldown → evidence. Start broad, narrow on click/filter, end with raw logs.
3. Rates and percentiles over averages. Averages hide problems; p95/p99 expose them.
4. Simple beats dense. One question per panel. No chart junk.
5. Validate with data. Never guess fields—discover schema first.

Entry Points

Choose your starting point:

Intake: What to Ask First

Before designing, clarify:

1. Audience & decision

   • Oncall triage? (fast refresh, error-focused)
   • Team health? (daily trends, SLO tracking)
   • Exec reporting? (weekly summaries, high-level)

2. Scope

   • Service, environment, region, cluster, endpoint?
   • Single service or cross-service view?

3. Dataset kind (mandatory first step)

   • Run scripts/metrics/datasets <deploy> to identify each dataset's kind
   • If kind is otel:metrics:v1 → this is a metrics dataset. Follow the Metrics path below.
   • Otherwise → this is an events/logs dataset. Follow the APL path below.

   ⚠️ NEVER run getschema on a metrics dataset. APL queries against otel:metrics:v1 datasets return 0 rows without error — you will waste calls widening time ranges before realizing it's the wrong discovery method.

   APL path (events/logs datasets):

   • Discover fields with getschema:

   ['dataset'] | where _time between (ago(1h) .. now()) | getschema
   

   • Continue to steps 4–5 below.

   Metrics path (otel:metrics:v1 datasets):

   • Run scripts/metrics/metrics-spec <deploy> <dataset> — mandatory before composing any MPL query
   • Discover available metrics: scripts/metrics/metrics-info <deploy> <dataset> metrics
   • Discover tags: scripts/metrics/metrics-info <deploy> <dataset> tags
   • Explore tag values: scripts/metrics/metrics-info <deploy> <dataset> tags <tag> values
   • If discovery returns empty results, retry with --start set to 7 days ago — sparse metrics (sensors, batch jobs, crons) may not have data in the default 24h window
   • find-metrics <value> searches tag values, not metric names — use it only when you know a specific entity name (service, host, device) to find which metrics are associated with it
   • Skip to the Metrics/MPL Blueprint below for panel design.

4. Golden signals (APL path)

   • Traffic: requests/sec, events/min
   • Errors: error rate, 5xx count
   • Latency: p50, p95, p99 duration
   • Saturation: CPU, memory, queue depth, connections

5. Drilldown dimensions (APL path)

   • What do users filter/group by? (service, route, status, pod, customer_id)

Dashboard Blueprint

Choose the blueprint that matches your dataset kind (identified in Intake step 3).

APL Blueprint (events/logs datasets)

1. At-a-Glance (Statistic panels)

Single numbers that answer "is it broken right now?"

• Error rate (last 5m)
• p95 latency (last 5m)
• Request rate (last 5m)
• Active alerts (if applicable)

2. Trends (TimeSeries panels)

Time-based patterns that answer "what changed?"

• Traffic over time
• Error rate over time
• Latency percentiles over time
• Stacked by status/service for comparison

3. Breakdowns (Table/Pie panels)

Top-N analysis that answers "where should I look?"

• Top 10 failing routes
• Top 10 error messages
• Worst pods by error rate
• Request distribution by status

4. Evidence (LogStream + SmartFilter)

Raw events that answer "what exactly happened?"

• LogStream filtered to errors
• SmartFilter for service/env/route
• Key fields projected for readability

Metrics/MPL Blueprint (metrics datasets)

Prerequisite: You MUST have run scripts/metrics/metrics-spec and scripts/metrics/metrics-info before designing panels. Never guess MPL syntax or metric/tag names.

🚨 ALIGNMENT RULE — non-negotiable for dashboard panels: Always align to the dashboard-supplied variable $__interval, not a fixed window. The dashboard runtime substitutes $__interval based on the active time range and panel width, so the same chart stays usable from a 5-minute to a 30-day view. Hard-coding align to 1m (or any constant) over-resolves long ranges and under-resolves short ones.

| align to $__interval using avg   ✅ dashboard panels
| align to 1m using avg            ❌ fixed window — wrong granularity at most time ranges

No param declaration needed in the chart query.apl — the dashboard runtime injects param $__interval: Duration; automatically. (The Grafana datasource does the same via a preamble; the Axiom-native dashboard runtime behaves identically — verified against working production dashboards.)

Exceptions: If you are pre-validating a query through scripts/metrics/metrics-query (which has no dashboard runtime), substitute a concrete duration for the test call only — do NOT commit that to the chart JSON. For genuinely sparse metrics where $__interval would round to an empty bucket (sensors, batch jobs, crons), a fixed wider window (e.g. 1h) is acceptable; document why in the chart description.

1. At-a-Glance (Statistic panels)

Current values for key metrics — answer "what's the state right now?"

• Latest value of primary metrics (e.g., current temperature, power draw)
• Use group using avg or group using last depending on metric type (gauge vs counter)

2. Trends (TimeSeries panels)

Metric trends over time — answer "what changed?"

• Primary metrics over time, grouped by key dimension
• Use align to $__interval using avg|sum|last for proper time bucketing — $__interval is supplied by the dashboard runtime
• Group by low-cardinality tags only (≤10 series per chart)

3. Breakdowns (TimeSeries or Table panels)

Per-entity detail — answer "where should I look?"

• Metrics broken down by entity (room, host, pod, service)
• Filter by tag values to keep series count manageable
• Use separate panels per dimension rather than one overloaded chart

4. Entity State (TimeSeries or Table panels)

Boolean/state metrics — answer "what is on/off/active?"

• Use align to $__interval using last for state metrics
• Sparse metrics may need wider fixed align intervals (1h+) to show data — this is the documented exception to the $__interval rule

Layout Auto-Normalization

The console uses react-grid-layout which requires minH, minW, moved, and static on every layout entry. The dashboard-create and dashboard-update scripts auto-fill these if omitted, so layout entries only need i, x, y, w, h.

Required Chart Structure

Every chart MUST have a unique id field. Every layout entry's i field MUST reference a chart id. Missing or mismatched IDs will corrupt the dashboard in the UI (blank state, unable to save/revert).

{
  "charts": [
    {
      "id": "error-rate",
      "name": "Error Rate",
      "type": "Statistic",
      "query": { "apl": "..." }
    }
  ],
  "layout": [
    {"i": "error-rate", "x": 0, "y": 0, "w": 3, "h": 2}
  ]
}

Use descriptive kebab-case IDs (e.g. error-rate, p95-latency, traffic-rps). The dashboard-validate and deploy scripts enforce this automatically.

Metrics/MPL Chart Contract

Metrics-backed charts require both query.apl (the MPL pipeline string) and query.metricsDataset (the dataset name). The metricsDataset field is what tells the backend to interpret apl as MPL rather than APL — omitting it causes the chart to misbehave even if the pipeline string is well-formed.

CRITICAL: Run scripts/metrics/metrics-spec <deployment> <dataset> before composing your first MPL query in a session. NEVER guess MPL syntax.

API gotcha: Set query.metricsDataset to the dataset name (e.g. "otel-metrics"). The create API rejects query.mpl even though GET responses for existing metrics dashboards may include it — put the MPL string in query.apl instead.

{
  "type": "TimeSeries",
  "query": {
    "apl": "`otel-metrics`:`http.server.duration`\n| where `service.name` == \"api\"\n| align to $__interval using avg\n| group by `service.name` using avg",
    "metricsDataset": "otel-metrics"
  }
}

Validate queries with scripts/metrics/metrics-query before embedding in dashboard JSON.

See reference/metrics-mpl.md for the full contract and discovery scripts.

Chart Types

Note: Dashboard queries inherit time from the UI picker—no explicit _time filter needed.

Validation: TimeSeries, Statistic, Table, Pie, LogStream, Note, MonitorList are fully validated by dashboard-validate. Heatmap, ScatterPlot, SmartFilter work but may trigger warnings.

Statistic

When: Single KPI, current value, threshold comparison.

['logs']
| where service == "api"
| summarize 
    total = count(),
    errors = countif(status >= 500)
| extend error_rate = round(100.0 * errors / total, 2)
| project error_rate

Pitfalls: Don't use for time series; ensure query returns single row.

TimeSeries

When: Trends over time, before/after comparison, rate changes.

// Single metric - use bin_auto for automatic sizing
['logs']
| summarize ['req/min'] = count() by bin_auto(_time)

// Latency percentiles - use percentiles_array for proper overlay
['logs']
| summarize percentiles_array(duration_ms, 50, 95, 99) by bin_auto(_time)

Best practices:

• Use bin_auto(_time) instead of fixed bin(_time, 1m) — auto-adjusts to time window
• Use percentiles_array() instead of multiple percentile() calls — renders as one chart
• Too many series = unreadable; use top N or filter

Table

When: Top-N lists, detailed breakdowns, exportable data.

['logs']
| where status >= 500
| summarize errors = count() by route, error_message
| top 10 by errors
| project route, error_message, errors

Pitfalls:

• Always use top N to prevent unbounded results
• Use project to control column order and names

Pie

When: Share-of-total for LOW cardinality dimensions (≤6 slices).

['logs']
| summarize count() by status_class = case(
    status < 300, "2xx",
    status < 400, "3xx",
    status < 500, "4xx",
    "5xx"
  )

Pitfalls:

• Never use for high cardinality (routes, user IDs)
• Prefer tables for >6 categories
• Always aggregate to reduce slices

LogStream

When: Raw event inspection, debugging, evidence gathering.

['logs']
| where service == "api" and status >= 500
| project-keep _time, trace_id, route, status, error_message, duration_ms
| take 100

Pitfalls:

• Always include take N (100-500 max)
• Use project-keep to show relevant fields only
• Filter aggressively—raw logs are expensive

Heatmap

When: Distribution visualization, latency patterns, density analysis.

['logs']
| summarize histogram(duration_ms, 15) by bin_auto(_time)

Best for: Latency distributions, response time patterns, identifying outliers.

Scatter Plot

When: Correlation between two metrics, identifying patterns.

['logs']
| summarize avg(duration_ms), avg(resp_size_bytes) by route

Best for: Response size vs latency correlation, resource usage patterns.

SmartFilter (Filter Bar)

When: Interactive filtering for the entire dashboard.

SmartFilter is a chart type that creates dropdown/search filters. Requires:

1. A SmartFilter chart with filter definitions
2. declare query_parameters in each panel query

Filter types:

• selectType: "apl" — Dynamic dropdown from APL query
• selectType: "list" — Static dropdown with predefined options
• type: "search" — Free-text input

Panel query pattern:

declare query_parameters (country_filter:string = "");
['logs'] | where isempty(country_filter) or ['geo.country'] == country_filter

See reference/smartfilter.md for full JSON structure and cascading filter examples.

Monitor List

When: Display monitor status on operational dashboards.

No APL needed—select monitors from the UI. Shows:

• Monitor status (normal/triggered/off)
• Run history (green/red squares)
• Dataset, type, notifiers

Note

When: Context, instructions, section headers.

Use GitHub Flavored Markdown for:

• Dashboard purpose and audience
• Runbook links
• Section dividers
• On-call instructions

Chart Configuration

Charts support JSON configuration options beyond the query. See reference/chart-config.md for full details.

Quick reference:

Common options (all charts):

• overrideDashboardTimeRange: boolean
• overrideDashboardCompareAgainst: boolean
• hideHeader: boolean

APL Patterns

Time Filtering in Dashboards vs Ad-hoc Queries

Dashboard panel queries do NOT need explicit time filters. The dashboard UI time picker automatically scopes all queries to the selected time window.

// DASHBOARD QUERY — no time filter needed
['logs']
| where service == "api"
| summarize count() by bin_auto(_time)

Ad-hoc queries (Axiom Query tab, axiom-sre exploration) MUST have explicit time filters:

// AD-HOC QUERY — always include time filter
['logs']
| where _time between (ago(1h) .. now())
| where service == "api"
| summarize count() by bin_auto(_time)

Bin Size Selection

Prefer bin_auto(_time) — it automatically adjusts to the dashboard time window.

Manual bin sizes (only when auto doesn't fit your needs):

Cardinality Guardrails

Prevent query explosion:

// GOOD: bounded
| summarize count() by route | top 10 by count_

// BAD: unbounded high-cardinality grouping
| summarize count() by user_id  // millions of rows

Field Escaping

Fields with dots need bracket notation:

| where ['kubernetes.pod.name'] == "frontend"

Fields with dots IN the name (not hierarchy) need escaping:

| where ['kubernetes.labels.app\\.kubernetes\\.io/name'] == "frontend"

Golden Signal Queries

Traffic:

| summarize requests = count() by bin_auto(_time)

Errors (as rate %):

| summarize total = count(), errors = countif(status >= 500) by bin_auto(_time)
| extend error_rate = iff(total > 0, round(100.0 * errors / total, 2), 0.0)
| project _time, error_rate

Latency (use percentiles_array for proper chart overlay):

| summarize percentiles_array(duration_ms, 50, 95, 99) by bin_auto(_time)

Layout Composition

Grid Principles

• Dashboard width = 12 units
• Typical panel: w=3 (quarter), w=4 (third), w=6 (half), w=12 (full)
• Stats row: 4 panels × w=3, h=2
• TimeSeries row: 2 panels × w=6, h=4
• Tables: w=6 or w=12, h=4–6
• LogStream: w=12, h=6–8

Section Layout Pattern

Row 0-1:  [Stat w=3] [Stat w=3] [Stat w=3] [Stat w=3]
Row 2-5:  [TimeSeries w=6, h=4] [TimeSeries w=6, h=4]
Row 6-9:  [Table w=6, h=4] [Pie w=6, h=4]
Row 10+:  [LogStream w=12, h=6]

Naming Conventions

• Use question-style titles: "Error rate by route" not "Errors"
• Prefix with context if multi-service: "[API] Error rate"
• Include units: "Latency (ms)", "Traffic (req/s)"

Dashboard Settings

Refresh Rate

Dashboard auto-refreshes at configured interval. Options: 15s, 30s, 1m, 5m, etc.

⚠️ Query cost warning: Short refresh (15s) + long time range (90d) = expensive queries running constantly.

Recommendations:

Sharing

All dashboards created via API tokens are shared with everyone in the org (owner: "X-AXIOM-EVERYONE"). Private dashboards are not supported with API tokens.

Data visibility is still governed by dataset permissions—users only see data from datasets they can access.

URL Time Range Parameters

?t_qr=24h (quick range), ?t_ts=...&t_te=... (custom), ?t_against=-1d (comparison)

Setup

Run scripts/setup to check requirements (curl, jq, ~/.axiom.toml).

Config in ~/.axiom.toml (shared with axiom-sre):

[deployments.prod]
url = "https://api.axiom.co"
token = "xaat-your-token"
org_id = "your-org-id"

Deployment

Scripts

⚠️ Two axiom-api scripts exist with different behaviors. scripts/axiom-api rewrites URLs for the dashboard app API (app.*). scripts/metrics/axiom-api uses raw URLs and supports edge deployment routing. Using the wrong one will produce 404 errors.

Targeted Chart Updates

Use scripts/dashboard-chart-patch when changing one existing chart and the dashboard layout, metadata, and other charts should remain untouched. It calls PATCH /v2/dashboards/uid/{uid}/charts/{chartId} with a JSON Merge Patch under the chart request field.

Patch files contain only the chart fields to change:

{
  "name": "Error Rate (5m)",
  "query": { "apl": "['logs'] | summarize errors=countif(status >= 500)" },
  "config": { "stale": null }
}

null removes an existing field. Nested objects merge recursively. If id is present in the patch, it must match the <chart-id> path argument. The server validates the resulting full dashboard before saving.

Use --version <version> for optimistic concurrency after fetching the dashboard with dashboard-get. Use --overwrite only when last-write-wins behavior is intended. Continue using dashboard-update for layout changes, multi-chart edits, dashboard metadata, owner, refresh interval, or time window updates.

Workflow

⚠️ CRITICAL: Always validate queries BEFORE deploying.

APL workflow:

1. Design dashboard (sections + panels)
2. Write APL for each panel
3. Build JSON (from template or manually)
4. Validate queries using axiom-sre with explicit time filter
5. dashboard-validate to check structure
6. dashboard-create or dashboard-update to deploy
7. dashboard-link to get URL — NEVER construct Axiom URLs manually (org IDs and base URLs vary per deployment)
8. Share link with user

Metrics/MPL workflow:

1. Run scripts/metrics/metrics-spec to learn MPL syntax
2. Run scripts/metrics/metrics-info to discover metrics and tags
3. Design dashboard using the Metrics/MPL Blueprint
4. Write MPL for each panel
5. Validate queries with scripts/metrics/metrics-query using explicit time range
6. Build JSON: put the full MPL string in query.apl AND set query.metricsDataset to the dataset name (required — denotes the chart as MPL). Do not set query.mpl (rejected by create API).
7. dashboard-validate to check structure
8. dashboard-create or dashboard-update to deploy
9. dashboard-link to get URL
10. Share link with user

Sibling Skill Integration

spl-to-apl: Translate Splunk SPL → APL. Map timechart → TimeSeries, stats → Statistic/Table. See reference/splunk-migration.md.

axiom-sre: Discover schema with getschema, explore baselines, identify dimensions, then productize into panels.

query-metrics: Discover metrics datasets, metric names, tags, and tag values. Metrics discovery scripts are also vendored locally in scripts/metrics/.

Templates

Pre-built templates in reference/templates/:

Placeholders: {{service}}, {{dataset}}

Usage:

scripts/dashboard-from-template service-overview "my-service" "my-dataset" ./dashboard.json
scripts/dashboard-validate ./dashboard.json
scripts/dashboard-create prod ./dashboard.json

⚠️ Templates assume field names (service, status, route, duration_ms). Discover your schema first and use sed to fix mismatches.

Common Pitfalls

Reference

• reference/chart-config.md — All chart configuration options (JSON)
• reference/metrics-mpl.md — Metrics/MPL chart contract and discovery scripts
• reference/smartfilter.md — SmartFilter/FilterBar full configuration
• reference/chart-cookbook.md — APL patterns per chart type
• reference/layout-recipes.md — Grid layouts and section blueprints
• reference/splunk-migration.md — Splunk panel → Axiom mapping
• reference/design-playbook.md — Decision-first design principles
• reference/templates/ — Ready-to-use dashboard JSON files

For APL syntax: https://axiom.co/docs/apl/introduction

Source: pinchbench/skill — distributed by TomeVault.