# Datadog Dashboards

> Datadog Dashboards

- Skill: `futuregerald/datadog-dashboards` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add futuregerald/datadog-dashboards`
- Raw SKILL.md: https://api.skillmd.com/api/skills/futuregerald/datadog-dashboards/raw
- Safety review: pending (external: skill-scanner FAIL, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: futuregerald (https://skillmd.com/u/futuregerald)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/futuregerald/datadog-dashboards

---


# Datadog Dashboards

## Core principle

A dashboard is correct only when it **renders what the user wants against real
data**. Valid JSON and correct-looking queries are necessary but not sufficient.
The failure modes that matter — empty tiles, blank cells, numbers that mean
something other than the title claims — are **data-dependent and invisible in
static review**. Never declare a dashboard done from inspection alone. Verify
against live data.

This skill exists because static/LLM review reliably catches schema, field-name,
and consistency bugs and reliably **misses** runtime data behavior (sparse joins,
empty facets, count-vs-cardinality drift, no-data-vs-zero). Only running the
queries closes that gap.

## The three gates (all required, in order)

**Gate 1 — Structure.** JSON parses; every widget is well-formed.
Run `scripts/validate_dashboard.py <file>`. Fix all errors; weigh warnings.

**Gate 2 — Data.** Every query returns what you expect, confirmed by **running it
against Datadog** (MCP `search_datadog_logs` / `analyze_datadog_logs` / metrics,
or the UI). Field names, facets, and measures must exist in the *actual emitted
events* — not in a PR description, doc, or memory.
Use `scripts/extract_queries.py <file>` to get the per-widget checklist.

**Gate 3 — Intent.** For each section and each widget, ask *"what question does
this answer, and will it render that?"* A widget can pass Gates 1–2 and still
answer the wrong question or mislead.

Do not tell the user a dashboard is ready until all three gates pass. If Gate 2
is impossible (no data access), say so explicitly — do not substitute inspection
for verification.

## Systematic debugging discipline

Treat every widget as a **hypothesis** about what will render, then test it.

- **Evidence before assertions.** Before claiming a widget works (or a review is
  clean), run its query and read the actual output. Reviewing your own mental
  model with a copy of your own mental model will not catch an error in that model.
- **Reproduce, don't guess.** When a tile is empty or a number is off, run the
  widget's exact query in the explorer and observe. Do not theorize a cause.
- **One variable at a time.** Isolate: is the event landing? the field name? the
  facet? the compute? the formula? Narrow before fixing.
- **Verify names at the source.** Pull a real event with `extra_fields:["*"]` and
  read the true `@attribute` paths. Codebases and PRs lie about field names; live
  events do not.

## Workflow — creating a dashboard

1. **Capture intent per section.** Before writing JSON, write one sentence per
   section: "this section answers **X** for **audience Y**." Keep it — it is the
   Gate-3 checklist.
2. **Discover the real schema.** Before writing any query, confirm actual event
   names, field paths, facet values, and measures by querying live data. Never
   trust a field name you have not seen in a live event.
3. **Draft the JSON.** See `references/datadog-json.md` for widget shapes,
   formulas (`default_zero`, `cardinality`), conditional_formats, per-widget time.
4. **Gate 1** — validate structure.
5. **Gate 2** — run every query against live data; confirm non-empty and shaped
   as intended. Watch `references/query-pitfalls.md` while doing this.
6. **Gate 3** — for each section, confirm it renders the sentence from step 1.
7. **Iterate.**

## Workflow — reviewing a dashboard

Apply the same three gates to existing JSON. Specifically:

- Run `validate_dashboard.py` (Gate 1).
- **Extract every query and run each against live data** (Gate 2) — do **not**
  approve from static reading. This is exactly the class of bug static review
  misses.
- For each section, state what a viewer would conclude and whether that matches
  the user's intent (Gate 3).
- When you report a review, mark findings and say plainly which gates you could
  and could not verify (e.g. "Gate 2 verified against prod; ffuf tile returns 15
  runs" vs "Gate 2 not run — no data access").

## Gate 3 — "will it render what they want?" checklist

For every widget and section, ask:

- What exact question does this answer?
- Against current data, what will it show — a number, a trend, empty? Is empty
  **meaningful** (healthy zero) or a **bug** (wrong field / missing facet / dark
  telemetry)?
- Can "no data" be told apart from "healthy zero"? Every failure count needs a
  **denominator/total** beside it.
- Does the number **mean what the title claims**? Check denominator semantics,
  count-vs-cardinality, and that approximations are labeled `(approx)`.
- Is the **most useful information at the top** of the section and the board?

## Pitfalls

Before writing or reviewing queries, read `references/query-pitfalls.md` — the
catalog of reasons a query returns no/wrong data: field-name reality gap,
no-data-vs-zero, sparse-join nulls, count-vs-cardinality, denominator semantics,
missing facets/measures, division-by-zero, conditional_formats /
`cell_display_mode` conflicts, env-dimension mismatch, retention & metric tag
cardinality, phrase-search tokenization.

## Bundled resources

- `scripts/validate_dashboard.py` — Gate 1 structural linter (JSON, widget shape,
  sparse-join risk, cell_display_mode/conditional_formats conflict, section
  numbering, stale note cross-references).
- `scripts/extract_queries.py` — dumps a per-widget query checklist for Gate 2.
- `references/query-pitfalls.md` — the data-returns failure-mode catalog.
- `references/datadog-json.md` — widget/query/formula schema essentials.

