# Visual Data Dictionary

> Build a Data Schematic with its Data Dictionary beside it — an interactive qsv viz smart dashboard driven by an LLM-inferred JSON Schema data dictionary, browsable in-page next to the charts. Use when the user asks for a Data Schematic, a visual data dictionary, a documented dashboard or a dictionary-driven dashboard, or wants to explore and document a CSV at the same time. Optionally bins rows into GeoJSON regions.

- Skill: `dathere/visual-data-dictionary` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add dathere/visual-data-dictionary`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dathere/visual-data-dictionary/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: dathere (https://skillmd.com/u/dathere)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/dathere/visual-data-dictionary

---


# /visual-data-dictionary

> **Scope: repo-local.** This lives beside `build-dashboard`, `release-prep` and
> `review-respond` at the top of `.claude/skills/`, which `package-plugin.js` and
> `package-mcpb.js` do **not** archive — they ship only `.claude/skills/skills/`.
> So this skill is available when working in the qsv repo and is *not* part of the
> distributed plugin. That is deliberate: the packaged skills drive qsv through the
> `mcp__qsv__*` MCP tools, while this one drives the **qsv CLI** directly and needs
> `Bash` plus `python3`. Shipping it would require rewriting it against the MCP
> tool surface, which has no equivalent for the GeoJSON inspection or the HTML
> verification below (nor for the optional Stage 6 browser pass).
>
> **Requires:** `qsv` on `PATH`, `python3`, and an LLM endpoint for `describegpt`.
> Stage 6 (optional) additionally needs a browser-automation MCP — any one
> (Playwright MCP, claude-in-chrome, …); skip the stage when none is available.

Turn a CSV into a self-contained HTML Data Schematic whose panels are chosen from an
LLM-inferred data dictionary, with that dictionary embedded beside the charts.

Four stages, plus one optional fine-tune and one optional browser pass, in this
order and no other:

1. **denull** — blank null sentinels so numeric columns are actually numeric
2. **describegpt** — infer a JSON Schema data dictionary from the *cleaned* data
   - **2.5 fine-tune** (optional) — hand-correct the dictionary in a terminal UI
     before it drives the Data Schematic
3. **geojson** (optional) — fetch US boundaries with `--geojson auto`, or pick a
     feature id key by inspecting a supplied file
4. **viz smart** — render the Data Schematic, dictionary-driven, dictionary-embedded
   - **5 verify** — check the HTML, then report
   - **6 tour refinement** (optional, browser) — step through the guided Tour and
     refine its `x-qsv.tour` narration against what actually rendered

The order is load-bearing. Clean first, then describe, then draw. A dictionary
built from dirty data documents a `String` column that is really a number, and
`viz smart` will then chart it as a category or skip it outright.

## IMPORTANT

You must **execute bash commands**. Never invent qsv flags — if unsure, run
`qsv <cmd> --help`. Skip any step already satisfied by conversation context.
Defer to CLAUDE.md when it conflicts with this skill.

## Naming

Given input `data.csv`, derive:

| var | value | note |
|---|---|---|
| `STEM` | `data` | basename minus extension |
| `WORK` | `data.denulled.csv`, or `data.csv` if nothing was cleaned | what stages 2–4 read |
| `SCHEMA` | `<WORK stem>.schema.json` | `viz --dictionary infer` reuses this exact name |
| `OUT` | `data.html` | **always the ORIGINAL stem**, per the user's expectation |

Never write to the input path. `denull --apply` refuses to overwrite its own
input (it compares file identity, so a hard link is caught too), but pick a
distinct `-o` anyway.

---

## Stage 0 — Preconditions

```bash
command -v qsv >/dev/null || { echo "qsv not on PATH"; exit 1; }
test -f "$INPUT" || { echo "no such file: $INPUT"; exit 1; }
qsv headers "$INPUT" | head -30
qsv count "$INPUT"
```

Only CSV/TSV/SSV. If handed a spreadsheet, convert first (`qsv excel`).

Build the index and stats cache once — every later stage reuses them:

```bash
qsv index "$INPUT"
qsv stats "$INPUT" --everything --stats-jsonl --force > /dev/null
```

## Stage 1 — denull

**Report first. Always show the user before changing their data.**

```bash
qsv denull "$INPUT"
```

Read the `verdict` column:

- **No rows, or no `confirmed` row** → nothing to clean. Set `WORK="$INPUT"` and
  go to Stage 2. Do not create a copy.
- **One or more `confirmed`** → show the table, then:

```bash
qsv denull --apply "$INPUT" -o "${STEM}.denulled.csv"
qsv index "${STEM}.denulled.csv"
qsv stats "${STEM}.denulled.csv" --everything --stats-jsonl --force > /dev/null
```

Set `WORK="${STEM}.denulled.csv"`.

`--apply` prints its report to **stderr** and the cleaned CSV to `-o`, and blanks
sentinels **only** in the columns it confirmed. Every other column is copied
through byte-for-byte.

Sanity check worth doing: each confirmed column's `rows_affected` should equal
its `nullcount` in the new stats.

```bash
qsv stats "${STEM}.denulled.csv" | qsv select field,type,nullcount | qsv table
```

Two things to tell the user, because they are not obvious:

- `denull` only confirms columns that would **promote to a numeric type** once
  blanked. A categorical column holding `NULL` (e.g. `status` = ok/pending/NULL)
  is deliberately left alone — blanking it promotes nothing. Stage 2 will still
  surface it.
- Numeric sentinels (`-999`, `9999`) are **not** detectable by any scan: they
  parse as valid numbers. Only Stage 2's LLM can propose them, and only a human
  should apply them.

## Stage 2 — describegpt → JSON Schema dictionary

### Resolve the LLM endpoint

Detect, then prompt only if nothing is found. Do **not** print key values.

```bash
for v in QSV_LLM_BASE_URL OPENAI_API_KEY QSV_LLM_APIKEY ANTHROPIC_API_KEY; do
  val=$(printenv "$v" 2>/dev/null); [ -n "$val" ] && echo "$v is set"
done
curl -s -m 2 http://localhost:1234/v1/models >/dev/null 2>&1 && echo "LM Studio on :1234"
curl -s -m 2 http://localhost:11434/api/tags  >/dev/null 2>&1 && echo "ollama on :11434"
```

Both LM Studio and ollama speak the OpenAI-compatible API, so **both** list models
the same way and both take a `/v1` base URL. Only the port differs:

| server | `--base-url` | list models |
|---|---|---|
| LM Studio | `http://localhost:1234/v1` | `curl -s http://localhost:1234/v1/models` |
| ollama | `http://localhost:11434/v1` | `curl -s http://localhost:11434/v1/models` |

```bash
# honor an explicit QSV_LLM_BASE_URL first; only probe local servers when it is unset
BASE_URL="${QSV_LLM_BASE_URL:-}"
[ -z "$BASE_URL" ] && curl -s -m 2 http://localhost:1234/v1/models  >/dev/null 2>&1 && BASE_URL=http://localhost:1234/v1
[ -z "$BASE_URL" ] && curl -s -m 2 http://localhost:11434/api/tags >/dev/null 2>&1 && BASE_URL=http://localhost:11434/v1

[ -n "$BASE_URL" ] && curl -s "$BASE_URL/models" \
  | python3 -c 'import sys,json;[print(m["id"]) for m in json.load(sys.stdin)["data"]]'
```

`/api/tags` is only a liveness probe for ollama — it returns ollama's native
shape, not the OpenAI `{"data":[...]}` envelope. List models from `/v1/models`
either way.

If nothing is found, use **AskUserQuestion** for base URL + model. Never guess a
model name. Offer the models the server actually reports; do not type one from
memory.

### Generate

First ask with **AskUserQuestion**: *"Who is the Data Schematic's guided Tour
for?"* — default `TOUR_AUDIENCE="Explain like I'm 10"`; any free-text audience
works ("a board of directors", "data journalists", …).

```bash
qsv describegpt "$WORK" \
  --dictionary --description --two-pass --infer-content-type \
  --format JSONSchema \
  --tour-audience "$TOUR_AUDIENCE" \
  ${BASE_URL:+--base-url "$BASE_URL"} --model "$MODEL" \
  -o "$SCHEMA"
```

- `--infer-content-type` is **mandatory here**, not optional: `viz smart` routes
  panels off each field's `role` and `concept`, and those are only inferred under
  this flag. Without it the dictionary loads and changes nothing. It is also the
  only way to get the three dictionary hints that unlock extra panels:
  per-field `x-qsv.gauge_range` (turns a measure's KPI tile into a gauge; kept
  only when the observed data lies inside the range), per-field
  `x-qsv.denominator` on a region column (adds a **per-capita rate map** beside
  the raw count map), and the dataset-level `x-qsv.relationships` array, whose
  `"kind": "pipeline"` entry is the **only** source of the pipeline
  funnel/bridge panel.
- `x-qsv.denominator` is the odd one out: qsv **derives** it, the LLM does not
  propose it. The model's whole contribution is tagging one column
  `measure.population` (a count of people or households IN a region another
  column names); `describegpt` then attaches the hint to every region column
  that can hold it, checking from the stats cache alone that the counts are
  plausible. Two population-shaped columns are an ambiguity it refuses rather
  than guesses at, so you get nothing. This is what makes a rate map reachable
  without hand-editing the JSON or paying for a `--denominator census` fetch of
  a number the file already carries.
- Pass `--context-file <file>` when the user has a glossary, README or codebook.
  Better context yields better roles, concepts and labels, hence a better
  Data Schematic. (`viz --dictionary-context` is the same thing for the `infer` path,
  which this skill does not take.)
- `--two-pass` roughly doubles cost and latency. It is what lets the model relate
  fields to one another (`street_no` + `street` + `city` + `zip` = one address),
  which is what makes the routing good.
- Naming it `<WORK stem>.schema.json` means a later
  `qsv viz smart "$WORK" --dictionary infer` finds and **reuses** it instead of
  paying for the LLM again. Delete the file to force a re-infer.
- `--tour-audience` makes describegpt also write a dataset-level `x-qsv.tour`
  narration — the prose the Data Schematic's guided Tour speaks — in the
  audience's register. The audience shapes ONLY the tour prose; labels and
  descriptions keep their normal register. Stage 6 refines it in a browser.

Optionally add `--infer-null-values` to have the model propose null sentinels
into each property's `x-qsv` object, split into `null_values` (confirmed present
by qsv) and `null_candidates` (guesses, each stamped `confirm_required: true`).
This is the only route to numeric sentinels like `-999`. It is **reported, never
applied** — nothing downstream acts on it.

Verify the dictionary carries what `viz` needs before spending time on Stage 4:

```bash
python3 - "$SCHEMA" <<'PY'
import json, sys
s = json.load(open(sys.argv[1]))
p = s["properties"]
have = sum(1 for v in p.values() if v.get("x-qsv", {}).get("role"))
print(f"role/concept on {have}/{len(p)} columns")
if have == 0:
    print("WARNING: no roles inferred — was --infer-content-type passed?")
# panel-unlocking hints, so the user knows up front what will/won't be drawn
gauges = [k for k, v in p.items() if (v.get("x-qsv") or {}).get("gauge_range")]
# a derived denominator turns the region map into a count map PLUS a rate map
denoms = {k: ((v.get("x-qsv") or {}).get("denominator") or {}).get("column")
          for k, v in p.items()}
denoms = {k: c for k, c in denoms.items() if c}
# viz reads pipelines ONLY from the dataset-level x-qsv (see xq_pipelines in
# src/cmd/viz.rs) — a root-level "relationships" array draws nothing.
rels = (s.get("x-qsv") or {}).get("relationships") or []
pipes = [r for r in rels if r.get("kind") == "pipeline"]
print(f"gauge_range on {len(gauges)} measure(s): {', '.join(gauges) or '(none)'}")
print(f"denominator on {len(denoms)} region column(s): "
      + (", ".join(f"{k} -> {c}" for k, c in denoms.items()) or "(none)")
      + ("   [rate map]" if denoms else ""))
print(f"relationships: {len(rels)} ({len(pipes)} pipeline -> funnel/bridge panel)")
if not rels and s.get("relationships"):
    print("WARNING: relationships found at the ROOT, not under x-qsv — viz ignores"
          " those. Is this the flat JSON dictionary instead of JSONSchema?")
tour = (s.get("x-qsv") or {}).get("tour")
if tour:
    print(f"tour: version {tour.get('version')}, audience {tour.get('audience')!r}, "
          f"{len(tour.get('overrides') or {})} override(s), "
          f"{len(tour.get('panels') or {})} panel narration(s)")
else:
    print("tour: (none — was --tour-audience passed?)")
PY
```

No `gauge_range`, no `denominator` and no `pipeline` is a perfectly normal
outcome — most datasets have none of a canonical-scale measure, a per-region
population, or a staged process. Say so and move on; all three can be hand-added
later (see Stage 2.5).

## Stage 2.5 — Fine-tune the dictionary (optional, TUI)

`describegpt` is a good first draft, not gospel — and the draft is not even
stable: because the semantic half comes from an LLM, inferring twice over the
same data can return different `role`/`concept` assignments, and `role` decides
which panel a column gets (qsv issue #4407). **This stage is what makes a
Data Schematic reproducible.** The corrected dictionary — reviewed, kept beside
the data, committed if the data is versioned — is the artifact of record; every
later run reuses it instead of re-rolling the model.

The five fields that actually steer `viz smart` — `x-qsv.role`,
`x-qsv.concept`, `title` (label), `description` and `x-qsv.aggregation` — are
worth a human pass when the model mislabels a column: a code that should be an
`identifier` charted as a `measure`, a `geo.*` key left `unknown`, a per-unit
price summed into a meaningless total, a bland label. `edit_dictionary.py`
(beside this `SKILL.md`) is a curses UI that walks every column and, as you
edit, **previews how `viz smart` will route it** (Skip / Dimension / Temporal /
MapCoord / ProjectedCoord / Measure — the last showing its aggregation,
`Measure(sum)` for an additive amount, `Measure(mean)` for a ratio, a duration,
or anything you tag `aggregation: mean`), so you see the effect before
rendering. It touches only those five fields, preserves every other key, and
rewrites the file only if you save.

`aggregation` is the one field qsv can also **drop** on read, and the `!` flag
mirrors exactly when that happens: the token must be `sum`/`mean`,
`x-qsv.qsv_type` must be `Integer`/`Float` (or absent), `x-qsv.role` must be
empty or exactly `measure`, *and* the column must route to a measure at all. A
value failing any of those is flagged and left out of the ROUTE preview, because
viz silently falls back to its own name heuristic there. It catches the three
easy mistakes: `"average"` instead of `"mean"`, an aggregation left behind on a
column you just re-roled to `dimension`, and one on a column nothing classifies
(`Defer→stats`), where viz's stats floor discards it outright.

**Offer it with AskUserQuestion:** *"Hand-tune the data dictionary in a TUI
before rendering?"* If **no**, go straight to Stage 3 — but say plainly that the
Data Schematic then rests on an unreviewed draft, and that the dictionary can be
tuned and re-rendered at any time without paying for the LLM again.

If **yes**, you cannot drive it yourself — a curses TUI needs the user's real
terminal, and your Bash tool is a captured, non-interactive shell (the script
detects this and refuses). So run it **out-of-band**:

1. Show the current routing so the user knows the starting point (this works
   without a TTY):

   ```bash
   python3 "$SKILL_DIR/edit_dictionary.py" --summary "$SCHEMA"
   ```

   where `$SKILL_DIR` is this skill's own directory (the folder holding this
   `SKILL.md`).

2. Tell the user to run this in **their own terminal**, then **end your turn and
   wait** — do not proceed:

   ```
   python3 "<skill dir>/edit_dictionary.py" "<SCHEMA path>"
   ```

   Keys: `↑↓` move · `r` role · `c` concept · `l` label · `d` description ·
   `a` aggregation · `s` save · `q` quit. `role`/`concept` open a filterable
   picker (type to filter; off-vocab values are allowed but flagged with `*`).
   `aggregation` offers `sum` / `mean` / clear only — that is the whole
   vocabulary qsv accepts — and clearing removes the key rather than blanking
   it, restoring qsv's own guess.

3. When the user says they're done, re-read the file: re-run the **Stage 2
   coverage check** and the `--summary` above, and show a short **before/after**
   of any rows whose role/concept/route changed. Then continue to Stage 3.

Because the dictionary keeps its `<WORK stem>.schema.json` name, Stage 4 picks up
the edited file with no extra wiring. If the user edits nothing, the file is
untouched byte-for-byte — treat that as a normal "looks good" outcome.

Scope note: the TUI deliberately does **not** edit null sentinels
(`--infer-null-values` output). Those are reported-never-applied and have no
`viz smart` effect, so editing them here would change nothing downstream.

Seven keys that *do* affect the Data Schematic are outside the TUI, and are
**hand-edited in the JSON** — this is the supported path for them, not a
violation of the "never hand-write the schema" rule. (`x-qsv.denominator` is the
one qsv now normally fills in for you; hand-editing it is a correction, or the
route to an area/household denominator describegpt will not derive.)

| key | where | effect |
|---|---|---|
| `x-qsv.gauge_range` | per property, `[min, max]` | KPI tile becomes a **gauge**. `describegpt` proposes it for canonical-scale measures; qsv drops it if the data falls outside the range |
| `x-qsv.target` | per property, a number | KPI tile gains a **"vs target" delta**. Never inferred — it is a goal only the user knows |
| `x-qsv.currency` | per property, an ISO-4217 code (`"USD"`) | KPI tile is prefixed with the currency **symbol** (`$192B`) and the panel subtitle names the currency. `describegpt` proposes it for money columns; qsv drops it unless the column is a numeric measure that reads as money (concept `measure.money` or `measure.amount`, or content type `money`) |
| `x-qsv.denominator` | per property on a REGION column, `{"column": "<name>", "level": "geo.county"}` | the region map gains a **rate panel** beside the raw count ("per 10,000 residents"). Normally **derived** by describegpt from a `measure.population` column (#4523); hand-edit to point at households/area, or to correct it. An explicit `--denominator census`/`--denominator-key` flag outranks it (plain `--denominator <col>` is `viz choropleth` only and is REJECTED by `viz smart` - in `smart` the column form IS this dictionary key). The optional `level` names the geography the denominator column is defined AT (a `DENOMINATOR_REGION_CONCEPTS` token such as `geo.state`/`geo.county`); describegpt copies it from the measure's own `geo_level` proposal, so do not add a `geo_level` key to the measure column yourself - that second copy is free to drift. qsv refuses a hint three ways: more distinct values than the region key could hold constant, a `level` naming a **different geography** than the region key (#4526 - a coarser denominator makes a confident, wrong rate map that cardinality alone cannot detect), and a denominator that is **constant across every region** (#4547 - the rate panel would just be the count panel rescaled) |
| `x-qsv.unit` | per property, a curated UCUM code (`km`, `Cel`, `kWh`) | the KPI tile and panel subtitle name the **unit** (`18.4 °C`, `1.2B kWh`), and it follows the number into hover text and pair/3D axis titles. `describegpt` proposes it for numeric measures; qsv re-derives the display symbol from its own curated table, so an **off-table code silently vanishes** - and codes are matched **byte-exactly** (UCUM is case-sensitive: `Cel`, not `cel`). The guardrail also clears it when a measure is downgraded to a dimension, so a declared unit cannot sit on something that is not a quantity |
| `x-qsv.relationships` | dataset level, `{"kind":"pipeline", …}` | draws the **pipeline** panel |
| `x-qsv.tour` | dataset level | replaces the guided Tour's built-in narration: `overrides` keyed by step id, `panels` keyed by RAW field name or `@kind` token, `panel_order` picks/orders the panel spotlights (capped by `--tour-steps`, default 8). `version` MUST stay the **integer** `1` — viz silently discards the whole block on anything else. Plain text only; `language` (if present) is BCP-47 and must match the page locale or overrides are dropped. Written by `--tour-audience` (Stage 2), refined in Stage 6 |

(`x-qsv.aggregation` used to be a fifth row here; it is now edited with `a` in
the TUI above. Its meaning is unchanged: `sum` or `mean` on a numeric measure,
declaring how the column combines across a group and overriding qsv's
column-NAME heuristic in both directions. Use `mean` for anything per-unit or
per-record — a unit price, a rating, a temperature, a duration — and `sum` only
for a quantity each row contributes. It is the **language-neutral** signal: the
name heuristic is English-first and cannot read non-English column names, issue
#4401.)

**One exception to "overrides in both directions"** (issue #4528): a
REGION-LEVEL column — one a region declares as its `x-qsv.denominator`, or one
tagged `measure.population`/`measure.area` — holds a value that repeats
identically on every row of its region. `viz smart` collapses it to one value
per owning region before aggregating, and that collapse outranks an explicit
`x-qsv.aggregation`. #4401's precedence is about what a measure MEANS
(extensive vs intensive); this is about the data's SHAPE — the rows are
duplicates of one regional value, and even a genuinely extensive measure must
not be counted once per duplicate. So writing `aggregation: sum` on a
population column does **not** restore a row-wise total. On tidy
one-row-per-region data the collapse is a no-op. Two consequences to expect:
its grouped bar shows the region's own figure (or, grouped by something
coarser, the sum of the distinct regional values), and its **KPI tile is
omitted entirely** when regions repeat, because no available aggregation can
state a true dataset-wide total without a region-keyed dedupe pass.

For a pipeline, both encodings are hand-editable — stages as columns
(`"members"` in process order, **widest/upstream first**, the opposite direction
from `"kind":"ordered"`), or stages as row values (`"stage_column"` + an ordered
`"stages"` list + an optional `"value_column"` to sum). Declared order is
authoritative: if a stage outruns its predecessor, viz draws a **bridge** of
signed differences instead of a funnel, rather than a band wider than the one
above it. Offer these edits only when the Stage 2 check showed a plausible
candidate; do not invent a target.

## Stage 3 — GeoJSON (optional)

Ask with **AskUserQuestion**: *"Bin rows into GeoJSON regions?"*

If **no**, skip to Stage 4 with no geo flags.

If **yes**:

### 3a. Check the data can actually be binned

`viz smart` reaches a region **two** ways, and only one of them needs coordinates:

- **Region key** — `--locations <col>` (with `--location-mode`) where the column already
  identifies the region: an ISO-3 country code, a 2-letter US state code, a country name, a
  GeoJSON feature id, county FIPS/GEOID. **No coordinate pair required.** This is the path that
  works against a custom `--geojson` file, and the feature id must JOIN these values (Stage 3c).
- **Place NAME** — two different routes, and one precondition that governs both. The
  precondition: `geo.city`/`town`/`municipality` columns are nominated as region candidates only
  on a **geocode-enabled build** (`geocodable_name_candidates` returns nothing otherwise, and
  degrades silently). Check `qsv --version` for `geocode` before promising either route.
  - *Direct*, with any `--geojson` file whose feature ids ARE those names: an ordinary join, no
    aliases, so Stage 3c's overlap check settles it. `examples/viz/nyc_neighborhoods.geojson`
    (keyed by `properties.name`) is exactly this shape.
  - *Alias-based city → county FIPS*, `--geojson auto`/`census` only — the alias map is
    synthesized by automatic Census resolution, so a custom file publishes none. `viz smart`
    drives this itself: it forward-geocodes implicitly and does **not** need the `--geocode`
    flag, which is why Stage 4 not passing `--geocode` is no obstacle. (The explicit flag is a
    `viz choropleth` concern and is restricted to the `iso3`/`usa-states` location modes.)

  So do not reject a name column against a custom file — test the overlap first. Require a
  region-CODE column only when the names do not join, or when the build has no geocode support.
- **Point-in-polygon** — `--lat`/`--lon`, each row's coordinates tested against
  the polygons.

Check BOTH before concluding anything, and check the **dictionary first** — `viz smart`
identifies its region column from the dictionary's concepts, not from header spelling, so
`$SCHEMA` is the authoritative answer and a header regex is only a fallback:

```bash
# authoritative: only these geo.* leaves key a polygon. NOT every geo.* concept does -
# geo.latitude/longitude/coordinate_pair/street_address/ip_address/timezone/geonames_id name a
# point or an attribute, and treating them as region keys is the same false positive inverted.
python3 - "$SCHEMA" <<'PROBE'
import json, sys
# mirrors viz.rs REGION_CODE_LEAVES + CITY_NAME_LEAVES
CODE = {"zip_code","zip","postal_code","zcta","census_tract","county","county_fips","state",
        "state_fips","country","country_code","place_fips","fips"}
CITY = {"city","town","municipality"}   # need a geocode-enabled BUILD, not the --geocode flag
props = json.load(open(sys.argv[1])).get("properties", {})
con = {k: str((v.get("x-qsv") or {}).get("concept", "")) for k, v in props.items()}
leaf = lambda c: c.split(".", 1)[1] if c.startswith("geo.") else None
code = [k for k, c in con.items() if leaf(c) in CODE]
city = [k for k, c in con.items() if leaf(c) in CITY]
pair = ([k for k, c in con.items() if leaf(c) == "latitude"],
        [k for k, c in con.items() if leaf(c) == "longitude"])
print("region-code columns :", code or "(none)")
print("city-name columns   :", city or "(none)")
if city:
    print("  ^ nominated as region candidates only on a geocode-enabled BUILD")
    print("    (check `qsv --version` for `geocode`). NOT the --geocode flag,")
    print("    which `viz smart` never needs - see 3a above for the two routes.")
print("lat/lon PAIR        :", "yes" if all(pair) else "no  # a lone lat or lon bins nothing")
PROBE
```

Do not reach for a region-name regex: it cannot spell every geography (`tract`, `zcta`,
`municipality`, `town`, `iso3` all miss a `state|county|country` pattern), and a false
negative here is exactly the mistake this stage used to make. Only when the probe finds
**neither** a region-code column (nor a city/place-name column — which on a geocode-enabled build
either joins a custom GeoJSON keyed by those names directly, or resolves to county FIPS on the
`auto` path) **nor** a complete lat/lon pair does the GeoJSON have no effect — a lone `geo.timezone` or `geo.ip_address`
is not a region key and does not count — say so then, and offer
to proceed without it. A dataset carrying county names (or FIPS codes) and no coordinates at
all maps perfectly well, so do not talk the user out of it.

### 3b. Get the file

Accept a local path, an `http(s)` URL, or a shortcut name defined in
`QSV_GEOJSON_SHORTCUTS` (a JSON map of `name` → `{path, id}`; the shortcut's `id`
supplies `--feature-id-key` when you don't pass one).

There is also a **fourth form, and for US data it is usually the right one**:
`--geojson auto` (or `census`) fetches US county, ZIP Code Tabulation Area,
census tract or place boundaries from the Census TIGERweb service, scoped to the
states the data names, and sets `--feature-id-key` to `properties.GEOID` itself.
The user supplies nothing but the CSV — no file to source, and Stage 3c below is
unnecessary. Prefer it over hunting for a boundary file. To chart a **rate** rather than a count, pair it
with `--denominator census` **only for county or state maps** - Census denominators exist for
those two geographies alone, and the fetch hard-errors without a free `QSV_CENSUS_API_KEY`.
For a ZCTA, tract or place layer, get the denominator from the data instead: a
`measure.population` column in the dictionary (describegpt derives `x-qsv.denominator` from
it), or `--denominator-key` pointing at a boundary property the fetched features carry.

### 3c. Discover the feature id key — do not guess it

`--feature-id-key` defaults to `id`, which is usually wrong. In `viz smart`'s
point-in-polygon mode the key **labels each binned region**, so it must be
present on every feature, unique across all of them, and *meaningful to a human*.
Uniqueness alone is not enough: `properties.shape_area` is perfectly unique and
completely useless as a label.

⚠️ That "meaningful to a human" rule is the **point-in-polygon** rule, where the key LABELS
each binned region. On the **region-key** path it is the wrong test and will cost you the
choropleth: there the key must *join* — its values have to overlap the distinct values of the
region column. Pick the key whose values match the CSV (GEOIDs match GEOIDs, names match
names), verify the overlap before rendering, and put the human-readable property in
`--feature-name-key` instead, which exists precisely to supply hover labels. Choosing a
display name here while the CSV holds GEOIDs resolves nothing and renders no map.

#### Region-key or name path — ask qsv, do not reimplement the match

`qsv viz --check-geojson-key` scores **every** candidate feature-id path against the distinct
values of the region column and ranks them by overlap. Use it and take its answer:

```bash
qsv viz choropleth "$WORK" --locations "$REGION_COL" --geojson "$GEOJSON" --check-geojson-key
```

```
3221 distinct --locations values scored against --geojson 'counties.geojson':
     3221/3221 100.0%  properties.GEOID
        0/3221   0.0%  properties.NAME  (unmatched e.g. 01001, 01003, 01005)

Use: --feature-id-key properties.GEOID
```

It scores through the same matcher the render path binds with, so a path it reports as a full
match **will** bind at render time — zero-padding (`6` vs `06037`), ASCII case folding, and the
refusal to guess between ambiguous folds (with features `CA` and `ca`, the value `Ca` matches
neither) all come out identical by construction. That is why this replaced a hand-written
scorer here: a reimplementation that is one tier more generous than `viz` reports a join you
will not get.

It reads the same `--geojson` sources `viz` does (local path, `http(s)` URL, or a
`QSV_GEOJSON_SHORTCUTS` name) and needs no valid `--feature-id-key` to run — finding one is its
job. Candidates include nested paths under `properties` at ANY depth and top-level foreign members,
not just `properties.<field>`, so a boundary file that keys off either is still scored. Treat a partial
match as a warning, not a pass. If nothing matches, the region values and the boundary file
disagree, or that GeoJSON cannot key them; say so rather than rendering an empty map.

⚠️ **Skip this check entirely when `$GEOJSON` is an automatic Census spec** — `auto`, `census`,
`census:<layer>` or either with an `@<year>` vintage (Stage 3b's fourth form). Every one of them
is refused, deliberately. Automatic Census resolution picks the feature-id key itself
(`properties.GEOID`) and prints its own region coverage, so there is nothing to discover; and a
place-NAME column binds through an alias map the check does not model, which would score every
candidate at 0% on a setup that renders correctly. Go straight to Stage 4 and read the coverage
line `viz` reports.

#### Point-in-polygon path — rank by uniqueness and readability

There is no join to test on this path (rows are binned by geometry), so the question is which
property makes a good **label**. That is a judgement about the boundary file alone, which the
script below answers.

It mirrors exactly one rule from `viz`: `build_pip_features` skips features without
Polygon/MultiPolygon geometry, so the script ranks over those features only. Rank over every
raw feature instead and a skipped point that duplicates an id makes that id look non-unique,
which reports a file as unkeyable on a key that would have labelled every binned region. That
one geometry check is the only parity this script needs — the *match tiers* it used to
reimplement now live behind `--check-geojson-key` above.

It accepts the same **file** source forms `--geojson` does — a local path,
an `http(s)` URL, or a `QSV_GEOJSON_SHORTCUTS` name. If you only handle local
paths here, a URL or shortcut fails at discovery even though `viz` would have
accepted it.

```bash
python3 - "$GEOJSON" <<'PY'
import json, sys, re, os, collections, urllib.request

def load_geojson(src):
    """Local path, http(s) URL, or a QSV_GEOJSON_SHORTCUTS name.

    Mirror viz's resolution order (src/cmd/viz.rs resolve_and_validate_geojson): an
    http(s) URL or an EXISTING local file is a direct source; only a value that is
    neither is looked up as a shortcut NAME. This keeps a local file whose name
    collides with a shortcut loading as the file (as viz does), and it never lets
    a malformed QSV_GEOJSON_SHORTCUTS break a direct file/URL input.
    """
    hint = None
    is_url = src.startswith(("http://", "https://"))
    if not is_url and not os.path.isfile(src):
        raw = os.environ.get("QSV_GEOJSON_SHORTCUTS")
        if not raw:
            sys.exit(f"--geojson '{src}' is not an existing file or http(s) URL, "
                     "and QSV_GEOJSON_SHORTCUTS is not set")
        shortcuts = json.loads(raw)           # invalid JSON surfaces as an error
        if src not in shortcuts:
            sys.exit(f"unknown --geojson shortcut '{src}'; "
                     f"defined: {', '.join(sorted(shortcuts)) or '(none)'}")
        entry = shortcuts[src]
        hint = entry.get("id")                # shortcut may carry its own id key
        src = entry["path"]
        is_url = src.startswith(("http://", "https://"))
    if is_url:
        with urllib.request.urlopen(src, timeout=30) as r:
            return json.loads(r.read().decode("utf-8")), src, hint
    with open(src) as fh:
        return json.load(fh), src, hint

g, resolved, hint = load_geojson(sys.argv[1])
feats = g.get("features", [])
if not feats:
    sys.exit("no features")
# Rank over the features viz will actually BIN. build_pip_features skips anything without
# Polygon/MultiPolygon geometry, so ranking over every raw feature reports a key as non-unique
# whenever a skipped point duplicates it - and then declares the file unusable on a key that
# would have labelled every binned region perfectly.
feats = [f for f in feats
         if (f.get("geometry") or {}).get("type") in ("Polygon", "MultiPolygon")]
if not feats:
    sys.exit("no Polygon/MultiPolygon features - viz cannot bin rows into this file")
print(f"source: {resolved}")
if hint:
    print(f"shortcut supplies --feature-id-key {hint} (override below if you prefer)")

# Geometry-derived / bookkeeping fields: unique, but meaningless as a region label.
NOISE = re.compile(r"shape|area|leng|length|perim|acres|sqmi|aland|awater|"
                   r"intptlat|intptlon|^lat|^lon|_x$|_y$|"
                   r"date|time|edited|created|updated|version", re.I)

def floatish(v):
    return isinstance(v, float) or (isinstance(v, str) and re.fullmatch(r"[+-]?\d+\.\d+", v.strip()))

cands = collections.defaultdict(list)
for f in feats:
    if f.get("id") is not None:
        cands["id"].append(f["id"])
    for k, v in (f.get("properties") or {}).items():
        if isinstance(v, (str, int, float)):
            cands[f"properties.{k}"].append(v)

good, other = [], []
for key, vals in cands.items():
    if len(vals) != len(feats):                    # missing on some feature
        continue
    if len(set(map(str, vals))) != len(feats):     # not unique
        continue
    demote = bool(NOISE.search(key)) or all(floatish(v) for v in vals)
    (other if demote else good).append((key, vals[:3]))

def show(title, rows):
    print(f"\n{title}")
    if not rows:
        print("  (none)")
    for key, sample in rows:
        print(f"  {key:<32} e.g. {sample}")

print(f"{len(feats)} usable (polygon) features")

show("RECOMMENDED feature-id-key (unique, meaningful):", good)
show("Unique but geometry/bookkeeping - avoid:", other)
if not good and not other:
    print("\nNo property is unique across all features. This GeoJSON cannot key regions as-is.")
PY
```

What you offer via **AskUserQuestion** depends on the path Stage 3a identified:

- **Region key or name path** — do not offer this ranking at all. Take
  `--check-geojson-key`'s highest scorer, ideally a full match. Readability is irrelevant here
  and actively misleading: a numeric `properties.GEOID`/`OBJECTID` that joins is correct, while
  a pretty `properties.hood` that joins nothing renders an empty choropleth.
- **Point-in-polygon path** — offer the **RECOMMENDED** keys and favour a short region code or
  name (`properties.nta2020`, `properties.hood`) over a surrogate key
  (`properties.OBJECTID`, a GUID): here the value really does label each binned region.

If nothing is unique, say so plainly: the GeoJSON cannot key regions as-is.

Then pick `--feature-name-key` (e.g. `properties.name`) for human-readable hover labels — that
is where a readable property belongs. Stage 4 passes it only when you set `FEATURE_NAME_KEY`.

## Stage 4 — Render

Ask for `--dataset-pid` with **AskUserQuestion** (a persistent identifier — a DOI,
ARK, Handle, or a URL). It is optional; allow the user to skip it.

```bash
qsv viz smart "$WORK" \
  --smarter --bivariate \
  --dictionary "$SCHEMA" --dict-info \
  ${GEOJSON:+--geojson "$GEOJSON"} \
  ${FEATURE_ID_KEY:+--feature-id-key "$FEATURE_ID_KEY"} \
  ${FEATURE_NAME_KEY:+--feature-name-key "$FEATURE_NAME_KEY"} \
  ${DATASET_PID:+--dataset-pid "$DATASET_PID"} \
  -o "$OUT"
```

- `--smarter` runs `qsv moarstats --advanced` first, enriching the stats cache
  with distribution shape (bimodality, entropy, skewness, outlier share, Gini —
  the last unlocks Lorenz curves for the most unequal additive measures). Costs
  one extra pass and writes `<stem>.stats.csv` + sidecars + `.idx`. It applies
  only under default parsing: `--no-headers` or a custom `--delimiter` silently
  falls back to the standard Data Schematic.
- `--bivariate` adds a normalized-mutual-information heatmap plus — only when
  there are more than 8 chartable columns — a ranked "top relationships" bar.
  **It implicitly turns on `--dictionary infer` when `--dictionary` is not set**
  — so passing `$SCHEMA` explicitly is what stops viz from calling the LLM a
  *second* time. Never pass `--bivariate` without a dictionary in this workflow.
  Capped at 50 columns; wider datasets skip both panels with a warning.
- `--dict-info` embeds the dictionary in a side drawer next to the plots, adds an
  info icon per panel, and a "Data Dictionary" link under the title. The drawer
  also carries **download buttons for the sidecars this run actually read** —
  the schema, the charted frequency counts, the stats cache + metadata, and the
  bivariate CSV — all bundled *into* the HTML, so a recipient needs no access to
  your machine. Absolute local paths are stripped from the embedded metadata
  (sharing a Data Schematic does not disclose your directory layout); sidecars over
  4 MB are skipped with a note. **HTML only** — ignored with a note when
  exporting an image.
- `-o` must end in `.html`. An image extension (`.png`, `.svg`, …) silently
  switches viz to the static-export path, which needs a browser/webdriver and
  drops `--dict-info`.

### The data viewer drawer (`--preview-threshold`, default 50000)

Independent of `--dictionary`/`--dict-info`: an **(Explore)** link beside the row
count in the metadata table opens the underlying rows in a searchable bottom
drawer. Every row is embedded while the dataset has at most `<n>` rows; above
that only the first `<n>` are, and the link reads **(Preview)**.

This is the one flag here with a real cost: **embedded rows grow the HTML — and
the reader's browser memory — in proportion to rows × columns.** Tell the user
the size (Stage 5 prints it) rather than letting them discover it. Lower the
threshold, or pass `--preview-threshold 0` to drop the viewer entirely, when the
Data Schematic is meant to be emailed around.

### `--photos` — ask first, never enable silently

If a column holds image URLs, `--photos` makes dwelling on a map point reveal
that row's photo. It is **off by default and deliberately so**: images load from
whatever third-party

…(truncated)