Visual Data Dictionary

/visual-data-dictionary

Scope: repo-local. This lives beside build-dashboard, release-prep and review-respond at the top of .claude/skills/, which package-plugin.js and package-mcpb.js do not archive — they ship only .claude/skills/skills/. So this skill is available when working in the qsv repo and is not part of the distributed plugin. That is deliberate: the packaged skills drive qsv through the mcp__qsv__* MCP tools, while this one drives the qsv CLI directly and needs Bash plus python3. Shipping it would require rewriting it against the MCP tool surface, which has no equivalent for the GeoJSON inspection or the HTML verification below (nor for the optional Stage 6 browser pass).

Requires: qsv on PATH, python3, and an LLM endpoint for describegpt. Stage 6 (optional) additionally needs a browser-automation MCP — any one (Playwright MCP, claude-in-chrome, …); skip the stage when none is available.

Turn a CSV into a self-contained HTML Data Schematic whose panels are chosen from an LLM-inferred data dictionary, with that dictionary embedded beside the charts.

Four stages, plus one optional fine-tune and one optional browser pass, in this order and no other:

1. denull — blank null sentinels so numeric columns are actually numeric
2. describegpt — infer a JSON Schema data dictionary from the cleaned data
   • 2.5 fine-tune (optional) — hand-correct the dictionary in a terminal UI before it drives the Data Schematic
3. geojson (optional) — fetch US boundaries with --geojson auto, or pick a feature id key by inspecting a supplied file
4. viz smart — render the Data Schematic, dictionary-driven, dictionary-embedded
   • 5 verify — check the HTML, then report
   • 6 tour refinement (optional, browser) — step through the guided Tour and refine its x-qsv.tour narration against what actually rendered

The order is load-bearing. Clean first, then describe, then draw. A dictionary built from dirty data documents a String column that is really a number, and viz smart will then chart it as a category or skip it outright.

IMPORTANT

You must execute bash commands. Never invent qsv flags — if unsure, run qsv <cmd> --help. Skip any step already satisfied by conversation context. Defer to CLAUDE.md when it conflicts with this skill.

Naming

Given input data.csv, derive:

Never write to the input path. denull --apply refuses to overwrite its own input (it compares file identity, so a hard link is caught too), but pick a distinct -o anyway.

Stage 0 — Preconditions

command -v qsv >/dev/null || { echo "qsv not on PATH"; exit 1; }
test -f "$INPUT" || { echo "no such file: $INPUT"; exit 1; }
qsv headers "$INPUT" | head -30
qsv count "$INPUT"

Only CSV/TSV/SSV. If handed a spreadsheet, convert first (qsv excel).

Build the index and stats cache once — every later stage reuses them:

qsv index "$INPUT"
qsv stats "$INPUT" --everything --stats-jsonl --force > /dev/null

Stage 1 — denull

Report first. Always show the user before changing their data.

qsv denull "$INPUT"

Read the verdict column:

• No rows, or no confirmed row → nothing to clean. Set WORK="$INPUT" and go to Stage 2. Do not create a copy.
• One or more confirmed → show the table, then:

qsv denull --apply "$INPUT" -o "${STEM}.denulled.csv"
qsv index "${STEM}.denulled.csv"
qsv stats "${STEM}.denulled.csv" --everything --stats-jsonl --force > /dev/null

Set WORK="${STEM}.denulled.csv".

--apply prints its report to stderr and the cleaned CSV to -o, and blanks sentinels only in the columns it confirmed. Every other column is copied through byte-for-byte.

Sanity check worth doing: each confirmed column's rows_affected should equal its nullcount in the new stats.

qsv stats "${STEM}.denulled.csv" | qsv select field,type,nullcount | qsv table

Two things to tell the user, because they are not obvious:

• denull only confirms columns that would promote to a numeric type once blanked. A categorical column holding NULL (e.g. status = ok/pending/NULL) is deliberately left alone — blanking it promotes nothing. Stage 2 will still surface it.
• Numeric sentinels (-999, 9999) are not detectable by any scan: they parse as valid numbers. Only Stage 2's LLM can propose them, and only a human should apply them.

Stage 2 — describegpt → JSON Schema dictionary

Resolve the LLM endpoint

Detect, then prompt only if nothing is found. Do not print key values.

for v in QSV_LLM_BASE_URL OPENAI_API_KEY QSV_LLM_APIKEY ANTHROPIC_API_KEY; do
  val=$(printenv "$v" 2>/dev/null); [ -n "$val" ] && echo "$v is set"
done
curl -s -m 2 http://localhost:1234/v1/models >/dev/null 2>&1 && echo "LM Studio on :1234"
curl -s -m 2 http://localhost:11434/api/tags  >/dev/null 2>&1 && echo "ollama on :11434"

Both LM Studio and ollama speak the OpenAI-compatible API, so both list models the same way and both take a /v1 base URL. Only the port differs:

# honor an explicit QSV_LLM_BASE_URL first; only probe local servers when it is unset
BASE_URL="${QSV_LLM_BASE_URL:-}"
[ -z "$BASE_URL" ] && curl -s -m 2 http://localhost:1234/v1/models  >/dev/null 2>&1 && BASE_URL=http://localhost:1234/v1
[ -z "$BASE_URL" ] && curl -s -m 2 http://localhost:11434/api/tags >/dev/null 2>&1 && BASE_URL=http://localhost:11434/v1

[ -n "$BASE_URL" ] && curl -s "$BASE_URL/models" \
  | python3 -c 'import sys,json;[print(m["id"]) for m in json.load(sys.stdin)["data"]]'

/api/tags is only a liveness probe for ollama — it returns ollama's native shape, not the OpenAI {"data":[...]} envelope. List models from /v1/models either way.

If nothing is found, use AskUserQuestion for base URL + model. Never guess a model name. Offer the models the server actually reports; do not type one from memory.

Generate

First ask with AskUserQuestion: "Who is the Data Schematic's guided Tour for?" — default TOUR_AUDIENCE="Explain like I'm 10"; any free-text audience works ("a board of directors", "data journalists", …).

qsv describegpt "$WORK" \
  --dictionary --description --two-pass --infer-content-type \
  --format JSONSchema \
  --tour-audience "$TOUR_AUDIENCE" \
  ${BASE_URL:+--base-url "$BASE_URL"} --model "$MODEL" \
  -o "$SCHEMA"

• --infer-content-type is mandatory here, not optional: viz smart routes panels off each field's role and concept, and those are only inferred under this flag. Without it the dictionary loads and changes nothing. It is also the only way to get the three dictionary hints that unlock extra panels: per-field x-qsv.gauge_range (turns a measure's KPI tile into a gauge; kept only when the observed data lies inside the range), per-field x-qsv.denominator on a region column (adds a per-capita rate map beside the raw count map), and the dataset-level x-qsv.relationships array, whose "kind": "pipeline" entry is the only source of the pipeline funnel/bridge panel.
• x-qsv.denominator is the odd one out: qsv derives it, the LLM does not propose it. The model's whole contribution is tagging one column measure.population (a count of people or households IN a region another column names); describegpt then attaches the hint to every region column that can hold it, checking from the stats cache alone that the counts are plausible. Two population-shaped columns are an ambiguity it refuses rather than guesses at, so you get nothing. This is what makes a rate map reachable without hand-editing the JSON or paying for a --denominator census fetch of a number the file already carries.
• Pass --context-file <file> when the user has a glossary, README or codebook. Better context yields better roles, concepts and labels, hence a better Data Schematic. (viz --dictionary-context is the same thing for the infer path, which this skill does not take.)
• --two-pass roughly doubles cost and latency. It is what lets the model relate fields to one another (street_no + street + city + zip = one address), which is what makes the routing good.
• Naming it <WORK stem>.schema.json means a later qsv viz smart "$WORK" --dictionary infer finds and reuses it instead of paying for the LLM again. Delete the file to force a re-infer.
• --tour-audience makes describegpt also write a dataset-level x-qsv.tour narration — the prose the Data Schematic's guided Tour speaks — in the audience's register. The audience shapes ONLY the tour prose; labels and descriptions keep their normal register. Stage 6 refines it in a browser.

Optionally add --infer-null-values to have the model propose null sentinels into each property's x-qsv object, split into null_values (confirmed present by qsv) and null_candidates (guesses, each stamped confirm_required: true). This is the only route to numeric sentinels like -999. It is reported, never applied — nothing downstream acts on it.

Verify the dictionary carries what viz needs before spending time on Stage 4:

python3 - "$SCHEMA" <<'PY'
import json, sys
s = json.load(open(sys.argv[1]))
p = s["properties"]
have = sum(1 for v in p.values() if v.get("x-qsv", {}).get("role"))
print(f"role/concept on {have}/{len(p)} columns")
if have == 0:
    print("WARNING: no roles inferred — was --infer-content-type passed?")
# panel-unlocking hints, so the user knows up front what will/won't be drawn
gauges = [k for k, v in p.items() if (v.get("x-qsv") or {}).get("gauge_range")]
# a derived denominator turns the region map into a count map PLUS a rate map
denoms = {k: ((v.get("x-qsv") or {}).get("denominator") or {}).get("column")
          for k, v in p.items()}
denoms = {k: c for k, c in denoms.items() if c}
# viz reads pipelines ONLY from the dataset-level x-qsv (see xq_pipelines in
# src/cmd/viz.rs) — a root-level "relationships" array draws nothing.
rels = (s.get("x-qsv") or {}).get("relationships") or []
pipes = [r for r in rels if r.get("kind") == "pipeline"]
print(f"gauge_range on {len(gauges)} measure(s): {', '.join(gauges) or '(none)'}")
print(f"denominator on {len(denoms)} region column(s): "
      + (", ".join(f"{k} -> {c}" for k, c in denoms.items()) or "(none)")
      + ("   [rate map]" if denoms else ""))
print(f"relationships: {len(rels)} ({len(pipes)} pipeline -> funnel/bridge panel)")
if not rels and s.get("relationships"):
    print("WARNING: relationships found at the ROOT, not under x-qsv — viz ignores"
          " those. Is this the flat JSON dictionary instead of JSONSchema?")
tour = (s.get("x-qsv") or {}).get("tour")
if tour:
    print(f"tour: version {tour.get('version')}, audience {tour.get('audience')!r}, "
          f"{len(tour.get('overrides') or {})} override(s), "
          f"{len(tour.get('panels') or {})} panel narration(s)")
else:
    print("tour: (none — was --tour-audience passed?)")
PY

No gauge_range, no denominator and no pipeline is a perfectly normal outcome — most datasets have none of a canonical-scale measure, a per-region population, or a staged process. Say so and move on; all three can be hand-added later (see Stage 2.5).

Stage 2.5 — Fine-tune the dictionary (optional, TUI)

describegpt is a good first draft, not gospel — and the draft is not even stable: because the semantic half comes from an LLM, inferring twice over the same data can return different role/concept assignments, and role decides which panel a column gets (qsv issue #4407). This stage is what makes a Data Schematic reproducible. The corrected dictionary — reviewed, kept beside the data, committed if the data is versioned — is the artifact of record; every later run reuses it instead of re-rolling the model.

The five fields that actually steer viz smart — x-qsv.role, x-qsv.concept, title (label), description and x-qsv.aggregation — are worth a human pass when the model mislabels a column: a code that should be an identifier charted as a measure, a geo.* key left unknown, a per-unit price summed into a meaningless total, a bland label. edit_dictionary.py (beside this SKILL.md) is a curses UI that walks every column and, as you edit, previews how viz smart will route it (Skip / Dimension / Temporal / MapCoord / ProjectedCoord / Measure — the last showing its aggregation, Measure(sum) for an additive amount, Measure(mean) for a ratio, a duration, or anything you tag aggregation: mean), so you see the effect before rendering. It touches only those five fields, preserves every other key, and rewrites the file only if you save.

aggregation is the one field qsv can also drop on read, and the ! flag mirrors exactly when that happens: the token must be sum/mean, x-qsv.qsv_type must be Integer/Float (or absent), x-qsv.role must be empty or exactly measure, and the column must route to a measure at all. A value failing any of those is flagged and left out of the ROUTE preview, because viz silently falls back to its own name heuristic there. It catches the three easy mistakes: "average" instead of "mean", an aggregation left behind on a column you just re-roled to dimension, and one on a column nothing classifies (Defer→stats), where viz's stats floor discards it outright.

Offer it with AskUserQuestion: "Hand-tune the data dictionary in a TUI before rendering?" If no, go straight to Stage 3 — but say plainly that the Data Schematic then rests on an unreviewed draft, and that the dictionary can be tuned and re-rendered at any time without paying for the LLM again.

If yes, you cannot drive it yourself — a curses TUI needs the user's real terminal, and your Bash tool is a captured, non-interactive shell (the script detects this and refuses). So run it out-of-band:

1. Show the current routing so the user knows the starting point (this works without a TTY):

   python3 "$SKILL_DIR/edit_dictionary.py" --summary "$SCHEMA"
   

   where $SKILL_DIR is this skill's own directory (the folder holding this SKILL.md).

2. Tell the user to run this in their own terminal, then end your turn and wait — do not proceed:

   python3 "<skill dir>/edit_dictionary.py" "<SCHEMA path>"
   

   Keys: ↑↓ move · r role · c concept · l label · d description · a aggregation · s save · q quit. role/concept open a filterable picker (type to filter; off-vocab values are allowed but flagged with *). aggregation offers sum / mean / clear only — that is the whole vocabulary qsv accepts — and clearing removes the key rather than blanking it, restoring qsv's own guess.

3. When the user says they're done, re-read the file: re-run the Stage 2 coverage check and the --summary above, and show a short before/after of any rows whose role/concept/route changed. Then continue to Stage 3.

Because the dictionary keeps its <WORK stem>.schema.json name, Stage 4 picks up the edited file with no extra wiring. If the user edits nothing, the file is untouched byte-for-byte — treat that as a normal "looks good" outcome.

Scope note: the TUI deliberately does not edit null sentinels (--infer-null-values output). Those are reported-never-applied and have no viz smart effect, so editing them here would change nothing downstream.

Seven keys that do affect the Data Schematic are outside the TUI, and are hand-edited in the JSON — this is the supported path for them, not a violation of the "never hand-write the schema" rule. (x-qsv.denominator is the one qsv now normally fills in for you; hand-editing it is a correction, or the route to an area/household denominator describegpt will not derive.)

(x-qsv.aggregation used to be a fifth row here; it is now edited with a in the TUI above. Its meaning is unchanged: sum or mean on a numeric measure, declaring how the column combines across a group and overriding qsv's column-NAME heuristic in both directions. Use mean for anything per-unit or per-record — a unit price, a rating, a temperature, a duration — and sum only for a quantity each row contributes. It is the language-neutral signal: the name heuristic is English-first and cannot read non-English column names, issue #4401.)

One exception to "overrides in both directions" (issue #4528): a REGION-LEVEL column — one a region declares as its x-qsv.denominator, or one tagged measure.population/measure.area — holds a value that repeats identically on every row of its region. viz smart collapses it to one value per owning region before aggregating, and that collapse outranks an explicit x-qsv.aggregation. #4401's precedence is about what a measure MEANS (extensive vs intensive); this is about the data's SHAPE — the rows are duplicates of one regional value, and even a genuinely extensive measure must not be counted once per duplicate. So writing aggregation: sum on a population column does not restore a row-wise total. On tidy one-row-per-region data the collapse is a no-op. Two consequences to expect: its grouped bar shows the region's own figure (or, grouped by something coarser, the sum of the distinct regional values), and its KPI tile is omitted entirely when regions repeat, because no available aggregation can state a true dataset-wide total without a region-keyed dedupe pass.

For a pipeline, both encodings are hand-editable — stages as columns ("members" in process order, widest/upstream first, the opposite direction from "kind":"ordered"), or stages as row values ("stage_column" + an ordered "stages" list + an optional "value_column" to sum). Declared order is authoritative: if a stage outruns its predecessor, viz draws a bridge of signed differences instead of a funnel, rather than a band wider than the one above it. Offer these edits only when the Stage 2 check showed a plausible candidate; do not invent a target.

Stage 3 — GeoJSON (optional)

Ask with AskUserQuestion: "Bin rows into GeoJSON regions?"

If no, skip to Stage 4 with no geo flags.

If yes:

3a. Check the data can actually be binned

viz smart reaches a region two ways, and only one of them needs coordinates:

• Region key — --locations <col> (with --location-mode) where the column already identifies the region: an ISO-3 country code, a 2-letter US state code, a country name, a GeoJSON feature id, county FIPS/GEOID. No coordinate pair required. This is the path that works against a custom --geojson file, and the feature id must JOIN these values (Stage 3c).

• Place NAME — two different routes, and one precondition that governs both. The precondition: geo.city/town/municipality columns are nominated as region candidates only on a geocode-enabled build (geocodable_name_candidates returns nothing otherwise, and degrades silently). Check qsv --version for geocode before promising either route.

  • Direct, with any --geojson file whose feature ids ARE those names: an ordinary join, no aliases, so Stage 3c's overlap check settles it. examples/viz/nyc_neighborhoods.geojson (keyed by properties.name) is exactly this shape.
  • Alias-based city → county FIPS, --geojson auto/census only — the alias map is synthesized by automatic Census resolution, so a custom file publishes none. viz smart drives this itself: it forward-geocodes implicitly and does not need the --geocode flag, which is why Stage 4 not passing --geocode is no obstacle. (The explicit flag is a viz choropleth concern and is restricted to the iso3/usa-states location modes.)

  So do not reject a name column against a custom file — test the overlap first. Require a region-CODE column only when the names do not join, or when the build has no geocode support.

• Point-in-polygon — --lat/--lon, each row's coordinates tested against the polygons.

Check BOTH before concluding anything, and check the dictionary first — viz smart identifies its region column from the dictionary's concepts, not from header spelling, so $SCHEMA is the authoritative answer and a header regex is only a fallback:

# authoritative: only these geo.* leaves key a polygon. NOT every geo.* concept does -
# geo.latitude/longitude/coordinate_pair/street_address/ip_address/timezone/geonames_id name a
# point or an attribute, and treating them as region keys is the same false positive inverted.
python3 - "$SCHEMA" <<'PROBE'
import json, sys
# mirrors viz.rs REGION_CODE_LEAVES + CITY_NAME_LEAVES
CODE = {"zip_code","zip","postal_code","zcta","census_tract","county","county_fips","state",
        "state_fips","country","country_code","place_fips","fips"}
CITY = {"city","town","municipality"}   # need a geocode-enabled BUILD, not the --geocode flag
props = json.load(open(sys.argv[1])).get("properties", {})
con = {k: str((v.get("x-qsv") or {}).get("concept", "")) for k, v in props.items()}
leaf = lambda c: c.split(".", 1)[1] if c.startswith("geo.") else None
code = [k for k, c in con.items() if leaf(c) in CODE]
city = [k for k, c in con.items() if leaf(c) in CITY]
pair = ([k for k, c in con.items() if leaf(c) == "latitude"],
        [k for k, c in con.items() if leaf(c) == "longitude"])
print("region-code columns :", code or "(none)")
print("city-name columns   :", city or "(none)")
if city:
    print("  ^ nominated as region candidates only on a geocode-enabled BUILD")
    print("    (check `qsv --version` for `geocode`). NOT the --geocode flag,")
    print("    which `viz smart` never needs - see 3a above for the two routes.")
print("lat/lon PAIR        :", "yes" if all(pair) else "no  # a lone lat or lon bins nothing")
PROBE

Do not reach for a region-name regex: it cannot spell every geography (tract, zcta, municipality, town, iso3 all miss a state|county|country pattern), and a false negative here is exactly the mistake this stage used to make. Only when the probe finds neither a region-code column (nor a city/place-name column — which on a geocode-enabled build either joins a custom GeoJSON keyed by those names directly, or resolves to county FIPS on the auto path) nor a complete lat/lon pair does the GeoJSON have no effect — a lone geo.timezone or geo.ip_address is not a region key and does not count — say so then, and offer to proceed without it. A dataset carrying county names (or FIPS codes) and no coordinates at all maps perfectly well, so do not talk the user out of it.

3b. Get the file

Accept a local path, an http(s) URL, or a shortcut name defined in QSV_GEOJSON_SHORTCUTS (a JSON map of name → {path, id}; the shortcut's id supplies --feature-id-key when you don't pass one).

There is also a fourth form, and for US data it is usually the right one: --geojson auto (or census) fetches US county, ZIP Code Tabulation Area, census tract or place boundaries from the Census TIGERweb service, scoped to the states the data names, and sets --feature-id-key to properties.GEOID itself. The user supplies nothing but the CSV — no file to source, and Stage 3c below is unnecessary. Prefer it over hunting for a boundary file. To chart a rate rather than a count, pair it with --denominator census only for county or state maps - Census denominators exist for those two geographies alone, and the fetch hard-errors without a free QSV_CENSUS_API_KEY. For a ZCTA, tract or place layer, get the denominator from the data instead: a measure.population column in the dictionary (describegpt derives x-qsv.denominator from it), or --denominator-key pointing at a boundary property the fetched features carry.

3c. Discover the feature id key — do not guess it

--feature-id-key defaults to id, which is usually wrong. In viz smart's point-in-polygon mode the key labels each binned region, so it must be present on every feature, unique across all of them, and meaningful to a human. Uniqueness alone is not enough: properties.shape_area is perfectly unique and completely useless as a label.

⚠️ That "meaningful to a human" rule is the point-in-polygon rule, where the key LABELS each binned region. On the region-key path it is the wrong test and will cost you the choropleth: there the key must join — its values have to overlap the distinct values of the region column. Pick the key whose values match the CSV (GEOIDs match GEOIDs, names match names), verify the overlap before rendering, and put the human-readable property in --feature-name-key instead, which exists precisely to supply hover labels. Choosing a display name here while the CSV holds GEOIDs resolves nothing and renders no map.

Region-key or name path — ask qsv, do not reimplement the match

qsv viz --check-geojson-key scores every candidate feature-id path against the distinct values of the region column and ranks them by overlap. Use it and take its answer:

qsv viz choropleth "$WORK" --locations "$REGION_COL" --geojson "$GEOJSON" --check-geojson-key

3221 distinct --locations values scored against --geojson 'counties.geojson':
     3221/3221 100.0%  properties.GEOID
        0/3221   0.0%  properties.NAME  (unmatched e.g. 01001, 01003, 01005)

Use: --feature-id-key properties.GEOID

It scores through the same matcher the render path binds with, so a path it reports as a full match will bind at render time — zero-padding (6 vs 06037), ASCII case folding, and the refusal to guess between ambiguous folds (with features CA and ca, the value Ca matches neither) all come out identical by construction. That is why this replaced a hand-written scorer here: a reimplementation that is one tier more generous than viz reports a join you will not get.

It reads the same --geojson sources viz does (local path, http(s) URL, or a QSV_GEOJSON_SHORTCUTS name) and needs no valid --feature-id-key to run — finding one is its job. Candidates include nested paths under properties at ANY depth and top-level foreign members, not just properties.<field>, so a boundary file that keys off either is still scored. Treat a partial match as a warning, not a pass. If nothing matches, the region values and the boundary file disagree, or that GeoJSON cannot key them; say so rather than rendering an empty map.

⚠️ Skip this check entirely when $GEOJSON is an automatic Census spec — auto, census, census:<layer> or either with an @<year> vintage (Stage 3b's fourth form). Every one of them is refused, deliberately. Automatic Census resolution picks the feature-id key itself (properties.GEOID) and prints its own region coverage, so there is nothing to discover; and a place-NAME column binds through an alias map the check does not model, which would score every candidate at 0% on a setup that renders correctly. Go straight to Stage 4 and read the coverage line viz reports.

Point-in-polygon path — rank by uniqueness and readability

There is no join to test on this path (rows are binned by geometry), so the question is which property makes a good label. That is a judgement about the boundary file alone, which the script below answers.

It mirrors exactly one rule from viz: build_pip_features skips features without Polygon/MultiPolygon geometry, so the script ranks over those features only. Rank over every raw feature instead and a skipped point that duplicates an id makes that id look non-unique, which reports a file as unkeyable on a key that would have labelled every binned region. That one geometry check is the only parity this script needs — the match tiers it used to reimplement now live behind --check-geojson-key above.

It accepts the same file source forms --geojson does — a local path, an http(s) URL, or a QSV_GEOJSON_SHORTCUTS name. If you only handle local paths here, a URL or shortcut fails at discovery even though viz would have accepted it.

python3 - "$GEOJSON" <<'PY'
import json, sys, re, os, collections, urllib.request

def load_geojson(src):
    """Local path, http(s) URL, or a QSV_GEOJSON_SHORTCUTS name.

    Mirror viz's resolution order (src/cmd/viz.rs resolve_and_validate_geojson): an
    http(s) URL or an EXISTING local file is a direct source; only a value that is
    neither is looked up as a shortcut NAME. This keeps a local file whose name
    collides with a shortcut loading as the file (as viz does), and it never lets
    a malformed QSV_GEOJSON_SHORTCUTS break a direct file/URL input.
    """
    hint = None
    is_url = src.startswith(("http://", "https://"))
    if not is_url and not os.path.isfile(src):
        raw = os.environ.get("QSV_GEOJSON_SHORTCUTS")
        if not raw:
            sys.exit(f"--geojson '{src}' is not an existing file or http(s) URL, "
                     "and QSV_GEOJSON_SHORTCUTS is not set")
        shortcuts = json.loads(raw)           # invalid JSON surfaces as an error
        if src not in shortcuts:
            sys.exit(f"unknown --geojson shortcut '{src}'; "
                     f"defined: {', '.join(sorted(shortcuts)) or '(none)'}")
        entry = shortcuts[src]
        hint = entry.get("id")                # shortcut may carry its own id key
        src = entry["path"]
        is_url = src.startswith(("http://", "https://"))
    if is_url:
        with urllib.request.urlopen(src, timeout=30) as r:
            return json.loads(r.read().decode("utf-8")), src, hint
    with open(src) as fh:
        return json.load(fh), src, hint

g, resolved, hint = load_geojson(sys.argv[1])
feats = g.get("features", [])
if not feats:
    sys.exit("no features")
# Rank over the features viz will actually BIN. build_pip_features skips anything without
# Polygon/MultiPolygon geometry, so ranking over every raw feature reports a key as non-unique
# whenever a skipped point duplicates it - and then declares the file unusable on a key that
# would have labelled every binned region perfectly.
feats = [f for f in feats
         if (f.get("geometry") or {}).get("type") in ("Polygon", "MultiPolygon")]
if not feats:
    sys.exit("no Polygon/MultiPolygon features - viz cannot bin rows into this file")
print(f"source: {resolved}")
if hint:
    print(f"shortcut supplies --feature-id-key {hint} (override below if you prefer)")

# Geometry-derived / bookkeeping fields: unique, but meaningless as a region label.
NOISE = re.compile(r"shape|area|leng|length|perim|acres|sqmi|aland|awater|"
                   r"intptlat|intptlon|^lat|^lon|_x$|_y$|"
                   r"date|time|edited|created|updated|version", re.I)

def floatish(v):
    return isinstance(v, float) or (isinstance(v, str) and re.fullmatch(r"[+-]?\d+\.\d+", v.strip()))

cands = collections.defaultdict(list)
for f in feats:
    if f.get("id") is not None:
        cands["id"].append(f["id"])
    for k, v in (f.get("properties") or {}).items():
        if isinstance(v, (str, int, float)):
            cands[f"properties.{k}"].append(v)

good, other = [], []
for key, vals in cands.items():
    if len(vals) != len(feats):                    # missing on some feature
        continue
    if len(set(map(str, vals))) != len(feats):     # not unique
        continue
    demote = bool(NOISE.search(key)) or all(floatish(v) for v in vals)
    (other if demote else good).append((key, vals[:3]))

def show(title, rows):
    print(f"\n{title}")
    if not rows:
        print("  (none)")
    for key, sample in rows:
        print(f"  {key:<32} e.g. {sample}")

print(f"{len(feats)} usable (polygon) features")

show("RECOMMENDED feature-id-key (unique, meaningful):", good)
show("Unique but geometry/bookkeeping - avoid:", other)
if not good and not other:
    print("\nNo property is unique across all features. This GeoJSON cannot key regions as-is.")
PY

What you offer via AskUserQuestion depends on the path Stage 3a identified:

• Region key or name path — do not offer this ranking at all. Take --check-geojson-key's highest scorer, ideally a full match. Readability is irrelevant here and actively misleading: a numeric properties.GEOID/OBJECTID that joins is correct, while a pretty properties.hood that joins nothing renders an empty choropleth.
• Point-in-polygon path — offer the RECOMMENDED keys and favour a short region code or name (properties.nta2020, properties.hood) over a surrogate key (properties.OBJECTID, a GUID): here the value really does label each binned region.

If nothing is unique, say so plainly: the GeoJSON cannot key regions as-is.

Then pick --feature-name-key (e.g. properties.name) for human-readable hover labels — that is where a readable property belongs. Stage 4 passes it only when you set FEATURE_NAME_KEY.

Stage 4 — Render

Ask for --dataset-pid with AskUserQuestion (a persistent identifier — a DOI, ARK, Handle, or a URL). It is optional; allow the user to skip it.

qsv viz smart "$WORK" \
  --smarter --bivariate \
  --dictionary "$SCHEMA" --dict-info \
  ${GEOJSON:+--geojson "$GEOJSON"} \
  ${FEATURE_ID_KEY:+--feature-id-key "$FEATURE_ID_KEY"} \
  ${FEATURE_NAME_KEY:+--feature-name-key "$FEATURE_NAME_KEY"} \
  ${DATASET_PID:+--dataset-pid "$DATASET_PID"} \
  -o "$OUT"

• --smarter runs qsv moarstats --advanced first, enriching the stats cache with distribution shape (bimodality, entropy, skewness, outlier share, Gini — the last unlocks Lorenz curves for the most unequal additive measures). Costs one extra pass and writes <stem>.stats.csv + sidecars + .idx. It applies only under default parsing: --no-headers or a custom --delimiter silently falls back to the standard Data Schematic.
• --bivariate adds a normalized-mutual-information heatmap plus — only when there are more than 8 chartable columns — a ranked "top relationships" bar. It implicitly turns on --dictionary infer when --dictionary is not set — so passing $SCHEMA explicitly is what stops viz from calling the LLM a second time. Never pass --bivariate without a dictionary in this workflow. Capped at 50 columns; wider datasets skip both panels with a warning.
• --dict-info embeds the dictionary in a side drawer next to the plots, adds an info icon per panel, and a "Data Dictionary" link under the title. The drawer also carries download buttons for the sidecars this run actually read — the schema, the charted frequency counts, the stats cache + metadata, and the bivariate CSV — all bundled into the HTML, so a recipient needs no access to your machine. Absolute local paths are stripped from the embedded metadata (sharing a Data Schematic does not disclose your directory layout); sidecars over 4 MB are skipped with a note. HTML only — ignored with a note when exporting an image.
• -o must end in .html. An image extension (.png, .svg, …) silently switches viz to the static-export path, which needs a browser/webdriver and drops --dict-info.

The data viewer drawer (--preview-threshold, default 50000)

Independent of --dictionary/--dict-info: an (Explore) link beside the row count in the metadata table opens the underlying rows in a searchable bottom drawer. Every row is embedded while the dataset has at most <n> rows; above that only the first <n> are, and the link reads (Preview).

This is the one flag here with a real cost: embedded rows grow the HTML — and the reader's browser memory — in proportion to rows × columns. Tell the user the size (Stage 5 prints it) rather than letting them discover it. Lower the threshold, or pass --preview-threshold 0 to drop the viewer entirely, when the Data Schematic is meant to be emailed around.

--photos — ask first, never enable silently

If a column holds image URLs, --photos makes dwelling on a map point reveal that row's photo. It is off by default and deliberately so: images load from whatever third-party

…(truncated)