Transform
Author and refactor the dbt project: both the SQL transformations (staging to
marts, tests, docs) and the semantic layer on top (entities, dimensions,
measures, metrics). Both are the same job, writing reviewable diffs to the dbt
project, which is the source of truth. This is the building half of the loop. It
writes only to the repo, as reviewable diffs, and runs against a dev target only.
How to drive it
uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" <subcommand> [flags]
dex runs its engine through uv, which is a prerequisite and is not installed by
Claude Code. If the shell reports uv: command not found, stop and tell the user
to install it (curl -LsSf https://astral.sh/uv/install.sh | sh, or
brew install uv, or pipx install uv), then re-run. Never fall back to editing
the dbt project by hand instead: the validation, the diffs, and the dev-target
gating live in the engine, so any other path is unguarded.
The first command in a fresh environment installs the engine, so it can take tens
of seconds where later ones take well under a second. --warm pays that install up
front and exits without running anything:
uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" --warm
Offer it once at setup. It is not something to run before an ordinary command.
You author the dbt file content; the engine validates it, computes the diffs,
and stores the proposal as a plan. Hand content over with --edits-file <path>
(or - to read stdin), a JSON payload:
{"edits": [
{"path": "models/staging/stg_orders.sql", "kind": "model_sql", "content": "..."},
{"path": "models/staging/stg_orders.yml", "kind": "schema_yml", "content": "..."},
{"path": "snapshots/snap_orders.sql", "kind": "snapshot_sql", "content": "..."},
{"path": "seeds/country_vat.csv", "kind": "seed_csv", "content": "..."},
{"path": "tests/assert_totals_reconcile.sql", "kind": "test_sql", "content": "..."},
{"path": "analyses/email_skew.sql", "kind": "analysis_sql", "content": "..."},
{"path": "models/marts/dim_orders.sql", "kind": "model_sql", "op": "delete"}
]}
kind is model_sql, schema_yml, semantic_yml (optional on
semantic define|update|plan, which imply it), macro_sql (a macro file under
the project's macro paths), snapshot_sql (a snapshot under the snapshot
paths), seed_csv (a seed's CSV under the seed paths), test_sql (a singular
test or a generic test definition under the test paths), analysis_sql (SQL dbt
compiles but never runs, under the analysis paths), packages_yml,
project_yml (the project-root dbt_project.yml), or profiles_yml (the
project-root profiles.yml). Model SQL must be a single read-only SELECT once
its jinja is stripped; semantic YAML is validated against MetricFlow's schemas,
cross-reference-checked, and (when dbt is available) parsed by dbt itself before
the plan is accepted; a macro file must hold only macro definitions and jinja
comments. A snapshot must hold exactly one {% snapshot %} block whose
config() names a unique_key and a strategy of timestamp (with
updated_at) or check (with check_cols), and whose body is a single
read-only SELECT. A seed must parse as CSV with a named, duplicate-free header
and one field per column on every row, and stays under 5,000 data rows and 1 MiB
(past that it is data rather than a lookup: load it into the warehouse and
source() it). A test_sql file is read to decide which of the two shapes
sharing the test paths it is: one holding {% test %} blocks is a generic test
definition and must hold only those and jinja comments, balanced; anything else
is a singular test and must be a single read-only SELECT. A singular test that
names no ref() or source() is warned about, not refused, because it runs
against nothing and passes unconditionally. An analysis must be a single
read-only SELECT too, even though dbt only compiles it. project_yml must keep
a name; profiles_yml must reference
every secret via {{ env_var('NAME') }} (a literal credential is refused so
none reaches the diff). Config kinds, snapshots and seeds are all parsed by dbt
at plan time.
Each kind is confined to its own family of paths, and filing one in the wrong
family is refused naming both fixes. schema_yml is the exception, accepted
beside a model, a snapshot, a seed, a test or an analysis, because that is where
dbt expects a snapshot's tests, a seed's column types, a singular test's severity
and an analysis's description declared.
Three things here are called a test, and they are not interchangeable.
Generic tests are declared inside a schema.yml (data_tests: on a model or a
column). Unit tests come from transform test --scaffold <model>, which writes a
unit_tests: block, also schema_yml. Singular tests and generic test
definitions are files under test-paths, and test_sql is the kind for those.
A seed puts values, not logic, into a diff, and a diff goes into git and stays
there. So a seed whose header names a column that looks like personal data is
refused, and the refusal names the pii_overrides entry in .dex/config.yml
that a human can add to clear it. Detection reads names and types and never
values (everywhere in dex), so it cannot see personal data hiding under a
neutral column name: do not build a seed out of warehouse rows you have not
looked at.
dbt build runs seeds, snapshots and singular tests natively, so transform build after an apply is all it takes; there is no separate seed or test step. A
snapshot writes a table and a test runs a scanning SELECT, so both are priced in
the cost handshake; a seed scans nothing and an analysis is never built at all,
so neither is. A singular test and an analysis build no relation and nothing can
ref() either, so neither is a node: neither enters maintain's drift baseline,
and deleting one raises no dangling-reference guard.
op is upsert (the default: create or update, carrying content) or
delete (remove the file, no content). A delete is a first-class reviewable
diff like any other edit, so a reclassification or refactor is one plan rather
than a plan plus a manual rm. Deletes are guarded: the plan is refused if any
file that survives it still ref()s a deleted model, naming the offenders.
Carry the edits that remove those references in the same plan (for a rename,
delete the old model, create the new one, and update every referrer to
point at it, all together) so the post-change project is validated as one unit.
An unconfirmed delete against a file a human edited after planning surfaces as
needs_confirmation, never a silent removal.
For a rename or a removal, reach for transform rename / transform remove
instead of assembling the edits yourself. They generate the whole change from the
reference graph and refuse when they cannot promise it is complete, which is the
guarantee hand-assembly cannot give you.
Bootstrapping a project
If no dbt project exists in the repo, offer transform init before anything
else: transform plan needs a project to edit. Ask the user for the project
name and confirm the connector with them, then run:
uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" transform init "<name>" --connector <c>
The engine renders the whole skeleton (dbt_project.yml, models/staging/ and
models/marts/, a profiles.yml with a single dev target and no secrets) and
records connector, dbt_project_dir, and dbt_target: dev in
.dex/config.yml; do not hand-write these files yourself. Init never assumes a
connector: it errors rather than defaulting, so always pass the user's confirmed
choice (a connector: already committed in .dex/config.yml also counts).
Every connector is supported: DuckDB, BigQuery, Snowflake, Databricks,
Postgres, Redshift, and ClickHouse. DuckDB needs a warehouse path (--path, or the
duckdb.path config). BigQuery needs a GCP project (usually
bigquery.project in .dex/config.yml; confirm it with the user) and writes
builds to a dedicated dev dataset (bigquery.dev_dataset, default
dbt_dev); auth is Application Default Credentials, so if credentials are
missing tell the user to run gcloud auth application-default login, never
ask for a key. Snowflake writes builds to a dedicated
snowflake.dev_database/dev_schema on the pinned warehouse; Databricks
writes builds to a dedicated databricks.dev_catalog/dev_schema on the
pinned SQL warehouse (if credentials are missing tell the user to run
databricks auth login, never ask for a token); Postgres writes builds to a
dedicated postgres.dev_schema (default dbt_dev), with the password
reaching dbt only through the PGPASSWORD environment variable. Redshift
writes builds to a dedicated redshift.dev_schema (default dbt_dev): with
a redshift.workgroup pinned the profile renders IAM auth (temporary
credentials from the AWS chain, nothing persisted), otherwise the password
reaches dbt only through the REDSHIFT_PASSWORD environment variable.
ClickHouse writes builds to a dedicated clickhouse.dev_database (default
dbt_dev), rendered as the profile's schema: because dbt-clickhouse has no
database: key, with the password reaching dbt only through the
CLICKHOUSE_PASSWORD environment variable; the rendered profile also carries
a custom_settings block whose env_var references are how transform build
turns the confirmed budget into a per-statement server-side cap, so do not
strip them from a profile you edit. All of them discover their connections and refuse with the fix named when none
resolves. Init refuses if any dbt project already exists.
When the user wants staging/intermediate/marts isolated in their own
datasets/schemas (a common ask when the warehouse is shared with unrelated
work), offer --layered-schemas: init then also scaffolds
models/intermediate/, a generate_schema_name override, and per-folder
+schema: config, so builds land in staging_dev / intermediate_dev /
marts_dev instead of one shared dev namespace. Do not hand-write that macro;
existing projects can adopt it later via transform macro generate_schema_name. Note dbt warns about "unused configuration paths" until
the first model lands in each layer folder; that resolves itself.
Init also checks (free, metadata-only) whether each namespace the project
would build into already exists with content, and warns naming the namespace
and a few object names. The warning is advisory: relay it to the user and ask
whether the content is theirs (a previous dev build) or unrelated; when it is
unrelated, discard the freshly scaffolded project (nothing has been built),
point the config at a different dev namespace, and re-run init. A "could not
check" note just means no connection was reachable at init time.
dbt SQL models
transform plan "<intent>" --edits-file <path|-> validates the edits and
returns them as diffs with a plan id. Nothing is applied yet. Add
--scaffold <table> (repeatable) to generate a staging skeleton
(stg_<table>.sql plus per-model YAML with key tests and PII meta) from the
.dex/ cache instead of, or on top of, hand-authored edits.
When you edit a model that already exists, the plan reports what your change
does to its row population under data.row_attribution: every predicate,
join, source and grain change is named, and each is measured on its own against
the prior model. Read it before applying. A change you were not asked to make
carrying a non-zero delta is the signal to look at: the model still compiles
and the columns are still right, and it is now returning a different set of
rows. It is advisory, never a refusal, because changing the filter is sometimes
the job. On DuckDB the deltas are measured automatically; on a billed connector
the changes are named for free and measuring them needs --attribute-rows
(then the usual --confirm --budget once priced), so ask the user before
spending. A change reported with attributed: false names why it could not be
measured; treat that as unknown, not as zero.
The plan also warns about the shape of what you authored, in warnings.
A SELECT list that diverges from the columns the model's schema.yml declares
is named in both directions; fix whichever side is actually stale, and say
which one you decided it was.
A warning that the model exposes a raw foreign key with no resolved
counterpart is dex reading a convention out of the project's own models: the
siblings it names all resolve keys of that shape, and it names the parent
model yours could resolve against. Prefer resolving it, by joining that parent
the way the siblings do. Where the raw key is deliberate (a fact-shaped model
in a dimension folder, a key the consumer needs verbatim), say so plainly to
the user rather than quietly leaving it. Never switch the check off to make
the warning go away: conventions.resolved_keys: false in .dex/config.yml
is a decision about the house's style, so recommend it for the user to accept,
the same way you would a pii_overrides entry.
transform apply [plan-id] writes the plan into the dbt project (the latest
unapplied plan when no id is given; any plan kind, semantic included). The
result is still a reviewable git diff for the user. If a human edited a file
after the plan was made, nothing is written: the divergence comes back as
diffs with needs_confirmation, and you should re-plan against current state
(or, only when the user says so, re-run with --confirm).
transform plans lists stored plans (pending and applied, newest first), so
you never need to browse .dex/plans/ by hand.
transform references <name> [more...] answers "where is this used" before you
change it. Reach for this whenever a change has to land in more than one
place: removing a project variable, renaming a column, deleting a model,
changing what a macro returns. Editing the files you happen to have open and
hoping that was all of them is the failure this prevents, and it is a quiet
one, because the project still compiles with one use left behind.
It is repo-only and free on every connector, so there is never a cost reason
not to run it. The positional is variadic, so one call covers a whole rename.
--kind narrows to model, source, seed, snapshot, macro, var,
column, metric, entity, dimension or measure; leave it off when you
are not sure what the project calls the thing, and the answer will tell you.
Read data.completeness before you act on the list. When it says incomplete,
data.limits says why and data.indeterminate lists the call sites dex could
not resolve, each with a file and a line. Those are references that may name
what you asked about, so open them and decide yourself; do not treat the list
of resolved hits as exhaustive when the verdict says it is not. A bare column
name is matched across the project (scope: name_matched), so qualify it as
model.column when you want the lineage separated from same-named columns
elsewhere.
Once you know where a name is used, transform rename and transform remove
below make the change; you do not have to carry the list into hand edits.
transform rename <kind> <old> <new> generates every edit the rename needs
and stores them as one plan: the definition, every model that selects the name,
every schema.yml that documents or tests it, every semantic reference, and a
seed header. Kinds are column, var, model, seed, snapshot, macro,
source. Repo-only and free, like references.
Use this instead of editing the files yourself. Retyping a rename across
nine files and missing the tenth is the failure mode this exists for, and it is
a quiet one: the project still compiles.
Name a column as model.column. A bare name is refused, and the refusal lists
the models that define a column of that name so you can pick. That asymmetry
with references is deliberate: a report you read can afford to be imprecise
and a rewrite cannot, because renaming a bare id project-wide would rewrite
every unrelated id there is.
It refuses rather than half-applying, and each refusal names what to fix:
a reference dex could not resolve statically, a name an installed package also
defines, a column handed to a macro as a literal string (dex cannot tell a
column argument from a display label), a SELECT list it cannot read. Fix what
it names and re-run. There is no override flag, because a completeness
guarantee you can switch off is a suggestion. A bare select * is not a
refusal: it carries the column through under the new name with no edit, and the
plan's notes says so.
Read data.sites against the transform references output you ran first. It
counts occurrences per reference form in the same vocabulary, so the two
agreeing is your evidence that nothing was dropped between reading and writing.
transform remove <kind> <name> removes the definition and verifies every
read is gone, refusing while any survives and naming each with a file and line.
It never rewrites a read, and that boundary is the point rather than a gap.
{% if var('using_department') %} can be deleted or unguarded, and
{{ var('x') }} sitting in an expression has no value dex may invent. You are
the one who knows. Author those edits yourself and pass them with
--edits-file in the same call: they are validated and stored in the same
plan, so the removal is still atomic.
transform place <column> --targets <a,b> --expr "<sql>" answers where a
derived column that several models need should be defined. It walks ref()
upward from every target, takes the lowest model they all descend from that
already projects the inputs your expression reads, defines the column there,
and threads it down every chain. The inputs come from parsing --expr, so
there is no separate list to get out of sync with it.
Read data.reasoning before you apply. It names the ancestor, why it is
the lowest, which targets descend from it, and the chain. You are supposed to
be able to disagree with it; --explain gives you the same answer with no plan
stored, which is the cheap way to ask.
When data.strategy is per_target the shared definition was not available
and the reasoning says why: no common ancestor, or the lowest one is missing an
input, or two candidates tie. dex will not go further upstream to pull an input
down, because that turns one placement into an unbounded rewrite of everything
above it. The fallback duplicates the derivation in each target and those
copies will drift, so relay the reason to the user rather than applying it on
their behalf. Often the named fix (add the missing column to the ancestor
first) is what they actually want.
transform build --target dev runs dbt build against a dev target. The
engine surfaces a cost preflight first and runs only with --confirm (plus a
--budget on billed connectors). dbt itself has no dry-run, but the engine
compiles the project and dry-runs each node itself, so on BigQuery the first
unconfirmed call already returns needs_confirmation with estimated_bytes
and a per_table_bytes breakdown, the same shape the scanning explore
commands use. Never invent a --budget figure: read the reported estimate
(per_table_bytes is the actionable half, since it names which node is
driving the cost) and confirm with a --budget grounded in that number.
If the build is refused over the ceiling, the refusal carries a calibration
line from .dex/spend.jsonl: what this connector's recent commands billed as
a fraction of estimate, or a sentence saying there is too little history to
say. Builds over-estimate most on a partitioned or clustered warehouse, so
relay it, and note that the ceiling binds on the estimate rather than on what
settles, so a budget set at that fraction of the estimate is refused again.
A suggested_session_ceiling on that envelope is the project's one-time ask
for a cumulative daily cap, separate from --budget: relay it and add the
user's answer (--session-ceiling <value> or --no-session-ceiling) to the
same re-issue, which records it in .dex/config.yml for good.
Each statement dbt runs is capped server-side by the profile's
maximum_bytes_billed, and the envelope reports billed bytes afterward.
Production-looking targets are refused
outright; --confirm cannot override that. dbt runs with its working
directory pinned to the project dir, so relative paths in profiles.yml
resolve against the project. When the project declares packages
(packages.yml) and dbt_packages/ is missing, the engine runs dbt deps
automatically before the build.
transform build --verify is how you answer "is it right", not just "did it
run". A green build tells you dbt executed. It does not tell you the model
holds the rows it should, and that is where the expensive defects live: an
inner join written where a left join was meant loses rows, raises nothing, and
passes every uniqueness and not-null test over the smaller result. --verify
sweeps the nodes this build touched and reports the findings in the same
envelope, under data.verification. Reach for it whenever the build was meant
to prove a change is correct, which is most of the time you build at all.
Read data.verification.ran before reading anything else. It is always
present, because a build that did not verify and a build that verified and
found nothing look identical otherwise, and only the second one means the
models are clean. When it ran, findings is ranked the way maintain verify
ranks it, scope names the models covered, and suppressed names each class
that could not run and why. Relay a suppression rather than reading past it:
it is the difference between "checked and clean" and "not checked".
Findings never fail the build and never appear in errors. Do not treat one
as a build failure or re-run to make it go away: relay the finding, its two
counts, and the join it names, and let the user decide. A failed build still
reports which node failed and which were skipped because of it, which is
usually a faster read than the dbt log.
On a billed connector the sweep is priced into the build's own estimate as a
(row counts) line, so the --budget you already read off the unconfirmed
envelope covers both. Never add a second budget for it. If the envelope comes
back ok with a data.offer, the build is done and billed and the offer buys
only the counts it could not afford; relay the number rather than re-running
the build.
transform deps installs dbt packages explicitly (also the refresh path when
dbt_packages/ exists but is stale). No confirmation needed: deps writes only
inside the project and never touches the warehouse.
Shipped macros
transform macro lists the macros dex ships; transform macro <name>
proposes scaffolding one into the project's macro directory as a plan,
applied with transform apply like any other. The user's copy is theirs to
edit; re-running the command diffs it back against the shipped version (a
warning says whether it is customized or stale), and applying that plan
overwrites deliberately.
unpivot_json_object turns a JSON object column with dynamic keys (the
NoSQL-sourced shape: a Firestore/Mongo/DynamoDB document keyed by a related
entity's id) into one row per top-level key. Use it instead of hand-rolling
JSON SQL; it renders a complete SELECT:
select id, key as related_id, value as attrs
from (
{{ unpivot_json_object(relation=ref('stg_entities'),
json_column='attributes', passthrough=['id']) }}
)
The contract on every connector: one row per top-level key, key a plain
string, value the warehouse's native semi-structured type (BigQuery JSON,
Snowflake VARIANT, Databricks VARIANT, Postgres jsonb, Redshift SUPER,
DuckDB JSON, ClickHouse raw JSON text in a String), a NULL object yields no
rows, and a nested object's own field
names never surface as top-level keys. For a string-typed source column
pass the parse expression as json_column (parse_json(payload) on
BigQuery, Snowflake, and Databricks; json_parse(payload) on Redshift);
Postgres, DuckDB, and ClickHouse accept JSON-bearing text directly. Databricks needs
VARIANT support (DBR 15.3+ or a current SQL warehouse). Two BigQuery quirks
are absorbed by the macro, so do not "fix" them back in: a JSON path
argument must be a compile-time literal (the macro reads values with the
subscript operator, which accepts a computed key), and JSON_KEYS recurses
into nested objects unless depth-limited (the macro pins depth 1). When a
planned model calls the macro and the project lacks it, the plan warns and
names the scaffold command; scaffold it rather than inlining a copy.
Preparing the dev target
Before the cost gate, and for free, transform build refuses two things and
names the fix for each. Neither costs anything to check, so both surface on the
unconfirmed call rather than after a budget has been agreed.
Config that has drifted from the profile. transform init renders
.dex/config.yml into the project's profiles.yml, and dbt reads only the
profile from then on. If a later config edit never reached it (a retargeted
dev_database, a different warehouse), the build refuses and names both values
and both files. Edit one to match the other. The engine never rewrites
profiles.yml, which you may legitimately have hand-edited.
A dev target that does not exist. On Snowflake, dbt creates schemas but never
databases, so a missing dev_database is refused with the CREATE DATABASE
statement to run; dex will not create it for you, because its only writes are
reviewable diffs inside the repo. On Postgres, Redshift, and ClickHouse, dbt creates the dev
namespace but only if the profile's user may, so the missing privilege is what
gets refused, with the CREATE SCHEMA/GRANT statement to run. On ClickHouse
that check can also come back with no verdict, because a server may not let dex
read another user's grants; it then warns instead of guessing, and the build
proceeds with dbt's own error as the backstop. On DuckDB the dev target is a database file,
and dbt would happily create an empty one, then fail every source() relation
with a confusing catalog error. The convention there: copy the shared source
warehouse to the dev target path (for example
cp shared/f1.duckdb <project>/dev.duckdb), or point the dev target at an
existing file. Projects without sources just get a warning and an empty
database, which is fine for model-only builds.
The semantic layer
semantic define ... and semantic update ... author and evolve the dbt
semantic models (entities, dimensions, measures, metrics) as plans. define
refuses names that already exist (use update); update refuses names that
do not (use define). For one logical change that mixes both (evolve existing
metrics and add the helpers they depend on), use semantic plan ...: it
accepts mixed intent and classifies each name, and the envelope reports the
split as defined, updated, unchanged, and removed.
Prefer --definitions-file over --edits-file for the semantic layer. A
real project keeps its metrics in one shared file, so a whole-file payload
means retyping every definition you are not touching: the diff and the
updated list then describe the whole file instead of your change, and every
restated line is a chance to corrupt a definition by hand. Send only what
changes instead:
{"definitions": [{"kind": "metric", "content": "name: ...\n..."}]}, where
kind is semantic_model or metric and content is that definition's YAML
body with no leading - . The name comes from the content, and path can be
omitted for anything the project already declares (the engine rewrites it
where it lives). Everything else in the file, comments included, is preserved
byte for byte. Reach for --edits-file when you are creating a file, moving a
definition between files, or emptying one, and when the engine refuses a layout
it will not splice into.
Removing one definition is the same payload with "op": "delete":
{"definitions": [{"kind": "metric", "name": "doubled", "op": "delete"}]}, the
name declared (there is no content to read it from) and no content beside it.
Nothing is removed for going unmentioned, so you can send a removal and an
edit in one payload and everything you did not name stays as it is. Use
semantic update or semantic plan, not define. The envelope reports it
under removed.
If a metric still reads what you are removing (its input is that metric, or a
measure of the semantic model you are removing), the plan is refused and the
reader is named: add that reader's own delete or update to the same payload,
in any order, and it goes through. A removal that would leave a file with no
semantic model or metric in it is refused too, because deleting or emptying a
file is a whole-file edit: do that with transform plan --edits-file and
"op": "delete".
unchanged means you re-stated a definition exactly as the project already
has it. It is not an error, but if a plan is entirely unchanged it changes
nothing, and the envelope warns as much: check whether you meant to edit
something.
Plan-time validation is layered so a plan that validates will build:
MetricFlow's schemas check the shape; the engine resolves every metric input
(ratio and derived metrics reference metrics, not measures; a measure only
becomes a metric via create_metric: true, and the error names that fix); and
finally the emitted YAML is run through dbt's own parser against a
throwaway copy of the project. A plan that fails parse is refused, not stored.
If dbt is not installed the parse degrades to a warning; --no-parse skips it
explicitly.
A semantic plan is applied like any other: transform apply [plan-id] writes
its YAML into the dbt project (no id applies the latest unapplied plan).
For native Ossie use semantic ossie define|update|plan with
--edits-file <path|->. Supply whole documents whose paths are listed in
semantic.ossie.files; the command implies the semantic_document kind,
validates the complete prospective configured layer, and writes the accepted
bytes exactly when the plan is later applied. This is a semantic-layer write
surface and does not make Ossie the transformation project.
The namespace guards match the dbt ones: define refuses a semantic-model name
the layer already has, update refuses one it does not, and plan accepts
both and reports each under defined or updated. Neither removes a model, and
a configured file may be absent before define, so a new document is planned
once its path is committed to config.
What validates an Ossie plan is not what validates a dbt one, and the
difference matters. There is no external parser to gate on: dex checks the
document's structure against the Ossie schema it pins (needs [ossie]), its
internal consistency in pure Python, and each SQL expression's syntax through
the dialect engine (needs [sql], which every connector extra carries).
Without [sql] the third layer degrades to a named skipped-validation note,
never to a silent pass. All three run over the complete prospective layer, your
edits overlaid on the other configured documents, before a plan is stored.
It then checks the references against the exploration cache, opening no
connection. A source relation the cached inventory positively lacks, or a
column absent from a relation the cache profiled, refuses and stores no plan.
Anything the cache cannot speak to is a named note instead: an unprofiled
relation, a computed or non-SQL expression, a quoted identifier, a query-backed
source. Read those notes rather than treating them as failures; they say what
was not checked.
Accepted bytes are written exactly as authored on apply. dex does not parse and
re-serialize the document, so comments, key order, quoting and whitespace all
survive, and a configured document the payload did not mention is untouched. A
target file that changed after planning refuses the whole apply rather than
writing part of it, unless you confirm the overwrite deliberately.
references/ossie-walkthrough.md in the engine repository runs the whole
sequence on a local warehouse if you want to see it end to end.
dbt cannot parse semantic models in a project without a MetricFlow time
spine; the engine warns when one is missing and defers the parse gate until
one exists. Author it like any other model (a day-grain date model plus YAML
with a time_spine: config) in the same or a separate plan.
viz preview is not yet implemented (it returns not_implemented); the Viz
integration arrives later.
Guardrails (enforced in the engine, not here)
- Writes confined to the repo, and within it to two disjoint surfaces: the dbt
project's authored path families (models, macros, snapshots, seeds, tests,
analyses) plus the project-root manifests dbt keeps there, and the exact
native semantic documents named in
semantic.ossie.files. Neither surface can
reach the other, an absolute path or a .. escape is refused on both, and dex
never writes to source warehouse data.
- Dev-target only. Prod-target execution is never initiated by dex.
- Cost surfaced before any spend. A build that would spend requires explicit
confirmation and a session budget.
- Propose, don't impose. Human edits to the project (SQL and semantic YAML) and
to a native semantic document are authoritative; on conflict the engine
surfaces a diff and asks rather than overwriting.
- PII flags propagate from the cache into emitted dbt (model and column
meta),
never example values. Stamping is presence-based at any confidence; only a
column cleared by a human pii_overrides entry in .dex/config.yml is
scaffolded without the meta.
1---2name: transform3description: Use this to author and change a dbt project or a semantic layer: bootstrap a project in a repo that has none (`transform init`), write or refactor model SQL from staging to marts, add tests and docs in schema.yml, manage dependencies, and define or update the semantic layer, whether that is dbt semantic models (MetricFlow: entities, dimensions, measures, metrics) or native Apache Ossie documents in a repo with no dbt project at all. Reach for this rather than editing model files by hand whenever the change spans more than one file or has to stay consistent with the rest of the project: it validates the edit against the real schema before writing, returns the change as a reviewable diff with a plan id, and catches the class of error that only surfaces at `dbt run`, such as wrong column names, broken refs, or a materialization that fights the project config. On a large project that check is worth more than the round trip costs. It applies to bug-fix tickets too: "this model returns wrong numbers, fix it" is a t4---56# Transform78Author and refactor the dbt project: both the SQL transformations (staging to9marts, tests, docs) and the semantic layer on top (entities, dimensions,10measures, metrics). Both are the same job, writing reviewable diffs to the dbt11project, which is the source of truth. This is the building half of the loop. It12writes only to the repo, as reviewable diffs, and runs against a dev target only.1314## How to drive it1516```bash17uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" <subcommand> [flags]18```1920dex runs its engine through `uv`, which is a prerequisite and is not installed by21Claude Code. If the shell reports `uv: command not found`, stop and tell the user22to install it (`curl -LsSf https://astral.sh/uv/install.sh | sh`, or23`brew install uv`, or `pipx install uv`), then re-run. Never fall back to editing24the dbt project by hand instead: the validation, the diffs, and the dev-target25gating live in the engine, so any other path is unguarded.2627The first command in a fresh environment installs the engine, so it can take tens28of seconds where later ones take well under a second. `--warm` pays that install up29front and exits without running anything:3031```bash32uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" --warm33```3435Offer it once at setup. It is not something to run before an ordinary command.3637You author the dbt file content; the engine validates it, computes the diffs,38and stores the proposal as a plan. Hand content over with `--edits-file <path>`39(or `-` to read stdin), a JSON payload:4041```json42{"edits": [43 {"path": "models/staging/stg_orders.sql", "kind": "model_sql", "content": "..."},44 {"path": "models/staging/stg_orders.yml", "kind": "schema_yml", "content": "..."},45 {"path": "snapshots/snap_orders.sql", "kind": "snapshot_sql", "content": "..."},46 {"path": "seeds/country_vat.csv", "kind": "seed_csv", "content": "..."},47 {"path": "tests/assert_totals_reconcile.sql", "kind": "test_sql", "content": "..."},48 {"path": "analyses/email_skew.sql", "kind": "analysis_sql", "content": "..."},49 {"path": "models/marts/dim_orders.sql", "kind": "model_sql", "op": "delete"}50]}51```5253`kind` is `model_sql`, `schema_yml`, `semantic_yml` (optional on54`semantic define|update|plan`, which imply it), `macro_sql` (a macro file under55the project's macro paths), `snapshot_sql` (a snapshot under the snapshot56paths), `seed_csv` (a seed's CSV under the seed paths), `test_sql` (a singular57test or a generic test definition under the test paths), `analysis_sql` (SQL dbt58compiles but never runs, under the analysis paths), `packages_yml`,59`project_yml` (the project-root `dbt_project.yml`), or `profiles_yml` (the60project-root `profiles.yml`). Model SQL must be a single read-only SELECT once61its jinja is stripped; semantic YAML is validated against MetricFlow's schemas,62cross-reference-checked, and (when dbt is available) parsed by dbt itself before63the plan is accepted; a macro file must hold only macro definitions and jinja64comments. A snapshot must hold exactly one `{% snapshot %}` block whose65`config()` names a `unique_key` and a `strategy` of `timestamp` (with66`updated_at`) or `check` (with `check_cols`), and whose body is a single67read-only SELECT. A seed must parse as CSV with a named, duplicate-free header68and one field per column on every row, and stays under 5,000 data rows and 1 MiB69(past that it is data rather than a lookup: load it into the warehouse and70`source()` it). A `test_sql` file is read to decide which of the two shapes71sharing the test paths it is: one holding `{% test %}` blocks is a generic test72definition and must hold only those and jinja comments, balanced; anything else73is a singular test and must be a single read-only SELECT. A singular test that74names no `ref()` or `source()` is warned about, not refused, because it runs75against nothing and passes unconditionally. An analysis must be a single76read-only SELECT too, even though dbt only compiles it. `project_yml` must keep77a `name`; `profiles_yml` must reference78every secret via `{{ env_var('NAME') }}` (a literal credential is refused so79none reaches the diff). Config kinds, snapshots and seeds are all parsed by dbt80at plan time.8182Each kind is confined to its own family of paths, and filing one in the wrong83family is refused naming both fixes. `schema_yml` is the exception, accepted84beside a model, a snapshot, a seed, a test or an analysis, because that is where85dbt expects a snapshot's tests, a seed's column types, a singular test's severity86and an analysis's description declared.8788**Three things here are called a test, and they are not interchangeable.**89Generic tests are declared inside a `schema.yml` (`data_tests:` on a model or a90column). Unit tests come from `transform test --scaffold <model>`, which writes a91`unit_tests:` block, also `schema_yml`. Singular tests and generic test92*definitions* are files under `test-paths`, and `test_sql` is the kind for those.9394**A seed puts values, not logic, into a diff, and a diff goes into git and stays95there.** So a seed whose header names a column that looks like personal data is96refused, and the refusal names the `pii_overrides` entry in `.dex/config.yml`97that a human can add to clear it. Detection reads names and types and never98values (everywhere in dex), so it cannot see personal data hiding under a99neutral column name: do not build a seed out of warehouse rows you have not100looked at.101102`dbt build` runs seeds, snapshots and singular tests natively, so `transform103build` after an apply is all it takes; there is no separate seed or test step. A104snapshot writes a table and a test runs a scanning SELECT, so both are priced in105the cost handshake; a seed scans nothing and an analysis is never built at all,106so neither is. A singular test and an analysis build no relation and nothing can107`ref()` either, so neither is a node: neither enters `maintain`'s drift baseline,108and deleting one raises no dangling-reference guard.109110`op` is `upsert` (the default: create or update, carrying `content`) or111`delete` (remove the file, no `content`). A delete is a first-class reviewable112diff like any other edit, so a reclassification or refactor is one plan rather113than a plan plus a manual `rm`. Deletes are guarded: the plan is refused if any114file that survives it still `ref()`s a deleted model, naming the offenders.115Carry the edits that remove those references in the same plan (for a rename,116`delete` the old model, `create` the new one, and `update` every referrer to117point at it, all together) so the post-change project is validated as one unit.118An unconfirmed delete against a file a human edited after planning surfaces as119`needs_confirmation`, never a silent removal.120121For a rename or a removal, reach for `transform rename` / `transform remove`122instead of assembling the edits yourself. They generate the whole change from the123reference graph and refuse when they cannot promise it is complete, which is the124guarantee hand-assembly cannot give you.125126### Bootstrapping a project127128If no dbt project exists in the repo, offer `transform init` before anything129else: `transform plan` needs a project to edit. Ask the user for the project130name and **confirm the connector with them**, then run:131132```bash133uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" transform init "<name>" --connector <c>134```135136The engine renders the whole skeleton (`dbt_project.yml`, `models/staging/` and137`models/marts/`, a `profiles.yml` with a single `dev` target and no secrets) and138records `connector`, `dbt_project_dir`, and `dbt_target: dev` in139`.dex/config.yml`; do not hand-write these files yourself. Init never assumes a140connector: it errors rather than defaulting, so always pass the user's confirmed141choice (a `connector:` already committed in `.dex/config.yml` also counts).142Every connector is supported: DuckDB, BigQuery, Snowflake, Databricks,143Postgres, Redshift, and ClickHouse. DuckDB needs a warehouse path (`--path`, or the144`duckdb.path` config). BigQuery needs a GCP project (usually145`bigquery.project` in `.dex/config.yml`; confirm it with the user) and writes146builds to a dedicated dev dataset (`bigquery.dev_dataset`, default147`dbt_dev`); auth is Application Default Credentials, so if credentials are148missing tell the user to run `gcloud auth application-default login`, never149ask for a key. Snowflake writes builds to a dedicated150`snowflake.dev_database`/`dev_schema` on the pinned warehouse; Databricks151writes builds to a dedicated `databricks.dev_catalog`/`dev_schema` on the152pinned SQL warehouse (if credentials are missing tell the user to run153`databricks auth login`, never ask for a token); Postgres writes builds to a154dedicated `postgres.dev_schema` (default `dbt_dev`), with the password155reaching dbt only through the `PGPASSWORD` environment variable. Redshift156writes builds to a dedicated `redshift.dev_schema` (default `dbt_dev`): with157a `redshift.workgroup` pinned the profile renders IAM auth (temporary158credentials from the AWS chain, nothing persisted), otherwise the password159reaches dbt only through the `REDSHIFT_PASSWORD` environment variable.160ClickHouse writes builds to a dedicated `clickhouse.dev_database` (default161`dbt_dev`), rendered as the profile's `schema:` because dbt-clickhouse has no162`database:` key, with the password reaching dbt only through the163`CLICKHOUSE_PASSWORD` environment variable; the rendered profile also carries164a `custom_settings` block whose `env_var` references are how `transform build`165turns the confirmed budget into a per-statement server-side cap, so do not166strip them from a profile you edit. All of them discover their connections and refuse with the fix named when none167resolves. Init refuses if any dbt project already exists.168169When the user wants staging/intermediate/marts isolated in their own170datasets/schemas (a common ask when the warehouse is shared with unrelated171work), offer `--layered-schemas`: init then also scaffolds172`models/intermediate/`, a `generate_schema_name` override, and per-folder173`+schema:` config, so builds land in `staging_dev` / `intermediate_dev` /174`marts_dev` instead of one shared dev namespace. Do not hand-write that macro;175existing projects can adopt it later via `transform macro176generate_schema_name`. Note dbt warns about "unused configuration paths" until177the first model lands in each layer folder; that resolves itself.178179Init also checks (free, metadata-only) whether each namespace the project180would build into already exists with content, and warns naming the namespace181and a few object names. The warning is advisory: relay it to the user and ask182whether the content is theirs (a previous dev build) or unrelated; when it is183unrelated, discard the freshly scaffolded project (nothing has been built),184point the config at a different dev namespace, and re-run init. A "could not185check" note just means no connection was reachable at init time.186187### dbt SQL models188189- `transform plan "<intent>" --edits-file <path|->` validates the edits and190 returns them as diffs with a plan id. Nothing is applied yet. Add191 `--scaffold <table>` (repeatable) to generate a staging skeleton192 (`stg_<table>.sql` plus per-model YAML with key tests and PII meta) from the193 `.dex/` cache instead of, or on top of, hand-authored edits.194- When you edit a model that already exists, the plan reports what your change195 does to its **row population** under `data.row_attribution`: every predicate,196 join, source and grain change is named, and each is measured on its own against197 the prior model. Read it before applying. A change you were not asked to make198 carrying a non-zero `delta` is the signal to look at: the model still compiles199 and the columns are still right, and it is now returning a different set of200 rows. It is advisory, never a refusal, because changing the filter is sometimes201 the job. On DuckDB the deltas are measured automatically; on a billed connector202 the changes are named for free and measuring them needs `--attribute-rows`203 (then the usual `--confirm --budget` once priced), so ask the user before204 spending. A change reported with `attributed: false` names why it could not be205 measured; treat that as unknown, not as zero.206- The plan also warns about the **shape** of what you authored, in `warnings`.207 A SELECT list that diverges from the columns the model's `schema.yml` declares208 is named in both directions; fix whichever side is actually stale, and say209 which one you decided it was.210- A warning that the model **exposes a raw foreign key with no resolved211 counterpart** is dex reading a convention out of the project's own models: the212 siblings it names all resolve keys of that shape, and it names the parent213 model yours could resolve against. Prefer resolving it, by joining that parent214 the way the siblings do. Where the raw key is deliberate (a fact-shaped model215 in a dimension folder, a key the consumer needs verbatim), say so plainly to216 the user rather than quietly leaving it. Never switch the check off to make217 the warning go away: `conventions.resolved_keys: false` in `.dex/config.yml`218 is a decision about the house's style, so recommend it for the user to accept,219 the same way you would a `pii_overrides` entry.220- `transform apply [plan-id]` writes the plan into the dbt project (the latest221 unapplied plan when no id is given; any plan kind, semantic included). The222 result is still a reviewable git diff for the user. If a human edited a file223 after the plan was made, nothing is written: the divergence comes back as224 diffs with `needs_confirmation`, and you should re-plan against current state225 (or, only when the user says so, re-run with `--confirm`).226- `transform plans` lists stored plans (pending and applied, newest first), so227 you never need to browse `.dex/plans/` by hand.228- `transform references <name> [more...]` answers "where is this used" before you229 change it. **Reach for this whenever a change has to land in more than one230 place**: removing a project variable, renaming a column, deleting a model,231 changing what a macro returns. Editing the files you happen to have open and232 hoping that was all of them is the failure this prevents, and it is a quiet233 one, because the project still compiles with one use left behind.234235 It is repo-only and free on every connector, so there is never a cost reason236 not to run it. The positional is variadic, so one call covers a whole rename.237 `--kind` narrows to `model`, `source`, `seed`, `snapshot`, `macro`, `var`,238 `column`, `metric`, `entity`, `dimension` or `measure`; leave it off when you239 are not sure what the project calls the thing, and the answer will tell you.240241 Read `data.completeness` before you act on the list. When it says `incomplete`,242 `data.limits` says why and `data.indeterminate` lists the call sites dex could243 not resolve, each with a file and a line. Those are references that *may* name244 what you asked about, so open them and decide yourself; do not treat the list245 of resolved hits as exhaustive when the verdict says it is not. A bare column246 name is matched across the project (`scope: name_matched`), so qualify it as247 `model.column` when you want the lineage separated from same-named columns248 elsewhere.249250 Once you know where a name is used, `transform rename` and `transform remove`251 below make the change; you do not have to carry the list into hand edits.252- `transform rename <kind> <old> <new>` generates **every** edit the rename needs253 and stores them as one plan: the definition, every model that selects the name,254 every `schema.yml` that documents or tests it, every semantic reference, and a255 seed header. Kinds are `column`, `var`, `model`, `seed`, `snapshot`, `macro`,256 `source`. Repo-only and free, like `references`.257258 **Use this instead of editing the files yourself.** Retyping a rename across259 nine files and missing the tenth is the failure mode this exists for, and it is260 a quiet one: the project still compiles.261262 Name a column as `model.column`. A bare name is refused, and the refusal lists263 the models that define a column of that name so you can pick. That asymmetry264 with `references` is deliberate: a report you read can afford to be imprecise265 and a rewrite cannot, because renaming a bare `id` project-wide would rewrite266 every unrelated `id` there is.267268 **It refuses rather than half-applying**, and each refusal names what to fix:269 a reference dex could not resolve statically, a name an installed package also270 defines, a column handed to a macro as a literal string (dex cannot tell a271 column argument from a display label), a SELECT list it cannot read. Fix what272 it names and re-run. There is no override flag, because a completeness273 guarantee you can switch off is a suggestion. A bare `select *` is *not* a274 refusal: it carries the column through under the new name with no edit, and the275 plan's `notes` says so.276277 Read `data.sites` against the `transform references` output you ran first. It278 counts occurrences per reference form in the same vocabulary, so the two279 agreeing is your evidence that nothing was dropped between reading and writing.280- `transform remove <kind> <name>` removes the **definition** and verifies every281 read is gone, refusing while any survives and naming each with a file and line.282283 It never rewrites a read, and that boundary is the point rather than a gap.284 `{% if var('using_department') %}` can be deleted or unguarded, and285 `{{ var('x') }}` sitting in an expression has no value dex may invent. You are286 the one who knows. Author those edits yourself and pass them with287 `--edits-file` in the same call: they are validated and stored in the same288 plan, so the removal is still atomic.289- `transform place <column> --targets <a,b> --expr "<sql>"` answers where a290 derived column that several models need should be *defined*. It walks `ref()`291 upward from every target, takes the lowest model they all descend from that292 already projects the inputs your expression reads, defines the column there,293 and threads it down every chain. The inputs come from parsing `--expr`, so294 there is no separate list to get out of sync with it.295296 **Read `data.reasoning` before you apply.** It names the ancestor, why it is297 the lowest, which targets descend from it, and the chain. You are supposed to298 be able to disagree with it; `--explain` gives you the same answer with no plan299 stored, which is the cheap way to ask.300301 When `data.strategy` is `per_target` the shared definition was not available302 and the reasoning says why: no common ancestor, or the lowest one is missing an303 input, or two candidates tie. dex will not go further upstream to pull an input304 down, because that turns one placement into an unbounded rewrite of everything305 above it. The fallback duplicates the derivation in each target and those306 copies will drift, so relay the reason to the user rather than applying it on307 their behalf. Often the named fix (add the missing column to the ancestor308 first) is what they actually want.309- `transform build --target dev` runs `dbt build` against a dev target. The310 engine surfaces a cost preflight first and runs only with `--confirm` (plus a311 `--budget` on billed connectors). dbt itself has no dry-run, but the engine312 compiles the project and dry-runs each node itself, so on BigQuery the first313 unconfirmed call already returns `needs_confirmation` with `estimated_bytes`314 and a `per_table_bytes` breakdown, the same shape the scanning `explore`315 commands use. Never invent a `--budget` figure: read the reported estimate316 (`per_table_bytes` is the actionable half, since it names which node is317 driving the cost) and confirm with a `--budget` grounded in that number.318 If the build is refused over the ceiling, the refusal carries a calibration319 line from `.dex/spend.jsonl`: what this connector's recent commands billed as320 a fraction of estimate, or a sentence saying there is too little history to321 say. Builds over-estimate most on a partitioned or clustered warehouse, so322 relay it, and note that the ceiling binds on the estimate rather than on what323 settles, so a budget set at that fraction of the estimate is refused again.324 A `suggested_session_ceiling` on that envelope is the project's one-time ask325 for a cumulative daily cap, separate from `--budget`: relay it and add the326 user's answer (`--session-ceiling <value>` or `--no-session-ceiling`) to the327 same re-issue, which records it in `.dex/config.yml` for good.328 Each statement dbt runs is capped server-side by the profile's329 `maximum_bytes_billed`, and the envelope reports billed bytes afterward.330 Production-looking targets are refused331 outright; `--confirm` cannot override that. dbt runs with its working332 directory pinned to the project dir, so relative paths in `profiles.yml`333 resolve against the project. When the project declares packages334 (`packages.yml`) and `dbt_packages/` is missing, the engine runs `dbt deps`335 automatically before the build.336- **`transform build --verify` is how you answer "is it right", not just "did it337 run".** A green build tells you dbt executed. It does not tell you the model338 holds the rows it should, and that is where the expensive defects live: an339 inner join written where a left join was meant loses rows, raises nothing, and340 passes every uniqueness and not-null test over the smaller result. `--verify`341 sweeps the nodes this build touched and reports the findings in the same342 envelope, under `data.verification`. Reach for it whenever the build was meant343 to prove a change is correct, which is most of the time you build at all.344345 Read `data.verification.ran` before reading anything else. It is always346 present, because a build that did not verify and a build that verified and347 found nothing look identical otherwise, and only the second one means the348 models are clean. When it ran, `findings` is ranked the way `maintain verify`349 ranks it, `scope` names the models covered, and `suppressed` names each class350 that could not run and why. Relay a suppression rather than reading past it:351 it is the difference between "checked and clean" and "not checked".352353 Findings never fail the build and never appear in `errors`. Do not treat one354 as a build failure or re-run to make it go away: relay the finding, its two355 counts, and the join it names, and let the user decide. A failed build still356 reports which node failed and which were skipped because of it, which is357 usually a faster read than the dbt log.358359 On a billed connector the sweep is priced into the build's own estimate as a360 `(row counts)` line, so the `--budget` you already read off the unconfirmed361 envelope covers both. Never add a second budget for it. If the envelope comes362 back `ok` with a `data.offer`, the build is done and billed and the offer buys363 only the counts it could not afford; relay the number rather than re-running364 the build.365- `transform deps` installs dbt packages explicitly (also the refresh path when366 `dbt_packages/` exists but is stale). No confirmation needed: deps writes only367 inside the project and never touches the warehouse.368369### Shipped macros370371- `transform macro` lists the macros dex ships; `transform macro <name>`372 proposes scaffolding one into the project's macro directory as a plan,373 applied with `transform apply` like any other. The user's copy is theirs to374 edit; re-running the command diffs it back against the shipped version (a375 warning says whether it is customized or stale), and applying that plan376 overwrites deliberately.377- `unpivot_json_object` turns a JSON object column with dynamic keys (the378 NoSQL-sourced shape: a Firestore/Mongo/DynamoDB document keyed by a related379 entity's id) into one row per top-level key. Use it instead of hand-rolling380 JSON SQL; it renders a complete SELECT:381382 ```sql383 select id, key as related_id, value as attrs384 from (385 {{ unpivot_json_object(relation=ref('stg_entities'),386 json_column='attributes', passthrough=['id']) }}387 )388 ```389390 The contract on every connector: one row per top-level key, `key` a plain391 string, `value` the warehouse's native semi-structured type (BigQuery JSON,392 Snowflake VARIANT, Databricks VARIANT, Postgres jsonb, Redshift SUPER,393 DuckDB JSON, ClickHouse raw JSON text in a String), a NULL object yields no394 rows, and a nested object's own field395 names never surface as top-level keys. For a string-typed source column396 pass the parse expression as `json_column` (`parse_json(payload)` on397 BigQuery, Snowflake, and Databricks; `json_parse(payload)` on Redshift);398 Postgres, DuckDB, and ClickHouse accept JSON-bearing text directly. Databricks needs399 VARIANT support (DBR 15.3+ or a current SQL warehouse). Two BigQuery quirks400 are absorbed by the macro, so do not "fix" them back in: a JSON path401 argument must be a compile-time literal (the macro reads values with the402 subscript operator, which accepts a computed key), and `JSON_KEYS` recurses403 into nested objects unless depth-limited (the macro pins depth 1). When a404 planned model calls the macro and the project lacks it, the plan warns and405 names the scaffold command; scaffold it rather than inlining a copy.406407### Preparing the dev target408409Before the cost gate, and for free, `transform build` refuses two things and410names the fix for each. Neither costs anything to check, so both surface on the411unconfirmed call rather than after a budget has been agreed.412413**Config that has drifted from the profile.** `transform init` renders414`.dex/config.yml` into the project's `profiles.yml`, and dbt reads only the415profile from then on. If a later config edit never reached it (a retargeted416`dev_database`, a different warehouse), the build refuses and names both values417and both files. Edit one to match the other. The engine never rewrites418`profiles.yml`, which you may legitimately have hand-edited.419420**A dev target that does not exist.** On Snowflake, dbt creates schemas but never421databases, so a missing `dev_database` is refused with the `CREATE DATABASE`422statement to run; dex will not create it for you, because its only writes are423reviewable diffs inside the repo. On Postgres, Redshift, and ClickHouse, dbt creates the dev424namespace but only if the profile's user may, so the missing privilege is what425gets refused, with the `CREATE SCHEMA`/`GRANT` statement to run. On ClickHouse426that check can also come back with no verdict, because a server may not let dex427read another user's grants; it then warns instead of guessing, and the build428proceeds with dbt's own error as the backstop. On DuckDB the dev target is a database file,429and dbt would happily create an empty one, then fail every `source()` relation430with a confusing catalog error. The convention there: copy the shared source431warehouse to the dev target path (for example432`cp shared/f1.duckdb <project>/dev.duckdb`), or point the dev target at an433existing file. Projects without sources just get a warning and an empty434database, which is fine for model-only builds.435436### The semantic layer437438- `semantic define ...` and `semantic update ...` author and evolve the dbt439 semantic models (entities, dimensions, measures, metrics) as plans. `define`440 refuses names that already exist (use `update`); `update` refuses names that441 do not (use `define`). For one logical change that mixes both (evolve existing442 metrics and add the helpers they depend on), use `semantic plan ...`: it443 accepts mixed intent and classifies each name, and the envelope reports the444 split as `defined`, `updated`, `unchanged`, and `removed`.445- **Prefer `--definitions-file` over `--edits-file` for the semantic layer.** A446 real project keeps its metrics in one shared file, so a whole-file payload447 means retyping every definition you are not touching: the diff and the448 `updated` list then describe the whole file instead of your change, and every449 restated line is a chance to corrupt a definition by hand. Send only what450 changes instead:451 `{"definitions": [{"kind": "metric", "content": "name: ...\n..."}]}`, where452 `kind` is `semantic_model` or `metric` and `content` is that definition's YAML453 body with no leading `- `. The name comes from the content, and `path` can be454 omitted for anything the project already declares (the engine rewrites it455 where it lives). Everything else in the file, comments included, is preserved456 byte for byte. Reach for `--edits-file` when you are creating a file, moving a457 definition between files, or emptying one, and when the engine refuses a layout458 it will not splice into.459- **Removing one definition is the same payload with `"op": "delete"`**:460 `{"definitions": [{"kind": "metric", "name": "doubled", "op": "delete"}]}`, the461 name declared (there is no content to read it from) and no `content` beside it.462 Nothing is removed for going unmentioned, so you can send a removal and an463 edit in one payload and everything you did not name stays as it is. Use464 `semantic update` or `semantic plan`, not `define`. The envelope reports it465 under `removed`.466- If a metric still reads what you are removing (its input is that metric, or a467 measure of the semantic model you are removing), the plan is refused and the468 reader is named: add that reader's own delete or update to the same payload,469 in any order, and it goes through. A removal that would leave a file with no470 semantic model or metric in it is refused too, because deleting or emptying a471 file is a whole-file edit: do that with `transform plan --edits-file` and472 `"op": "delete"`.473- `unchanged` means you re-stated a definition exactly as the project already474 has it. It is not an error, but if a plan is entirely `unchanged` it changes475 nothing, and the envelope warns as much: check whether you meant to edit476 something.477- Plan-time validation is layered so a plan that validates will build:478 MetricFlow's schemas check the shape; the engine resolves every metric input479 (ratio and derived metrics reference **metrics**, not measures; a measure only480 becomes a metric via `create_metric: true`, and the error names that fix); and481 finally the emitted YAML is run through **dbt's own parser** against a482 throwaway copy of the project. A plan that fails parse is refused, not stored.483 If dbt is not installed the parse degrades to a warning; `--no-parse` skips it484 explicitly.485- A semantic plan is applied like any other: `transform apply [plan-id]` writes486 its YAML into the dbt project (no id applies the latest unapplied plan).487- For native Ossie use `semantic ossie define|update|plan` with488 `--edits-file <path|->`. Supply whole documents whose paths are listed in489 `semantic.ossie.files`; the command implies the `semantic_document` kind,490 validates the complete prospective configured layer, and writes the accepted491 bytes exactly when the plan is later applied. This is a semantic-layer write492 surface and does not make Ossie the transformation project.493494 The namespace guards match the dbt ones: `define` refuses a semantic-model name495 the layer already has, `update` refuses one it does not, and `plan` accepts496 both and reports each under `defined` or `updated`. Neither removes a model, and497 a configured file may be absent before `define`, so a new document is planned498 once its path is committed to config.499500 What validates an Ossie plan is not what validates a dbt one, and the501 difference matters. There is no external parser to gate on: dex checks the502 document's structure against the Ossie schema it pins (needs `[ossie]`), its503 internal consistency in pure Python, and each SQL expression's syntax through504 the dialect engine (needs `[sql]`, which every connector extra carries).505 Without `[sql]` the third layer degrades to a named skipped-validation note,506 never to a silent pass. All three run over the complete prospective layer, your507 edits overlaid on the other configured documents, before a plan is stored.508509 It then checks the references against the exploration cache, opening no510 connection. A source relation the cached inventory positively lacks, or a511 column absent from a relation the cache profiled, refuses and stores no plan.512 Anything the cache cannot speak to is a named note instead: an unprofiled513 relation, a computed or non-SQL expression, a quoted identifier, a query-backed514 source. Read those notes rather than treating them as failures; they say what515 was not checked.516517 Accepted bytes are written exactly as authored on apply. dex does not parse and518 re-serialize the document, so comments, key order, quoting and whitespace all519 survive, and a configured document the payload did not mention is untouched. A520 target file that changed after planning refuses the whole apply rather than521 writing part of it, unless you confirm the overwrite deliberately.522523 `references/ossie-walkthrough.md` in the engine repository runs the whole524 sequence on a local warehouse if you want to see it end to end.525- dbt cannot parse semantic models in a project without a MetricFlow **time526 spine**; the engine warns when one is missing and defers the parse gate until527 one exists. Author it like any other model (a day-grain date model plus YAML528 with a `time_spine:` config) in the same or a separate plan.529- `viz preview` is not yet implemented (it returns `not_implemented`); the Viz530 integration arrives later.531532## Guardrails (enforced in the engine, not here)533534- Writes confined to the repo, and within it to two disjoint surfaces: the dbt535 project's authored path families (models, macros, snapshots, seeds, tests,536 analyses) plus the project-root manifests dbt keeps there, and the exact537 native semantic documents named in `semantic.ossie.files`. Neither surface can538 reach the other, an absolute path or a `..` escape is refused on both, and dex539 never writes to source warehouse data.540- Dev-target only. Prod-target execution is never initiated by dex.541- Cost surfaced before any spend. A build that would spend requires explicit542 confirmation and a session budget.543- Propose, don't impose. Human edits to the project (SQL and semantic YAML) and544 to a native semantic document are authoritative; on conflict the engine545 surfaces a diff and asks rather than overwriting.546- PII flags propagate from the cache into emitted dbt (model and column `meta`),547 never example values. Stamping is presence-based at any confidence; only a548 column cleared by a human `pii_overrides` entry in `.dex/config.yml` is549 scaffolded without the meta.