Databricks Architecture

Databricks Architecture Diagram

Section map (jump to the one canonical home for each topic):

• Designing from a request → Step 0 read the request's shape · Step 0.5 where extra context goes (desc/note/label/ai_reasoning) · Step 1 which diagram kind · Step 2 how much to infer · Step 2.5 how to draw the data layer.
• Workflow → copy a renderer HTML, render to PNG, iterate.
• The format → the JSON schema, worked example, Tabs, Positioning (placement resolution order), Columns/rowGrid, Relational placement, field tables (A node / An edge), Containers, physical layout, Annotations, Custom logos.
• Component catalog → the per-component wiring: map (the authoritative edge guide) + icon bank + Sources.
• Authoring rules → the terse do/don't checklist. Reference files → the .jsonc examples.

A Databricks solution architecture, drawn to fit what the request actually describes. That might be:

• a left→right flow (sources → pipeline → compute → dashboards/Genie/apps), the common shape for a single use-case; OR
• a layered platform — Databricks added ALONGSIDE an existing stack, or an ask written as explicit LAYERS (ingestion / governance / intelligence / agents / apps) → the layers organize the diagram (usually into lanes); OR
• a governance / infra containment picture (UC hierarchy, or cloud/VPC boxes).

Do NOT force every request into the sources→serve funnel. The funnel is one shape among several — match the diagram to the request's own structure (see Designing the architecture → Step 0). A prompt that lists layers, names external (non-Databricks) systems, or asks for an integration seam wants the layered shape, not the funnel.

The architecture must ultimately make sense. A user describes what they want in their own words and often leaves out the connective tissue — if they name dashboards and agents but no data layer or compute, ADD what's needed so the story holds together; a diagram missing its ingestion / compute / governance backbone isn't right. Respect what they asked for and keep their emphasis. Equally, don't over-engineer: if a detail wasn't asked for and doesn't help the story, leave it out. Aim for the smallest architecture that is complete and correct for THIS ask.

Overarching principle — fidelity scales with how specific the ask is. Two goals are in tension: do exactly what the user said and make the architecture make sense. Which wins depends on how precise and how recent the instruction is:

• A loose first description (a use-case, pasted notes, a rough vision — often hand-wavy, sometimes not 100% internally consistent) → your job is interpretive: honor the intent and emphasis as much as possible, but fill the gaps and quietly reconcile contradictions so the result holds together. Here make-it-make-sense leads. Don't refuse or flag every inconsistency — resolve it into a coherent diagram.
• A specific instruction ("connect X to Y", "put the model upstream", "drop the dashboard", "these three are one block") → respect it literally, even if you'd have drawn it differently. Precise beats your sense of a tidier architecture.
• A follow-up correction is the STRONGEST signal there is. When the user edits or corrects the diagram, do exactly that and do NOT silently revert it toward what seems cleaner to you — if their change looks odd, keep it and record the why in ai_reasoning, don't overrule it. Later + more-specific always wins over earlier + vaguer (and over your own defaults).

The rule of thumb: the vaguer the ask, the more you shape it; the more specific or corrective the ask, the more you obey it verbatim.

The file is a flat list of nodes + edges — a node is on the canvas iff it's in nodes (no visibility/state/diffing). Author STRUCTURE, not pixels: place with col/row lanes, relational fields, wraps boxes, and pin banners — you almost never write at. These compose in a fixed resolution order — full rules in Positioning below.

Edges follow the catalog wiring: lines. Each component's wiring: line (in the Component catalog) is the authoritative map of what it consumes/depends on: a plain entry is the normal edge (draw it when both components are present), one marked (optional) is added when the story calls for it. They're the typical connections, not an exhaustive whitelist — a genuine edge the story needs is fine; just don't invent a connection a component wouldn't actually have. Read the relevant rows before wiring.

Avoid floating components. A participating component — one that's IN the data flow — should connect to something; by default it doesn't sit on the canvas unwired. sql-lakehouse reads from the medallion; ai-bi-dashboard/genie read from the lakehouse; an app reads from lakebase/features and fronts a model; lakebase syncs from the pipeline and powers the app. If you place one of these and can't trace an edge into or out of it (following its wiring: line), you either forgot the edge or the tile doesn't belong — fix one or the other. The exception is GLOBAL / cross-cutting things that are ABOUT the whole architecture, not a step in it: a type:"note"/text annotation, the db-platform banner, a wrapping box, and platform-spanning governance (unity-catalog/governance-block) legitimately have no data-flow edge — they frame the diagram rather than participate in it. But a consumption/app/state tile (ai-bi-dashboard, databricks-apps, lakebase, genie, sql-lakehouse, a model) is NOT global — it should be wired. This is the failure to watch for in wide multi-lane layouts: those tiles land far from the data they read and get left stranded. Wiring is a design step, not an afterthought: after placing tiles, trace the flow end-to-end — walk each source through the pipeline to a consumer/entry point and draw every hop. When a global/governing tile DOES draw an edge, it attaches to what it actually governs — ai-gateway to MODEL calls (model-serving, apps), unity-catalog to data — never chained through whatever tile happens to sit next to it.

Designing the architecture from a request (architecture-first)

Design a coherent, functionally-complete architecture that solves the ask from the user's request (their prompt / pasted text) — not from any pre-existing file or default, not a word-for-word transcription, not a fixed template.

Step 0 — READ THE REQUEST'S OWN STRUCTURE FIRST (do this before anything)

The request usually TELLS you the shape. Extract it before you pick components:

• Does it list LAYERS / SECTIONS? ("Layer 1 — ingestion, Layer 2 — governance, Layer 3 — intelligence, …", or numbered/bulleted sections.) → Use the layered shape (not the funnel), honoring the request's order and names. Layers often map to left→right lanes, but a layer→column mapping is a starting point, not a rule: a governance/cross-cutting layer usually SPANS the others (a pinned bar/band, not a lane), thin adjacent layers can share a lane, and a layer can run top→bottom. Choose the fewest lanes that keep the flow legible and the connected tiles near each other — many sparse lanes push consumers away from the data they read and leave them stranded. Do NOT collapse a rich multi-layer ask into the 5-tile funnel — that's the #1 failure mode. When the ask doesn't dictate its own order, a sane default spine is sources → data processing → domain + governance → compute → ML / agentic → apps → Genie One — a starting skeleton only; the ask's own structure and names always win. Keep each layer coherent — group same-role tiles into the same lane rather than scattering them (e.g. compute sits as one layer between data and the consumers). This is a preference, not a rule: a layered ask mentioning related tiles in different sentences isn't a reason to split them across lanes unless the ask actually wants them separated.
• Does it name EXTERNAL (non-Databricks) systems? (AWS Lambda/Kinesis/S3, an existing app, a third-party bus, "keep X, add Databricks alongside".) → Draw the external side in its OWN boundary box and Databricks in ANOTHER; the gap between them is the integration seam. Show it both ways if the ask mentions callbacks/bi-directional (subscribe IN + call APIs back OUT). Use file:cloud/<provider>/… logos for the external side.
• Does it map future-state to current gaps, name reference customers, or carry other context? → Those are type:"note" post-its by default (a text/box annotation only when you specifically want a plain caption or a titled badge), not core components — place them near what they describe (beside the relevant tile, or below/beside the diagram), just don't let them overlap or crowd the tiles.

Honor the structure the user handed you — if they wrote the layers/sections, use those names and order.

Step 0.5 — where does the EXTRA CONTEXT go? (keep the diagram dense)

A rich request carries far more detail than there are components. Don't drop it, and don't inflate the diagram with it — route each piece to the right lightweight slot, keeping every line short so the picture stays scannable:

• Context ABOUT a component (what a source/domain/table holds, what a tile is for) → a desc line under it (title + one short subtitle, like "Lakebase" / "Managed Postgres for app state"). Works on catalog tiles (override only when the default can't say it), and on source and logo nodes (set desc + it shows; a logo desc is a muted 2nd caption line). One tight phrase, not a sentence — the tile stays compact and the layout reserves room for it. If it needs a paragraph, it's not a desc.
• Context from the REQUEST that isn't about any one component (a rationale, a "keep X / we'll add Y", future-state wins, reference customers, an SLA, a caveat, a team/ownership fact) → a VISIBLE type:"note" post-it, placed beside/below the component it relates to. This is the DEFAULT for request context — if the user wrote it and it matters, it belongs on the canvas as a note, not compressed into a tile desc and NEVER parked in the hidden ai_reasoning field. Lean toward MORE notes: add one per distinct point rather than cramming several into one or dropping them. (Use a plain text/box annotation only for a genuine caption/badge; never invent a fake component.)
• Context about a CONNECTION (why A feeds B, what flows, an order/step, a non-obvious hop) → a short edge label drawn ON the edge. Use it whenever the relationship isn't self-evident from position — but keep it to a few words ("Subscribe (streams)", "Query shared context", "Trigger APIs after decision") so edges stay legible and the diagram dense.
• Your OWN reasoning you want to keep but NOT show (why you chose a handle, a "don't add X" caution, why a row/col) → the node/edge ai_reasoning field — it round-trips verbatim and never renders (see rule 10). This is ONLY for your authoring rationale, never for request content the user should see — that goes to a visible post-it above.

The bar: every component and edge that isn't obvious gets a short label or desc; everything narrative goes to a note/annotation; nothing becomes a bogus tile, and no line runs long. Dense and self-explanatory beats sparse-but-cryptic or cluttered-with-paragraphs.

Step 1 — which KIND of diagram is this?

Pick the shape that fits what Step 0 found. Most SINGLE-use-case asks are (1); a multi-layer / "alongside an existing stack" ask is (2).

1. Flow (solution / demo) — a left→right DATA-FLOW story: a few sources (default ~4, see Sources) → Lakeflow+Genie → lakehouse/Lakebase → dashboard/Genie/app → Genie One → user. The common shape for ONE use-case ("predictive maintenance", "customer 360") or a data+AI feature ask ("Lakeflow Connect + SDP → a model endpoint"). Uses the catalog tiles + composites; how much to infer → Step 2; which reference(s) to learn from → Pick a starting point. Layout conventions: one box wraps the whole flow (usually not the raw sources) = "the Databricks Platform" (auto-renders behind its children — see Containers); db-platform + governance-block pin to its top-left/top-right (never a raw at); Genie One fronts the consumption tiles with auto-arrows.
2. Layered platform / integration — the request is organized as layers, and/or Databricks sits alongside an existing (non-Databricks) system. The layers organize the diagram in the ask's order (usually lanes, but see Step 0 — governance spans, thin layers merge, pick the fewest legible lanes); external systems get their own boundary box with an integration seam to the Databricks boundary box; governance (Unity Catalog) SPANS the layers as a pinned top bar or full-height band, not one lane tile. Build it from the ask's actual layers (boundary boxes + seam + layers-as-columns). This is the shape for enterprise / "add Databricks to our stack" asks — do NOT reduce it to the funnel. Logical DOMAINS / knowledge areas (a "five-domain context", a shared semantic layer, a set of business subject areas any agent can query) → draw each as a type:"logo" with icon:"file:vendor/genie-ontology" (the Genie Ontology mark = logical domain / knowledge), wrapped in a box titled for the layer.
3. Physical / governance — the Unity Catalog HIERARCHY: workspace → metastore → catalogs → schemas → tables. A containment picture (nested boxes), NOT a flow. Trigger: the ask is about UC objects / org structure. Use the container-box presets — see Databricks physical layout below.
4. Infra / networking — cloud/account topology: VPC / subnets / PrivateLink. Also containment (nested cloud boxes, the Containers pattern). Trigger: the ask is about networking / deployment / cloud accounts.

If the ask blends kinds (e.g. "the layered flow, inside our VPC"), compose them.

Step 2 — (solution/demo only) how much to infer

• Broad / use-case ask ("predictive maintenance", "fraud detection", "a governed data platform"): infer the full end-to-end shape the use-case implies. Predictive maintenance is a MODEL story → sensors/history → pipeline → ML training → registry → serving → scoring + dashboards. Don't under-scope to the couple of nouns typed — add the components that make it actually work.
• Named-component ask ("Lakeflow Connect + SDP into UC, served by a model endpoint"): assemble exactly those into a working whole — honor what they named + add the connective tissue that wires it (platform box, ingest ports, an entry point, the edges). Don't over-build past what they asked, don't under-build to isolated unconnected tiles.

Either way, components that belong together, wired so the flow reads correctly, on the governed platform.

Don't copy a reference verbatim — adapt it to THIS ask. References are for inspiration. Read the reference(s) to learn the PATTERN (which components connect, how the layout/handles/pins work), then build a fresh diagram for THIS ask. Read as many references as are relevant — most real asks mix several: take the ML platform's serving lane, add the agent-bricks supervisor, drop the governance bar on top, wrap it in the layered shape's boundary boxes, swap the sources. Pick whichever references inform the ask, learn from each, and compose your own. A reference matches the ask 1:1 only rarely — even then, adapt it (names, sources, which components) rather than reproduce it. See Pick a starting point for what each offers.

Step 2.5 — pick how to draw the DATA / INGEST layer (3 ways)

The bronze→silver→gold data layer can be drawn three ways. Pick by what the story emphasizes — don't default to one:

1. lakeflow-genie-block (or lakeflow-block) — the big unified block. Shows the whole ingest story in ONE block: Connect · Zerobus · raw-file landing on the left rail, the SDP medallion (bronze→silver→gold), Delta/Iceberg, and (genie variant) a "Built with Genie Code" footer. Use when ingestion IS part of the story — you want to showcase how data lands + gets processed, all products in one tidy block. Optional bronze_desc/silver_desc/gold_desc params add a short caption under each layer.

   { "id": "data", "type": "lakeflow-genie-block", "col": "pipeline",
     "params": { "gold_desc": "Business marts + metrics" } }
   

2. medallion-table — the simpler block. Just bronze→silver→gold in one compact tile, no ingest rail. Use when ingestion is NOT the focus but you still need the medallion — especially when you want the Feature Store / Metric Views forks off gold (params:{feature_store,metric_views}, wired via @out-fs/@out-mv/@out-gold). Same *_desc layer-caption params.

   { "id": "med", "type": "medallion-table", "col": "pipeline",
     "params": { "feature_store": true, "metric_views": true } }
   

3. DIY — compose it yourself. When you want to go into DETAIL and list the actual tables: a type:"box" per layer (title "Bronze"/"Silver"/"Gold") wrapping type:"logo" tiles (icon bronzeLayer/silverLayer/goldLayer, text = the table name). Tiles use col/row inside the box's column; each box needs no col/pad/size/z — it auto-sizes around its tiles (z is automatic — see Containers). Flow edges box→box. Optionally wrap all three in a parent box titled "Lakeflow Spark Declarative Pipelines" with the SDP logo (titleIcon:"sdpBrand") — a box titleIcon accepts ANY icon-library key.

   { "id": "b1", "type": "logo", "icon": "bronzeLayer", "text": "orders_raw",   "caption": "right", "col": "bronze", "row": 1 },
   { "id": "b2", "type": "logo", "icon": "bronzeLayer", "text": "events_raw",   "caption": "right", "col": "bronze", "row": 2 },
   { "id": "s1", "type": "logo", "icon": "silverLayer", "text": "orders",       "caption": "right", "col": "silver", "row": 1 },
   { "id": "s2", "type": "logo", "icon": "silverLayer", "text": "events",       "caption": "right", "col": "silver", "row": 2 },
   { "id": "g1", "type": "logo", "icon": "goldLayer",   "text": "customer_360", "caption": "right", "col": "gold",   "row": 1,
     "ai_reasoning": "only add ai_reasoning when there's a real authoring choice to record — don't invent" },
   { "id": "bronze-box", "type": "box", "title": "Bronze", "wraps": ["b1", "b2"] },
   { "id": "silver-box", "type": "box", "title": "Silver", "wraps": ["s1", "s2"] },
   { "id": "gold-box",   "type": "box", "title": "Gold",   "wraps": ["g1"] },
   { "id": "sdp-box", "type": "box", "title": "Lakeflow Spark Declarative Pipelines", "titleIcon": "sdpBrand",
     "wraps": ["bronze-box", "silver-box", "gold-box"], "pad": 28,
     "ai_reasoning": "hidden rationale goes INLINE as an ai_reasoning field — the JSON has no // comments" },
   { "id": "note-residency", "type": "note", "text": "PII stays in the EU region; owned by the Data Platform team.", "below": "sdp-box",  "gap": 32 },
   { "id": "note-sla",       "type": "note", "text": "Data refreshed every 15 min · end-to-end latency ~2 min source→Gold.", "below": "gold-box", "gap": 32 },
   { "id": "note-ownership", "type": "note", "text": "Gold layer owned by the data-insights team.", "rightOf": "note-sla", "gap": 24 }
   

   The block is plain JSON — no // comments. Anything you'd write as a comment about a node is either a visible caption (desc/label), a visible type:"note" post-it, or the invisible ai_reasoning field ON that node — never a // line. Edges for this block: bronze-box@r → silver-box@l and silver-box@r → gold-box@l (flow: true). Feed any of the three from the left with a few source tiles (vary them per demo) — into the ingest ports for option 1 (@in-lakeflow-connect/@in-zerobus/@in-direct), or into @l for options 2/3.

   Add AS MANY type:"note" post-its as the request has context for — one per distinct piece of non-component context (a constraint, an SLA, residency, ownership, a "keep X / add Y" principle, a caveat). The example shows three; a rich prompt may warrant more. Don't cram several facts into one post-it, and don't drop context because there's no tile for it — that's exactly what post-its are for. (Visible post-it vs invisible ai_reasoning field: the distinction lives in Step 0.5 — request narrative the user should see → type:"note"; your own hidden rationale → ai_reasoning.)

Workflow — how to make a diagram

The diagram is one self-contained HTML file with its JSON in an inline block. Steps:

1. cp renderer/architecture-viewer.html my-arch.html (or architecture-editor.html for a browser-editable copy with Load/Download buttons).
2. Replace the JSON inside <script type="application/json" id="architecture">…</script> with your array (schema below; plain JSON, no //). Learn the schema from The format below + the relevant reference/*.jsonc — then author your OWN diagram for the ask (don't paste a reference in whole).
3. Open in any browser (no server), or render + read the PNG to iterate (below).

Render loop (do this every edit):

npx playwright install chromium-headless-shell   # one-time (~90MB)
node renderer/render-arch.mjs my-arch.html        # → my-arch.png

Read my-arch.png — check components present, wired right, laid out clean; fix the JSON; repeat. Uses chromium-headless-shell over CDP (just the shell + node 18+, auto-found in Playwright's cache). If it reports "No Chrome/Chromium found", run the install line or set CHROME_PATH=/path/to/chrome.

Pick a starting point and read trusted example

Which reference(s) each request style maps to (most asks draw from more than one; see Designing the architecture above):

You can open multiple to compose them for trusted layout.

The format

The example below is the use-case starting point: when the user types a plain USE-CASE with no named components or layers ("a customer-360 demo", "churn analytics for retail"), start from this funnel and ADD the components that use-case requires — a prediction story adds ML training → registry → serving; an assistant story adds a supervisor + Knowledge Assistant; a real-time story adds streaming ingest. A pattern to build ON, not a floor to reduce everything to. (When the ask NAMES layers or an external stack, it is NOT this shape — use the layered reference per Step 0/1.) It also teaches the JSON schema + edges/handles/pins. Emit an array of tabs ([ … ], see Tabs). Plain JSON — no // comments. Any explanation lives INLINE as an ai_reasoning field on the node/edge it's about (never rendered, round-trips on save) — the example uses it to teach the schema, exactly as you'd document your own non-obvious choices:

{
  "name": "Customer 360",
  "story": "Ingest our Postgres, ERP, sensor, and PDF data into a governed lakehouse, then give the business a dashboard and a Genie Agent to ask questions in plain language — reached through Genie One, all on Databricks.",
  "columns": ["sources", "pipeline", "compute", "work", "entry"],
  "nodes": [
    { "id": "src-postgres", "type": "source", "col": "sources", "row": 1, "label": "Postgres", "icon": "file:vendor/postgresql",
      "ai_reasoning": "sources stack by row; each source's edge (below) names the Lakeflow ingest PORT it lands on — that target handle drives BOTH the port anchor and the flow animation" },
    { "id": "src-erp", "type": "source", "col": "sources", "row": 2, "label": "Acme ERP", "icon": "text",
      "ai_reasoning": "no vendor logo for this internal ERP → icon:\"text\" draws the label as a brand-colored text badge (niche/internal systems)" },
    { "id": "src-sensors", "type": "source", "col": "sources", "row": 3, "label": "Sensor data", "icon": "sensorSource",
      "ai_reasoning": "realtime stream → its edge targets @in-zerobus (particle-river animation)" },
    { "id": "src-docs", "type": "source", "col": "sources", "row": 4, "label": "PDF documents", "icon": "pdfLogo",
      "ai_reasoning": "files → its edge targets @in-direct (travelling-docs animation)" },
    { "id": "lakeflow-genie-block", "type": "lakeflow-genie-block", "col": "pipeline",
      "ai_reasoning": "the one data-layer block: ingest + bronze→silver→gold, built by Genie Code" },
    { "id": "sql-lakehouse", "type": "sql-lakehouse", "col": "compute",
      "ai_reasoning": "governed serving copy; the consumption lane (dashboard + Genie) reads from here" },
    { "id": "ai-bi-dashboard", "type": "ai-bi-dashboard", "col": "work", "row": 1 },
    { "id": "genie", "type": "genie", "col": "work", "row": 2 },
    { "id": "genie-one", "type": "genie-one", "col": "entry", "rot": 90,
      "ai_reasoning": "business-user entry point / interface onto everything to its left; persona pill built IN (no separate user node); rotated 90° into a slim lane; its edges auto-arrow (no flow/arrow needed)" },
    { "id": "db-platform", "type": "db-platform", "pin": { "at": "top-left", "to": "platform-box" },
      "ai_reasoning": "top-band banner PINNED to the box corner (never absolute at — those drift off-corner when the node set changes); a non-float pin RESERVES a top band so the box grows to enclose it" },
    { "id": "governance-block", "type": "governance-block", "pin": { "at": "top-right", "to": "platform-box" } },
    { "id": "platform-box", "type": "box",
      "ai_reasoning": "one white box wrapping the whole flow = 'all of this is the platform'; no z needed — a wrapping box auto-renders behind its children (see Containers)",
      "wraps": ["src-postgres", "src-erp", "src-sensors", "src-docs", "lakeflow-genie-block", "sql-lakehouse", "ai-bi-dashboard", "genie", "genie-one"] }
  ],
  "edges": [
    { "id": "e1", "from": "src-postgres", "to": "lakeflow-genie-block@in-lakeflow-connect", "flow": true,
      "ai_reasoning": "databases/SaaS land on @in-lakeflow-connect; naming the port on the target handle drives the port anchor AND the flow animation" },
    { "id": "e1b", "from": "src-erp", "to": "lakeflow-genie-block@in-lakeflow-connect", "flow": true },
    { "id": "e1c", "from": "src-sensors", "to": "lakeflow-genie-block@in-zerobus", "flow": true },
    { "id": "e1d", "from": "src-docs", "to": "lakeflow-genie-block@in-direct", "flow": true },
    { "id": "e2", "from": "lakeflow-genie-block", "to": "sql-lakehouse", "flow": true },
    { "id": "e3", "from": "sql-lakehouse", "to": "ai-bi-dashboard", "flow": true },
    { "id": "e4", "from": "sql-lakehouse", "to": "genie", "flow": true },
    { "id": "e6", "from": "genie-one", "to": "ai-bi-dashboard",
      "ai_reasoning": "Genie One fronts the consumption tiles; its edges auto-arrow away from it toward the resource" },
    { "id": "e7", "from": "genie-one", "to": "genie" }
  ]
}

Tabs — the file is an ARRAY of architectures

Top level is a JSON array, one element (the shape above) per tab; its name is the tab label. Multiple tabs = multiple views of one diagram (e.g. "Ingestion" / "Serving"). Always emit the full array [ { … } ], even for one tab (a bare object is accepted but re-serialized as an array). Emit every tab each time.

Positioning: how the strategies compose

You have several placement strategies and they are designed to layer, not compete. They resolve in this fixed order — each stage respects everything placed before it, so you can freely combine them:

1. at (explicit [x,y] pixels) — used verbatim, always wins. Escape hatch; you rarely write it.
2. Columns — col (lane) + row (order in lane), or a shared rowGrid matrix. The backbone of most diagrams.
3. Auto-seed — a bare node (no col/at/relational) that is only placed by being in some box's wraps gets stacked inside that box automatically (see Containers).
4. Relational — below/above/leftOf/rightOf/alignX/alignY position a node against another node's resolved box (see below). Runs after columns, so the anchor already has its place.
5. Boxes — wraps/bounds size around whatever their children resolved to (columns + relational + seeds), innermost first; a box may itself be placed with below/above/alignY.
6. Pins — pin docks a banner/persona into a box corner, last of all.

Precedence per node: at > pin > relational > col/row. Set at most ONE relational field per node.

Composition cheatsheet (all verified to work together):

• A rowGrid matrix (col+row) with a rightOf satellite off one tile inside a wraps platform box with a pinned governance banner — all at once. The box grows to include the satellite; the pin reserves its band on top.
• A wraps box whose children are a mix of col-placed tiles and bare auto-seeded ones — the placed tiles keep their lane, the bare ones stack below them inside the box.
• Nested boxes (wraps of wraps), one placed below another via a box-level below — the whole subtree shifts together.

Columns, and the optional row-grid

Default placement: col = X (which lane), row = order WITHIN that lane (top→bottom); each lane centers its own stack independently. Lanes don't line up row-for-row.

Set top-level rowGrid: true to turn row into a shared horizontal band across ALL lanes — same row number = same Y line in every column. col still sets X; only Y changes. Use it when you want a clean matrix (rows reading straight across):

rowGrid: true                columns →   pipeline    ml            serving
                            row 0                     UC Registry   Model Serving
                            row 1   ← (skipped: empty band = vertical space)
                            row 2    Batch Job        Model Training
                            row 3    Medallion        …             Lakebase

Rules under rowGrid:

• Rows sit on a FIXED PITCH — row N lands on the same Y line in every column, so tiles across columns register into clean horizontal rows. The whole grid is centered on y=0.
• row numbers are grid coordinates, not just order. SKIP a number to insert an empty row of vertical space (rows 0,2,4 are more spread out than 0,1,2) — the lever for opening room when edge labels between two rows collide.
• No row → that node falls back to stacking within its own col (e.g. leave the data sources row-less to just stack them).
• at / alignY / below / above still override a node's grid position (per node).
• Line the source up with what it feeds — give a source the SAME row as its target so the feed edge is a clean horizontal line (e.g. src-pdf and knowledge-assistant both on row 4; src-postgres and medallion-table both on row 3).
• A TALL node (medallion with forks, agent-bricks, lakeflow blocks) OVERFLOWS DOWNWARD in its OWN column — it spans into the next cell's space without pushing other columns or inflating the shared row. So a short tile at the same row in another column stays put (no gap from the tall neighbor). Just don't place another node in this column on the row(s) the tall node overflows into, or they'll overlap — give the tall node room below it in its lane.

Relational placement (place a node against another)

When a node's spot is best described relative to another node rather than by a lane, use ONE of these instead of guessing at. Each references another node's id; it resolves AFTER columns (so the anchor keeps its own place), and the engine auto-de-overlaps the result.

• One per node. If several are set only one applies (precedence alignX > alignY > leftOf > rightOf > above > below). at still wins over all.
• Chains resolve in order — A rightOf B, B rightOf C settles C→B→A.
• Siblings fan out — several nodes rightOf the SAME anchor spread along the perpendicular axis instead of stacking on one point.
• A satellite inside a wraps box is still enclosed — the box grows to include it. So rightOf a tile that's inside the platform box keeps the satellite in the box.
• Boxes may use below/above/alignY too (not leftOf/rightOf) — see Containers.

Top level (each element of the tabs array)

A node

…(truncated)