Industry Data Model Alignment (Active + Advisory)
Overview
This worker gives the Gold design an active voice for aligning to a Databricks Industry Vibe Data Model (or any canonical industry reference), instead of only grading alignment after the fact. It works in two touchpoints:
- Active (Phases 0 & 2): overlay the parsed source schema with industry entities, then steer domain assignment, conformed-dimension identification, grain/measure completeness, and Gold naming toward industry standards.
- Advisory (Phase 8): the crosswalk this worker writes is re-scored by
07-design-validation Validation 6 into INDUSTRY_ALIGNMENT.md.
REQUIRED BACKGROUND: Read references/industry-data-model-alignment.md for the Silver-vs-Gold nuance, the manifest schema, the crosswalk states, and the scoring rubric. This SKILL.md is the behavioral companion to that reference.
The Non-Negotiable Guardrail: Extract, Don't Generate
This worker is a lens over the source schema — never a source of new tables. It upholds databricks-expert-agent's "Extract, Don't Generate" principle: the industry model informs how you model what the source contains; it must never inject entities the source lacks.
| The worker MAY |
The worker MUST NOT |
| Map each parsed source table to an industry entity |
Invent a Gold table for an industry entity with no source data |
| Flag industry-core source tables at risk of exclusion |
Add columns not derivable from the source schema |
| Recommend industry-standard names for Gold objects (still built from source columns) |
Rename to industry terms that misrepresent the source column's meaning |
| Recommend domain grouping, conformed dims, measures to consider |
Silently auto-apply any recommendation |
Surface a coverage gap for human Waived/Planned disposition |
Fabricate the missing entity to close the gap |
A surfaced gap is resolved as Waived/Planned via SOURCE_TABLE_MAPPING.csv with a rationale — never by inventing a table. All recommendations are recorded in DESIGN_DECISIONS.md and confirmed, mirroring the orchestrator's "never silently correct a classification" rule.
When to Use / When to Skip
- Use when Phase 1 captured
industry_reference_source: vibe_generated | published and context/industry_reference.yaml exists (reduce a Vibe Silver model or a published vertical model to that manifest — see the reference doc, Mode A/B).
- Skip entirely when
industry_reference_source: none. Record "industry alignment: N/A" in the intake report and DESIGN_DECISIONS.md. Skipping is never a failure — this worker is optional and vertical-specific.
Phase 0: Coverage Overlay
Run immediately after classify_tables() in orchestrator Phase 0, before any modeling, so coverage gaps are visible up front.
Use scripts/industry_overlay.py:
build_industry_overlay(classified, reference_path) — annotates each source table with its industry entity (Covered by name/alias, Absorbed when attributes live in another table, or Gap), prints the overlay, and returns the crosswalk seed + provisional coverage %.- Fold the printed overlay into the Schema Intake Report. Every
core/extended Gap must get an explicit decision in Phase 1/2.
The manifest schema and matching rules live in references/industry-data-model-alignment.md.
Phase 2: Active Guidance Rules
Apply the industry reference as an active checklist during dimensional modeling. Each rule recommends and records in DESIGN_DECISIONS.md Section 8 — it never auto-applies.
| # |
Guidance |
Uses reference for |
Recorded as |
| 1 |
Canonical domain assignment |
Map source tables to the industry's domain names so Gold domains match industry language |
Domain column in the table inventory |
| 2 |
Conformed-dimension identification |
Industry models expose shared dims (customer/product/date); promote matched entities referenced by 2+ facts |
Bus-matrix note + 04-conformed-dimensions decision |
| 3 |
Grain & measure completeness |
Compare the entity's core_attributes/expected measures against your fact design; flag standard measures the source can support |
Grain/measure table + rationale |
| 4 |
Terminology steering |
When a source table maps to an entity but is named off-standard, recommend the industry name (only if it faithfully represents the source) |
Naming decision + old→new mapping |
| 5 |
Gap surfacing (not filling) |
For each core/extended Gap, force a Waived (out of scope) or Planned (later phase) decision |
SOURCE_TABLE_MAPPING.csv EXCLUDED/PLANNED row + rationale |
| 6 |
Sensitivity alignment |
Where the entity is PII, ensure the covering Gold table carries the PII tag |
table_properties.PII on the YAML (Phase 4) |
At the end of Phase 2, after gaps are dispositioned, write the crosswalk with write_crosswalk().
The Crosswalk Artifact (hand-off contract)
gold_layer_design/INDUSTRY_CROSSWALK.csv — the single artifact Validation 6 consumes. Matching is owned here; scoring/gating is owned by 07-design-validation.
| Column |
Meaning |
entity |
Industry reference entity name |
domain |
Industry domain |
importance |
core | extended | optional |
sensitivity |
PII | none |
state |
Covered | Absorbed | Waived | Planned | Gap |
gold_tables |
Pipe-delimited Gold table(s), or empty |
rationale |
Required for Waived/Planned/Gap |
Hand-off to Validation 6
07-design-validation Validation 6 loads INDUSTRY_CROSSWALK.csv (it does not re-match): it verifies each Covered/Absorbed row's gold_tables still exist in the current YAML, re-checks PII sensitivity against table_properties.PII, scores coverage/terminology/PII per the rubric, emits INDUSTRY_ALIGNMENT.md, and stays advisory (excluded from all_valid). The _norm/IMPORTANCE_WEIGHT constants live in scripts/industry_overlay.py.
Common Mistakes
| Mistake |
Why it's wrong |
Inventing dim_supplier to hit 100% |
Violates Extract-Don't-Generate; creates a table with no source data |
| Auto-renaming source tables to industry terms silently |
Recommendations must be recorded and confirmed, not applied invisibly |
| Making coverage a hard gate |
Reference is external, volatile, and a different layer; false failures result |
| Re-implementing matching in Validation 6 |
Two matchers drift; the crosswalk is the single source of truth |
| Forcing normalized industry structure into Gold |
Defeats the dimensional model's purpose |
Inputs
- From Phase 0: classified source table inventory (
classify_tables() output)
- From Phase 1:
industry_vertical, industry_reference_source, industry_reference_path
- From customer:
context/industry_reference.yaml (normalized manifest; template in assets/templates/)
Outputs
- Printed industry coverage overlay (Phase 0)
- Industry Alignment section appended to
DESIGN_DECISIONS.md (Phase 2)
gold_layer_design/INDUSTRY_CROSSWALK.csv (Phase 2, consumed by Validation 6)- Terminology/domain/conformed-dim recommendations feeding Phases 2–4
Design Notes to Carry Forward
Next Step
Continue Phase 2 dimensional modeling with the industry-informed decisions, then proceed through Phases 3–7. At Phase 8, 07-design-validation Validation 6 consumes the crosswalk and emits INDUSTRY_ALIGNMENT.md.
Reference Files
- Industry Data Model Alignment — manifest schema, Silver-vs-Gold nuance, crosswalk states, scoring rubric, report template
- Overlay script:
scripts/industry_overlay.py — matching, overlay, crosswalk writer
- Manifest template:
assets/templates/industry_reference.template.yaml
Related Skills
design-workers/07-design-validation — Validation 6 (consumes the crosswalk)design-workers/04-conformed-dimensions — conformed-dimension patterns (guidance rule 2)common/naming-tagging-standards — snake_case, tags (layer/domain/PII), dual-purpose descriptions
References
1---2name: 08-industry-alignment3description: Active, source-bounded alignment of a Gold dimensional design to Databricks Industry Vibe Data Models (and other canonical industry reference models — TM Forum SID, ARTS, ACORD, HL7, BIAN). Use when the customer operates in a recognizable vertical (retail, banking, healthcare, telecom, insurance, hospitality) and wants the Gold layer to cover industry-standard entities and use industry terminology, when checking industry coverage/gaps, or when the customer has a Vibe-generated Silver business model to align to. Overlays the parsed source schema with industry entities in Phase 0, steers domain assignment, conformed dimensions, grain/measure completeness, and naming in Phase 2, and emits an INDUSTRY_CROSSWALK the advisory Validation 6 re-scores. Does NOT invent tables absent from the source.4license: Apache-2.05---67# Industry Data Model Alignment (Active + Advisory)89## Overview1011This worker gives the Gold design an **active voice** for aligning to a **Databricks Industry Vibe Data Model** (or any canonical industry reference), instead of only grading alignment after the fact. It works in **two touchpoints**:12131. **Active (Phases 0 & 2):** overlay the parsed source schema with industry entities, then steer domain assignment, conformed-dimension identification, grain/measure completeness, and Gold naming toward industry standards.142. **Advisory (Phase 8):** the crosswalk this worker writes is re-scored by `07-design-validation` Validation 6 into `INDUSTRY_ALIGNMENT.md`.1516**REQUIRED BACKGROUND:** Read `references/industry-data-model-alignment.md` for the Silver-vs-Gold nuance, the manifest schema, the crosswalk states, and the scoring rubric. This SKILL.md is the behavioral companion to that reference.1718## The Non-Negotiable Guardrail: Extract, Don't Generate1920This worker is a **lens over the source schema — never a source of new tables.** It upholds `databricks-expert-agent`'s "Extract, Don't Generate" principle: the industry model informs *how* you model what the source contains; it must never inject entities the source lacks.2122| The worker MAY | The worker MUST NOT |23|----------------|---------------------|24| Map each *parsed source table* to an industry entity | Invent a Gold table for an industry entity with no source data |25| Flag industry-**core** source tables at risk of exclusion | Add columns not derivable from the source schema |26| Recommend industry-standard **names** for Gold objects (still built from source columns) | Rename to industry terms that misrepresent the source column's meaning |27| Recommend domain grouping, conformed dims, measures to *consider* | Silently auto-apply any recommendation |28| **Surface a coverage gap** for human `Waived`/`Planned` disposition | Fabricate the missing entity to close the gap |2930A surfaced gap is resolved as `Waived`/`Planned` via `SOURCE_TABLE_MAPPING.csv` with a rationale — never by inventing a table. All recommendations are recorded in `DESIGN_DECISIONS.md` and confirmed, mirroring the orchestrator's "never silently correct a classification" rule.3132## When to Use / When to Skip3334- **Use** when Phase 1 captured `industry_reference_source: vibe_generated | published` and `context/industry_reference.yaml` exists (reduce a Vibe Silver model or a published vertical model to that manifest — see the reference doc, Mode A/B).35- **Skip** entirely when `industry_reference_source: none`. Record "industry alignment: N/A" in the intake report and `DESIGN_DECISIONS.md`. Skipping is never a failure — this worker is optional and vertical-specific.3637## Phase 0: Coverage Overlay3839Run immediately after `classify_tables()` in orchestrator Phase 0, before any modeling, so coverage gaps are visible up front.4041Use `scripts/industry_overlay.py`:4243- `build_industry_overlay(classified, reference_path)` — annotates each *source* table with its industry entity (`Covered` by name/alias, `Absorbed` when attributes live in another table, or `Gap`), prints the overlay, and returns the crosswalk seed + provisional coverage %.44- Fold the printed overlay into the Schema Intake Report. Every `core`/`extended` `Gap` must get an explicit decision in Phase 1/2.4546The manifest schema and matching rules live in `references/industry-data-model-alignment.md`.4748## Phase 2: Active Guidance Rules4950Apply the industry reference as an active checklist during dimensional modeling. Each rule **recommends** and **records in `DESIGN_DECISIONS.md` Section 8** — it never auto-applies.5152| # | Guidance | Uses reference for | Recorded as |53|---|----------|--------------------|-------------|54| 1 | **Canonical domain assignment** | Map source tables to the industry's domain names so Gold domains match industry language | Domain column in the table inventory |55| 2 | **Conformed-dimension identification** | Industry models expose shared dims (customer/product/date); promote matched entities referenced by 2+ facts | Bus-matrix note + `04-conformed-dimensions` decision |56| 3 | **Grain & measure completeness** | Compare the entity's `core_attributes`/expected measures against your fact design; flag standard measures the source *can* support | Grain/measure table + rationale |57| 4 | **Terminology steering** | When a source table maps to an entity but is named off-standard, recommend the industry name (only if it faithfully represents the source) | Naming decision + old→new mapping |58| 5 | **Gap surfacing (not filling)** | For each core/extended `Gap`, force a `Waived` (out of scope) or `Planned` (later phase) decision | `SOURCE_TABLE_MAPPING.csv` EXCLUDED/PLANNED row + rationale |59| 6 | **Sensitivity alignment** | Where the entity is `PII`, ensure the covering Gold table carries the `PII` tag | `table_properties.PII` on the YAML (Phase 4) |6061At the end of Phase 2, after gaps are dispositioned, write the crosswalk with `write_crosswalk()`.6263## The Crosswalk Artifact (hand-off contract)6465`gold_layer_design/INDUSTRY_CROSSWALK.csv` — the single artifact Validation 6 consumes. Matching is owned **here**; scoring/gating is owned by `07-design-validation`.6667| Column | Meaning |68|--------|---------|69| `entity` | Industry reference entity name |70| `domain` | Industry domain |71| `importance` | `core` \| `extended` \| `optional` |72| `sensitivity` | `PII` \| `none` |73| `state` | `Covered` \| `Absorbed` \| `Waived` \| `Planned` \| `Gap` |74| `gold_tables` | Pipe-delimited Gold table(s), or empty |75| `rationale` | Required for `Waived`/`Planned`/`Gap` |7677## Hand-off to Validation 67879`07-design-validation` Validation 6 **loads** `INDUSTRY_CROSSWALK.csv` (it does not re-match): it verifies each `Covered`/`Absorbed` row's `gold_tables` still exist in the current YAML, re-checks `PII` sensitivity against `table_properties.PII`, scores coverage/terminology/PII per the rubric, emits `INDUSTRY_ALIGNMENT.md`, and stays **advisory** (excluded from `all_valid`). The `_norm`/`IMPORTANCE_WEIGHT` constants live in `scripts/industry_overlay.py`.8081## Common Mistakes8283| Mistake | Why it's wrong |84|---------|----------------|85| Inventing `dim_supplier` to hit 100% | Violates Extract-Don't-Generate; creates a table with no source data |86| Auto-renaming source tables to industry terms silently | Recommendations must be recorded and confirmed, not applied invisibly |87| Making coverage a hard gate | Reference is external, volatile, and a different layer; false failures result |88| Re-implementing matching in Validation 6 | Two matchers drift; the crosswalk is the single source of truth |89| Forcing normalized industry structure into Gold | Defeats the dimensional model's purpose |9091## Inputs9293- **From Phase 0:** classified source table inventory (`classify_tables()` output)94- **From Phase 1:** `industry_vertical`, `industry_reference_source`, `industry_reference_path`95- **From customer:** `context/industry_reference.yaml` (normalized manifest; template in `assets/templates/`)9697## Outputs9899- Printed industry coverage overlay (Phase 0)100- Industry Alignment section appended to `DESIGN_DECISIONS.md` (Phase 2)101- `gold_layer_design/INDUSTRY_CROSSWALK.csv` (Phase 2, consumed by Validation 6)102- Terminology/domain/conformed-dim recommendations feeding Phases 2–4103104## Design Notes to Carry Forward105106- [ ] Provisional coverage % from the Phase 0 overlay107- [ ] Every core/extended gap has a `Waived`/`Planned` disposition with rationale108- [ ] Conformed-dim promotions and terminology renames recorded in `DESIGN_DECISIONS.md` Section 8109- [ ] `INDUSTRY_CROSSWALK.csv` written and ready for Validation 6110111## Next Step112113Continue Phase 2 dimensional modeling with the industry-informed decisions, then proceed through Phases 3–7. At Phase 8, `07-design-validation` Validation 6 consumes the crosswalk and emits `INDUSTRY_ALIGNMENT.md`.114115## Reference Files116117- **[Industry Data Model Alignment](references/industry-data-model-alignment.md)** — manifest schema, Silver-vs-Gold nuance, crosswalk states, scoring rubric, report template118- **Overlay script:** `scripts/industry_overlay.py` — matching, overlay, crosswalk writer119- **Manifest template:** `assets/templates/industry_reference.template.yaml`120121## Related Skills122123- `design-workers/07-design-validation` — Validation 6 (consumes the crosswalk)124- `design-workers/04-conformed-dimensions` — conformed-dimension patterns (guidance rule 2)125- `common/naming-tagging-standards` — snake_case, tags (`layer`/`domain`/`PII`), dual-purpose descriptions126127## References128129- [Vibe Data Modeling (Databricks blog)](https://www.databricks.com/blog/reimagining-data-modeling-lakehouse-introducing-vibe-data-modeling)130- [lakehouse-industry-data-models](https://github.com/databricks-industry-solutions/lakehouse-industry-data-models)131- [AgentSkills.io Specification](https://agentskills.io/specification)
databricks-solutions