# Project Planning

> Create multi-phase project plans for Databricks data platform solutions with Agent Domain Framework and Agent Layer Architecture. Includes interactive Quick Start with key decisions, industry-specific domain patterns, complete phase document templates (Use Cases, Agents, Frontend), Genie Space integration patterns, deployment order requirements, and worked examples. Supports both acceleration mode (plan on a completed Gold layer) and workshop mode (`planning_mode: workshop`) that plans from the best available layer with hard artifact caps. Use when planning any Databricks solution post-Gold layer — observability, analytics, agent-based frameworks, or multi-artifact projects.

- Skill: `databricks-solutions/project-planning` (Agent Skill, multi-file: 25 files)
- Install (CLI): `npx skillmds@latest add databricks-solutions/project-planning`
- Raw SKILL.md: https://api.skillmd.com/api/skills/databricks-solutions/project-planning/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: databricks-solutions (https://skillmd.com/u/databricks-solutions)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/databricks-solutions/project-planning

---


# Project Plan Methodology for Databricks Solutions

## Planning Mode

**Default: Data Product Acceleration** — full breadth, all domains, all artifacts, **Gold layer required as planning basis**. This is the standard behavior described in this entire skill document below.

**Workshop mode** is available for Learning & Enablement scenarios with hard artifact caps **and layer flexibility** — it can plan from the best available source layer (Gold, Gold design YAML, Silver, Bronze, or source CSV). Workshop mode is NEVER activated unless the user includes the **exact phrase** `planning_mode: workshop` in their prompt.

> **Mode vs source layer:** `planning_mode` (acceleration | workshop) controls artifact caps and validation strictness. `planning_source.selected_layer` (gold | gold_design | silver | bronze | source_csv) records which input the plan was derived from and is set automatically by Phase 0 below. Acceleration mode FORCES `selected_layer = gold` (or `gold_design` only if explicitly allowed). Workshop mode picks the best available source via the Phase 0 priority order and stamps it onto every manifest.

### Mode Detection Rules

1. **Default is ALWAYS `acceleration`.** If the user does not explicitly declare workshop mode, use acceleration.
2. **Workshop mode requires EXPLICIT opt-in.** The user must include one of these EXACT phrases:
   - `planning_mode: workshop`
   - `"workshop mode"`
   - `"use workshop mode"`
3. **Do NOT infer workshop mode** from words like "small", "simple", "demo", "limited", "quick", "basic", "training", or "few". These are NOT triggers. A user may want a narrow-scope acceleration plan — that's still acceleration mode with fewer use cases.
4. **When in doubt, ask.** If the user's intent is ambiguous (e.g., "Create a plan for a workshop"), ask: *"Would you like full Data Product Acceleration mode (default) or Workshop mode with limited artifacts? To use workshop mode, include `planning_mode: workshop` in your request."*
5. **Confirm mode at the start.** The first line of any plan output should state the active mode:
   - `**Planning Mode:** Data Product Acceleration (default)`
   - `**Planning Mode:** Workshop (explicit opt-in — artifact caps active)`
6. **When workshop mode is activated,** read `references/workshop-mode-profile.md` for artifact caps, phase scope, and selection criteria. Do NOT read that reference otherwise.
7. **Propagate mode to manifests.** Add `planning_mode: workshop` or `planning_mode: acceleration` to all generated manifest YAML files. Downstream orchestrators seeing `workshop` MUST NOT expand beyond the listed artifacts via self-discovery.

## Overview

Comprehensive methodology for creating multi-phase project plans for Databricks data platform solutions. This skill combines interactive project planning with architectural methodology, including templates, worked examples, and quality standards.

**Key Assumption (mode-aware):**

- **Acceleration mode (default):** Planning starts AFTER Bronze ingestion AND Gold layer design are complete. Gold is the required planning basis. These are prerequisites, not phases. Phase 0 will stop with a remediation message if Gold is missing.
- **Workshop mode (`planning_mode: workshop`):** Planning AND deployment are layer-agnostic. Phase 0 selects the highest-fidelity input present from: deployed Gold, Gold design YAML, deployed Silver, deployed Bronze, or a source schema CSV. The selected layer is stamped onto every manifest as `planning_source.selected_layer`. Workshop manifests built from Silver or Bronze are marked `implementation_readiness: workshop_deployable` — downstream stages (semantic-layer, observability, ml, genai-agents) deploy directly against the selected layer. Workshop manifests built from a source CSV are marked `implementation_readiness: workshop_draft` (planning contract only — no live tables to deploy against). `requires_gold_promotion` is an **advisory** field; it is recommended for production but never blocks deployment.

## When to Use This Skill

Use this skill when:
- Creating architectural plans for Databricks data platform projects
- Building observability, analytics, or monitoring solutions
- Planning multi-artifact solutions (TVFs, Metric Views, Dashboards, Genie Spaces, Alerts, ML Models)
- Developing agent-based frameworks for platform management
- Creating frontend applications for data platform interaction
- Starting a new project after Gold layer is complete

## Idempotency Guard (Run FIRST)

**Before regenerating plans, detect existing artifacts to avoid clobbering work-in-progress.** A common failure mode is regenerating `plans/` wholesale on a re-run and overwriting user edits to manifests, addendums, or the Use Case Catalog.

```python
from pathlib import Path

PLANS_DIR = Path("plans")
if PLANS_DIR.exists() and any(PLANS_DIR.iterdir()):
    existing = sorted(p.relative_to(".") for p in PLANS_DIR.rglob("*") if p.is_file())
    print("Existing plan artifacts detected:")
    for p in existing:
        print(f"  {p}  (mtime={Path(p).stat().st_mtime})")
    print(
        "\nHow would you like to proceed?\n"
        "  - regenerate  (DELETE and rebuild all plan files — destructive)\n"
        "  - incremental (keep existing files, only emit MISSING artifacts)\n"
        "  - skip        (exit this orchestrator — recommended default)\n"
    )
```

**Rules:**
- **Default is `skip`.** If the user is silent or ambiguous, assume `skip` and exit with a summary of existing files.
- `regenerate` must be explicit. Confirm the action ("I will delete N files under `plans/` — proceed?") before doing anything destructive.
- `incremental` is the right choice when downstream orchestrators (semantic-layer, observability, ml, genai-agents) reported a missing manifest — only emit the missing manifest, not the whole tree.

**Escape flag:** Users can set `planning_allow_overwrite: true` in their prompt to skip the idempotency check (equivalent to choosing `regenerate` without interactive confirmation).

---

## Quick Start (5 Minutes)

### Fast Track: Create Your Project Plan

```bash
# 1. Verify prerequisites for your mode:
#    Acceleration (default):
#      - Bronze ingestion ✅
#      - Silver DLT streaming ✅
#      - Gold dimensional model ✅ (REQUIRED)
#    Workshop (planning_mode: workshop):
#      - At least ONE of: deployed Gold, gold_layer_design/yaml/, deployed Silver,
#        deployed Bronze, or data_product_accelerator/context/*.csv
#      - Phase 0 picks the highest-fidelity input automatically.

# 2. Run this prompt with your project info:
"Create a phased project plan for {project_name} with:
- Planning assets: {n} tables (Gold/Silver/Bronze depending on what is available)
- Use cases: {use_case_1, use_case_2, use_case_3, etc.}
- Target audience: {executives, analysts, data scientists}
- Agent domains: {domain1, domain2, domain3, domain4, domain5}"

# 3. Output: Complete plan structure in plans/ folder
#    - Acceleration emits gold-dependency-manifest.yaml.
#    - Workshop emits gold-dependency-manifest.yaml OR source-dependency-manifest.yaml
#      depending on the selected planning source layer.
```

### Key Decisions (Answer These First)

| Decision | Options | Your Choice |
|----------|---------|-------------|
| Agent Domains | Derive from business questions (typically 2-5) | __________ |
| Phase 1 Addendums | TVFs, Metric Views, Dashboards, Monitoring, Genie, Alerts, ML | __________ |
| Phase 2 Scope | AI Agents (optional) or skip | __________ |
| Phase 3 Scope | Frontend App (optional) or skip | __________ |
| Genie Space Count | Based on asset count vs 25-asset limit (see Rationalization) | __________ |
| Agent Architecture | Agents use Genie Spaces (recommended) or Direct SQL | __________ |
| Agent-Genie Mapping | 1:1, consolidated, or unified (based on asset volume) | __________ |

## Working Memory Management

This orchestrator spans 3 phases. To maintain coherence without context pollution:

**After each phase, persist a brief summary note** capturing:
- **Phase 1:** Domain list with Gold table mappings, addendum selections, business questions per domain, artifact count estimates
- **Phase 2:** Plan document file paths, cross-references verified, total artifact counts by type
- **Phase 3:** Manifest file paths (semantic-layer, observability, ml, genai-agents), validation results, summary counts

**What to keep in working memory:** Current phase's template, domain list + artifact inventory, and previous phase's summary. Discard intermediate outputs — they are on disk. Read templates from `assets/templates/` and references just-in-time, not upfront.

---

## Step-by-Step Workflow

### Phase 0: Planning Source Discovery (MANDATORY, runs before Phase 1)

This phase decides WHICH layer the plan will be derived from and stamps the answer onto every emitted manifest as `planning_source`. It runs in **both** modes; the only difference is which selections are allowed.

#### Step 0.1 — Inventory available planning inputs

Detect each potential planning source. Record presence/absence in working memory.

```python
from pathlib import Path
from databricks.sdk import WorkspaceClient

def detect_planning_sources(catalog: str, user_schema_prefix: str) -> dict:
    """Return a dict describing every potential planning source that exists.

    Priority order (highest fidelity first):
      1. deployed_gold     — live tables in <catalog>.<prefix>_gold
      2. gold_design       — gold_layer_design/yaml/*.yaml authored, deployment may or may not be done
      3. deployed_silver   — live tables in <catalog>.<prefix>_silver
      4. deployed_bronze   — live tables in <catalog>.<prefix>_bronze
      5. source_csv        — data_product_accelerator/context/*.csv (last resort)
    """
    w = WorkspaceClient()
    sources = {}
    for layer, schema in (
        ("deployed_gold",   f"{user_schema_prefix}_gold"),
        ("deployed_silver", f"{user_schema_prefix}_silver"),
        ("deployed_bronze", f"{user_schema_prefix}_bronze"),
    ):
        try:
            tables = list(w.tables.list(catalog_name=catalog, schema_name=schema))
            sources[layer] = {"schema": f"{catalog}.{schema}", "table_count": len(tables)} if tables else None
        except Exception:
            sources[layer] = None
    yaml_dir = Path("gold_layer_design/yaml")
    if yaml_dir.exists() and any(yaml_dir.glob("*.yaml")):
        sources["gold_design"] = {"path": str(yaml_dir), "yaml_count": len(list(yaml_dir.glob("*.yaml")))}
    else:
        sources["gold_design"] = None
    csvs = list(Path("data_product_accelerator/context").glob("*.csv"))
    sources["source_csv"] = {"paths": [str(c) for c in csvs]} if csvs else None
    return sources
```

#### Step 0.2 — Select the planning source by mode

| Mode | Allowed `selected_layer` values | Selection rule |
|------|---------------------------------|----------------|
| `acceleration` (default) | `deployed_gold`, `gold_design` | Pick `deployed_gold` if present; else `gold_design` ONLY when explicitly accepted; else **STOP** with a Gold-required remediation message. |
| `workshop` | `deployed_gold`, `gold_design`, `deployed_silver`, `deployed_bronze`, `source_csv` | Pick the highest-priority source present. Never silently fall through to a lower layer when a higher one exists. |

**Acceleration STOP message:**
> Planning in acceleration mode requires the Gold layer. Run the Gold Layer Design and Setup skills first, or re-run with `planning_mode: workshop` to plan from a lower layer.

**Workshop selection log (must be printed):**
```
Phase 0 — Planning source selected: <selected_layer>
  Available: deployed_gold=<bool>, gold_design=<bool>, deployed_silver=<bool>, deployed_bronze=<bool>, source_csv=<bool>
  Reason: highest-fidelity available input under workshop mode
```

#### Step 0.3 — Derive readiness markers

Compute the readiness fields that every emitted manifest must include. `requires_gold_promotion` is **advisory only** — it is a hint for production hardening, never a deployment gate.

```python
def readiness_for(selected_layer: str, mode: str) -> dict:
    if selected_layer == "deployed_gold":
        # Production-deployable from Gold.
        return {"implementation_readiness": "gold_ready",
                "requires_gold_promotion": False}
    if selected_layer == "gold_design":
        # Deployable once the Gold layer is provisioned.
        return {"implementation_readiness": "gold_design_only",
                "requires_gold_promotion": False}
    # Silver, Bronze, source CSV — workshop only.
    if mode != "workshop":
        raise SystemExit("Non-Gold planning sources are only allowed in workshop mode.")
    if selected_layer in {"deployed_silver", "deployed_bronze"}:
        # Workshop builds the semantic layer directly on top of Silver/Bronze.
        # Gold promotion is recommended for production but not required to deploy.
        return {"implementation_readiness": "workshop_deployable",
                "requires_gold_promotion": False}
    if selected_layer == "source_csv":
        # No live tables — planning contract only; downstream stages will not
        # attempt to deploy until at least one live layer exists.
        return {"implementation_readiness": "workshop_draft",
                "requires_gold_promotion": False}
    raise SystemExit(f"Unknown selected_layer={selected_layer!r}")
```

**Readiness state semantics:**

| `implementation_readiness` | When | Downstream behavior |
|---|---|---|
| `gold_ready` | Acceleration or workshop on `deployed_gold` | Full production deploy |
| `gold_design_only` | Acceleration or workshop on `gold_design` (Gold YAML, no live tables yet) | Deploy after Gold provisioning; live-catalog checks advisory |
| `workshop_deployable` | Workshop on `deployed_silver` or `deployed_bronze` | Deploy semantic layer / Genie Spaces directly against the Silver or Bronze schema; Gold promotion is an advisory next step |
| `workshop_draft` | Workshop on `source_csv` only | Planning contract only; downstream stages stop and ask for at least one live layer |

#### Step 0.4 — Stamp `planning_source` onto every manifest

Every manifest emitted by Phases 1–3 (semantic-layer, observability, ml, genai-agents, gold-dependency, source-dependency) MUST carry a top-level block:

```yaml
planning_source:
  selected_layer: deployed_gold | gold_design | deployed_silver | deployed_bronze | source_csv
  schema: "<catalog>.<schema>"            # e.g. main.acme_gold (omit/null for source_csv)
  source_yaml_dir: "gold_layer_design/yaml"   # only when selected_layer = gold_design
  source_csv_paths: ["data_product_accelerator/context/<file>.csv"]  # only for source_csv
  selected_at: "<ISO-8601 UTC>"
implementation_readiness: gold_ready | gold_design_only | workshop_deployable | workshop_draft
requires_gold_promotion: true | false   # advisory only; never a deployment gate
```

Downstream orchestrators (semantic-layer, observability, ml, genai-agents) read these fields:

- `gold_ready` / `gold_design_only` / `workshop_deployable` — proceed with deployment against the layer the manifest declares (`gold_schema` for Gold sources; `silver_schema` / `bronze_schema` for workshop deployments on Silver/Bronze).
- `workshop_draft` (only emitted when `selected_layer = source_csv`) — stop before deployment; the plan is a contract only.
- `requires_gold_promotion` is advisory; it influences messaging, not gating.

#### Step 0.5 — Mode-aware Phase 1 prerequisites table

The `Prerequisites Status` table in Phase 1 must reflect the selected source. Layers above the selected one are still valid; layers below it (or absent) are marked `N/A` or `Planned only`. See the updated `assets/templates/prerequisites-template.md` for the dynamic format.

---

### Phase 1: Requirements Gathering

#### Project Information

| Field | Your Value |
|-------|------------|
| Project Name | {project_name} |
| Business Domain | {hospitality, retail, healthcare, finance, etc.} |
| Primary Use Cases | {use_case_1, use_case_2, use_case_3, etc.} |
| Target Stakeholders | {executives, analysts, data scientists, operations} |

#### Prerequisites Status (filled by Phase 0)

The status of each layer must reflect what Phase 0 detected. Layers above the selected planning source are "✅ Complete"; the selected layer itself is the source the plan was derived from; layers below or absent are `N/A` or `Planned only`. Use the dynamic format from `assets/templates/prerequisites-template.md`.

| Layer | Count | Status (mode-aware) |
|-------|-------|---------------------|
| Bronze Tables | {n} | ✅ Complete / N/A / Planned only |
| Silver Tables | {m} | ✅ Complete / N/A / Planned only |
| Gold Dimensions | {d} | ✅ Complete / Designed only / N/A |
| Gold Facts | {f} | ✅ Complete / Designed only / N/A |
| **Selected planning source** | — | `{planning_source.selected_layer}` (from Phase 0) |

#### Define Agent Domains

Derive domains from your business questions and **planning-source table groupings** (see Artifact Rationalization Framework). Use Gold table groupings when `planning_source.selected_layer` is `deployed_gold` or `gold_design`; otherwise group on the selected source-layer tables (Silver / Bronze / source CSV entities). Do not force a fixed number — let the data model and use cases determine natural boundaries.

##### Required Reads (Before Proceeding)

Before defining domains and use cases, you MUST have read these references. Check each off in your Skill Usage Summary:

- [ ] `references/rationalization-framework.md` — domain sizing rules, TVF creation gate, Genie Space capacity planning, naming conventions
- [ ] `references/worked-example-wanderbricks.md` (if the project is WanderBricks or a hospitality/vacation-rental domain) OR `references/industry-domain-patterns.md` (otherwise) — domain naming, use case card depth, reference SQL patterns

If a worked example matches your project, treat it as the **primary format reference** for use case cards and artifact designs — adapt, don't reinvent. "MANDATORY" means read it; note in the Skill Usage Summary if the reference did not change your decisions, but still read it.

**Workshop mode:** `references/workshop-mode-profile.md` should already be loaded when `planning_mode: workshop` was detected — see its **Document Scope** section, since workshop mode changes artifact counts but NOT which documents to produce.

| Domain | Icon | Focus Area | Key Planning Assets | Est. Business Questions |
|--------|------|------------|---------------------|------------------------|
| {Domain 1} | {emoji} | {focus} | {tables from selected layer} | {count} |
| {Domain 2} | {emoji} | {focus} | {tables from selected layer} | {count} |
| ... | ... | ... | ... | ... |

**Sizing check:** If a domain has < 3 business questions, consider merging it. If two domains share > 70% of their planning assets (Gold/Silver/Bronze tables, depending on `planning_source.selected_layer`), consolidate.

See [Industry Domain Patterns](references/industry-domain-patterns.md) for examples by industry.

#### Phase 1 Addendum Selection

| # | Addendum | Include? | Artifact Count |
|---|----------|----------|----------------|
| 1.1 | ML Models | {Yes/No} | {count} |
| 1.2 | Table-Valued Functions | {Yes/No} | {count} |
| 1.3 | Metric Views | {Yes/No} | {count} |
| 1.4 | Lakehouse Monitoring | {Yes/No} | {count} |
| 1.5 | AI/BI Dashboards | {Yes/No} | {count} |
| 1.6 | Genie Spaces | {Yes/No} | {count} |
| 1.7 | Alerting Framework | {Yes/No} | {count} |

#### Key Business Questions by Domain

List 5-10 key questions per domain that the solution must answer:

**{Domain 1}:**
1. {Question 1}
2. {Question 2}
3. {Question 3}
4. {Question 4}
5. {Question 5}

#### Use Case Catalog

After defining business questions and selecting addendums, consolidate into a **Use Case Catalog** — one entry per distinct analytical or operational problem the solution will address. Each use case ties business questions to the **planning assets** (Gold tables in acceleration; selected-layer tables in workshop) and artifacts that solve them. Use `assets/templates/use-case-catalog-template.md` for the full format.

| UC# | Use Case Name | Domain | Planning Assets | Artifact Types | Example Question |
|-----|--------------|--------|-----------------|---------------|-----------------|
| UC-001 | {Descriptive Name} | {Domain} | `fact_*`, `dim_*`  *(or `silver_*` / `bronze_*` in workshop drafts)* | TVF, MV, Dashboard | "{Natural language question}?" |
| UC-002 | ... | ... | ... | ... | ... |

**Use Case Catalog Rules:**
- Every use case MUST include 3-5 business questions phrased in natural language
- Every business question from the domain sections above MUST map to at least one use case
- Every artifact in the addendum summaries MUST trace back to at least one use case question
- Questions should be phrased as stakeholders would ask them (these become Genie benchmark candidates)
- Group related questions into a single use case when they share the same planning assets (Gold tables in acceleration; selected-layer tables in workshop) and grain

See [Worked Example: Wanderbricks](references/worked-example-wanderbricks.md) for 3 fully worked-out use case cards.

**Stakeholder Checkpoint:** After generating the use case catalog, present the Use Case Summary table to the user for review. Whether to block depends on prompt specificity:

- **If the user's prompt listed explicit output steps** (e.g., "analyze Gold layer, generate use-case plans, produce manifests"): treat as pre-approval. Include the summary table at the top of your response and proceed without blocking. Note: *"Use cases derived from Gold layer analysis — let me know if adjustments are needed before the next pipeline stage."*
- **If the user's prompt was open-ended** (e.g., "create a plan"): pause and ask for confirmation before proceeding to addendum generation.

If the user requests changes after seeing the summary, update the catalog and domain questions before continuing.

### Phase 2: Plan Document Generation

Create plan documents using templates in the following order:

1. **README** — `assets/templates/plans-readme-template.md` (plan index)
2. **Prerequisites** — `assets/templates/prerequisites-template.md` (data layer summary)
3. **Use Case Catalog** — `assets/templates/use-case-catalog-template.md` (consolidated use case definitions)
4. **Phase 1 Master** — `assets/templates/phase1-use-cases-template.md` (analytics artifacts)
5. **Addendums** (selected in Phase 1):
   - TVFs — `assets/templates/phase1-tvfs-template.md`
   - Alerting — `assets/templates/phase1-alerting-template.md`
   - Genie Spaces — `assets/templates/phase1-genie-spaces-template.md`
6. **Phase 2** — `assets/templates/phase2-agent-framework-template.md` (AI agents)
7. **Phase 3** — `assets/templates/phase3-frontend-template.md` (user interface)

#### Phase 2 Completion Gate

Before proceeding to Phase 3 (Manifests), verify that ALL selected plan documents exist on disk.

> **CANONICAL NUMBERING REFERENCE.** Every filename in the table below matches [`assets/addendum-numbering.md`](./assets/addendum-numbering.md) — the single source of truth for Phase 1 addendum numbers. If you are adding a new addendum, extend `addendum-numbering.md` **first**, then use the new name here. Never invent a number (e.g. the stale `phase1-addendum-1.1-dashboards.md` is forbidden — dashboards are `1.5-aibi-dashboards.md`).

| Document | Template | Required? |
|----------|----------|-----------|
| `plans/README.md` | `plans-readme-template.md` | ALWAYS |
| `plans/prerequisites.md` | `prerequisites-template.md` | ALWAYS |
| `plans/use-case-catalog.md` | `use-case-catalog-template.md` | ALWAYS |
| `plans/phase1-use-cases.md` | `phase1-use-cases-template.md` | ALWAYS |
| `plans/phase1-addendum-1.2-tvfs.md` | `phase1-tvfs-template.md` | If TVFs selected |
| `plans/phase1-addendum-1.3-metric-views.md` | (inline) | If Metric Views selected |
| `plans/phase1-addendum-1.6-genie-spaces.md` | `phase1-genie-spaces-template.md` | If Genie Spaces selected |
| `plans/phase1-addendum-1.4-lakehouse-monitoring.md` | (inline) | If Monitoring selected |
| `plans/phase1-addendum-1.5-aibi-dashboards.md` | (inline) | If Dashboards selected |
| `plans/phase1-addendum-1.7-alerting.md` | `phase1-alerting-template.md` | If Alerting selected |
| `plans/phase1-addendum-1.1-ml-models.md` | (inline) | If ML selected |

**If any required document is missing, create it from its template before generating manifests.** Manifests reference these files in `generated_from.plan_addendums` — they must exist on disk. **Workshop mode does not waive this gate**: artifact counts inside each document are capped, but the document set is unchanged.

#### Phase 2 Step 5 — Emit Source Dependency Manifest (MANDATORY, layer-aware)

**Before Phase 3 manifest generation, extract every source-layer table/column referenced across all plan addendums into a single machine-readable manifest.** This becomes the contract validated against the live catalog in the next step.

The manifest filename and shape depend on the Phase 0 selected layer:

| `planning_source.selected_layer` | File path | Top-level key |
|----------------------------------|-----------|---------------|
| `deployed_gold` or `gold_design` | `plans/manifests/gold-dependency-manifest.yaml` | `gold_dependencies:` |
| `deployed_silver`, `deployed_bronze`, `source_csv` (workshop) | `plans/manifests/source-dependency-manifest.yaml` | `source_dependencies:` |

Acceleration mode emits ONLY `gold-dependency-manifest.yaml` (existing behavior). Workshop mode emits whichever file matches its selected layer; when the selected layer is Gold, it emits `gold-dependency-manifest.yaml` for backward compatibility. Both shapes share the same `referenced_by` semantics.

```yaml
# plans/manifests/gold-dependency-manifest.yaml  (acceleration, or workshop with Gold source)
planning_mode: acceleration  # or workshop — mirror the parent manifest's planning_mode
planning_source:
  selected_layer: deployed_gold   # or gold_design
  schema: "<catalog>.<gold_schema>"
implementation_readiness: gold_ready  # or gold_design_only
requires_gold_promotion: false
generated_from:
  plan_addendums:
    - plans/phase1-use-cases.md
    - plans/phase1-addendum-1.2-tvfs.md
    - plans/phase1-addendum-1.3-metric-views.md
    - plans/phase1-addendum-1.5-aibi-dashboards.md
    - plans/phase1-addendum-1.6-genie-spaces.md
gold_dependencies:
  - table: fact_booking_daily
    columns: [booking_key, property_key, booking_date, net_revenue, nights]
    referenced_by:
      - semantic-layer/metric_views/revenue_analytics_metrics.yaml
      - semantic-layer/tvfs/get_revenue_by_property
      - observability/dashboards/revenue_overview.lvdash.json
  - table: dim_property
    columns: [property_key, property_name, destination_id, is_current]
    referenced_by:
      - semantic-layer/metric_views/revenue_analytics_metrics.yaml
summary:
  total_tables: 12
  total_columns: 84
  total_referenced_by: 37
```

```yaml
# plans/manifests/source-dependency-manifest.yaml  (workshop only; Bronze/Silver/source CSV)
planning_mode: workshop
planning_source:
  selected_layer: deployed_silver   # or deployed_bronze, source_csv
  schema: "<catalog>.<silver_schema>"   # null for source_csv
  source_csv_paths: []                  # populated for source_csv
implementation_readiness: workshop_deployable   # workshop_draft only when selected_layer=source_csv
requires_gold_promotion: false                  # advisory only; never gates deployment
generated_from:
  plan_addendums:
    - plans/phase1-use-cases.md
source_dependencies:
  - table: silver_bookings
    columns: [booking_id, property_id, booking_date, gross_amount]
    referenced_by:
      - planning/use_case_cards/revenue_overview.md
summary:
  total_tables: 4
  total_columns: 22
  total_referenced_by: 6
```

**Rules:**
- One entry per **distinct** source table; union all column references from all plan addendums.
- `referenced_by` uses relative artifact paths so downstream fixes can trace artifacts back to the missing column.
- Emit this manifest even when `planning_mode: workshop` — the workshop cap applies to artifact counts, not to manifest accuracy.
- The shape is identical to Gold's: only the filename, top-level key (`source_dependencies` vs `gold_dependencies`), and `planning_source` block change.

#### Phase 2 Step 6 — Live-Catalog Intersection (STOP / WARN Rule, MANDATORY, mode-aware)

**Immediately after emitting the dependency manifest, query the live catalog and cross-reference every table/column reference.** Downstream stages all assume the planning source is consistent with the live catalog — catching gaps HERE saves 5+ deploy cycles later.

The validation behavior depends on `planning_mode` and `planning_source.selected_layer`:

| Mode + selected layer | Behavior | Artifacts |
|------------------------|----------|-----------|
| `acceleration` + `deployed_gold` | **Fail-loud STOP** if any gap | Emit `plans/gold-gap-remediation.md`; raise |
| `acceleration` + `gold_design` (only when explicitly accepted) | **Warn** (Gold may not be deployed yet) | Emit `plans/gold-gap-remediation.md`; do NOT raise |
| `workshop` + `deployed_gold` | **Fail-loud STOP** if any gap (same as acceleration) | Emit `plans/gold-gap-remediation.md`; raise |
| `workshop` + `gold_design` | **Warn** | Emit `plans/gold-gap-remediation.md`; continue |
| `workshop` + `deployed_silver` / `deployed_bronze` | **Warn** | Emit `plans/source-gap-remediation.md`; continue |
| `workshop` + `source_csv` | **Skip live intersection** (no live schema to compare against) | None |

```python
import yaml
from pathlib import Path
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
catalog = "<lakehouse_default_catalog>"

# Choose which manifest to load based on Phase 0 selection.
gold_path   = Path("plans/manifests/gold-dependency-manifest.yaml")
source_path = Path("plans/manifests/source-dependency-manifest.yaml")
manifest_path = gold_path if gold_path.exists() else source_path
manifest = yaml.safe_load(manifest_path.read_text())

planning_mode    = manifest.get("planning_mode", "acceleration")
selected_layer   = manifest.get("planning_source", {}).get("selected_layer", "deployed_gold")
target_schema    = manifest.get("planning_source", {}).get("schema")  # e.g. "<catalog>.<schema>"
deps_key         = "gold_dependencies" if "gold_dependencies" in manifest else "source_dependencies"

# Workshop + source_csv: nothing to intersect against, skip.
if selected_layer == "source_csv":
    print("ℹ Skipping live-catalog intersection — selected_layer=source_csv has no live schema.")
else:
    # Pull every table and column from the live target schema in one shot.
    schema_only = target_schema.split(".")[-1]
    live_cols = (
        spark.sql(f"""
          SELECT table_name, column_name, full_data_type
          FROM {catalog}.information_schema.columns
          WHERE table_schema = '{schema_only}'
        """).collect()
    )
    live_index = {}
    for row in live_cols:
        live_index.setdefault(row.table_name, {})[row.column_name] = row.full_data_type

    missing_tables, missing_columns = [], []
    for dep in manifest[deps_key]:
        tbl = dep["table"]
        if tbl not in live_index:
            missing_tables.append({"table": tbl, "referenced_by": dep["referenced_by"]})
            continue
        for col in dep["columns"]:
            if col not in live_index[tbl]:
                missing_columns.append({"table": tbl, "column": col, "referenced_by": dep["referenced_by"]})

    if missing_tables or missing_columns:
        # Pick remediation filename based on dependency kind.
        remediation_path = Path(
            "plans/gold-gap-remediation.md" if deps_key == "gold_dependencies"
            else "plans/source-gap-remediation.md"
        )
        remediation_path.write_text(
            f"# {('Gold' if deps_key == 'gold_dependencies' else 'Source')} Dependency Gap Remediation\n\n"
            f"The following references in plan addendums do not exist in `{target_schema}`.\n\n"
            "## Missing tables\n\n"
            + "\n".join(f"- **{m['table']}** — referenced by {m['referenced_by']}" for m in missing_tables)
            + "\n\n## Missing columns\n\n"
            + "\n".join(
                f"- **{m['table']}.{m['column']}** — referenced by {m['referenced_by']}"
                for m in missing_columns
            )
            + "\n\n## Next steps\n\n"
            "1. Acceleration / Gold source: add tables to `gold_layer_design/yaml/` and re-run "
            "`gold/01-gold-layer-setup`.\n"
            "2. Workshop / Silver or Bronze: extend the corresponding setup skill (silver/bronze) "
            "and re-run.\n"
            "3. Re-run this Planning skill to regenerate manifests.\n"
        )

        is_strict = (planning_mode == "acceleration" and selected_layer == "deployed_gold") \
                 or (planning_mode == "workshop"     and selected_layer == "deployed_gold")
        msg = (f"{deps_key} gap detected: {len(missing_tables)} missing tables, "
               f"{len(missing_columns)} missing columns. See {remediation_path}.")
        if is_strict:
            raise RuntimeError(msg + " STOP — downstream orchestrators cannot proceed.")
        else:
            print(f"⚠ {msg} Continuing under non-strict mode/layer; downstream manifests "
                  f"will carry implementation_readiness=" + manifest.get("implementation_readiness", ""))
    else:
        print(f"✅ {deps_key} intersected cleanly with live catalog `{target_schema}`.")
```

**Escape flag:** If the user has an out-of-band reason to bypass the gap (e.g., Gold is intentionally incomplete for a phased rollout), they can pass `planning_allow_gold_gap: true` in their prompt. In that case, still emit the remediation file as a warning, but proceed to Phase 3 with a prominent `gold_gap_acknowledged: true` marker in every downstream manifest. This flag does NOT relax mode-specific behavior; non-Gold workshop manifests already carry `implementation_readiness: workshop_deployable` (Silver/Bronze) or `workshop_draft` (source CSV).

### Phase 3: Manifest Generation (Plan-as-Contract)

After creating plan documents, generate **machine-readable YAML manifests** that downstream orchestrators consume as implementation contracts.

**Why manifests?** The "Extract, Don't Generate" principle applies to the planning-to-implementation handoff. Manifests ensure downstream orchestrators implement exactly what was planned — no missed artifacts, no naming inconsistencies.

**MANDATORY: Read the manifest generation guide:**

| # | Reference Path | What It Provides |
|---|----------------|------------------|
| 1 | `references/manifest-generation-guide.md` | Full manifest workflow, validation, consumption pattern |

**Steps:**
1. Review Gold layer YAML schemas in `gold_layer_design/yaml/`
2. For each plan addendum, extract the concrete artifact definitions
3. Generate 4 YAML manifests using templates from `assets/templates/manifests/`:
   - `plans/manifests/semantic-layer-manifest.yaml` — TVFs, Metric Views, Genie Spaces
   - `plans/manifests/observability-manifest.yaml` — Monitors, Dashboards, Alerts
   - `plans/manifests/ml-manifest.yaml` — Feature Tables, Models, Experiments
   - `plans/manifests/genai-agents-manifest.yaml` — Agents, Tools, Eval Datasets
4. For each artifact in a manifest, add `use_case_refs` listing the UC# it implements (from `plans/use-case-catalog.md`)
5. Validate all table/column references exist in Gold YAML
6. Verify summary counts match actual artifact counts
7. Run `python scripts/validate_use_case_coverage.py plans/use-case-catalog.md` to verify coverage
8. Commit manifests alongside plan documents

**Key principle:** Every artifact in a manifest MUST trace back to (a) a Gold layer table and (b) a business question from the plan addendum.

**Output Structure:**
```
plans/
├── use-case-catalog.md                    # Consolidated use case definitions
├── manifests/
│   ├── semantic-layer-manifest.yaml       # → consumed by semantic-layer/00-*
│   ├── observability-manifest.yaml        # → consumed by monitoring/00-*
│   ├── ml-manifest.yaml                   # → consumed by ml/00-*
│   └── genai-agents-manifest.yaml         # → consumed by genai-agents/00-*
```

**Downstream consumption:** Each downstream orchestrator (stages 6-9) has a **Phase 0: Read Plan** step that reads its manifest. If the manifest doesn't exist (e.g., user skipped Planning), the orchestrator falls back to self-discovery from Gold tables.

---

## Plan Structure Framework

### Standard Project Phases

```
plans/
├── README.md                              # Index and overview
├── use-case-catalog.md                    # Consolidated use case definitions
├── prerequisites.md                       # Bronze/Silver/Gold summary (optional)
├── phase1-use-cases.md                    # Analytics artifacts (master)
│   ├── phase1-addendum-1.1-ml-models.md
│   ├── phase1-addendum-1.2-tvfs.md
│   ├── phase1-addendum-1.3-metric-views.md
│   ├── phase1-addendum-1.4-lakehouse-monitoring.md
│   ├── phase1-addendum-1.5-aibi-dashboards.md
│   ├── phase1-addendum-1.6-genie-spaces.md
│   └── phase1-addendum-1.7-alerting.md
├── phase2-agent-framework.md              # AI Agents
├── phase3-frontend-app.md                 # User Interface
└── manifests/                             # Machine-readable contracts
    ├── semantic-layer-manifest.yaml       # → semantic-layer/00-*
    ├── observability-manifest.yaml        # → monitoring/00-*
    ├── ml-manifest.yaml                   # → ml/00-*
    └── genai-agents-manifest.yaml         # → genai-agents/00-*
```

### Phase Dependencies

```
Prerequisites (Bronze → Silver → Gold) → Phase 1 (Use Cases) → Phase 2 (Agents) → Phase 3 (Frontend)
         [COMPLETE]                               ↓
                                           All Addendums
```

## Agent Domain Framework

### Core Principle

**ALL artifacts across ALL phases MUST be organized by Agent Domain.** This ensures:
- Consistent categorization across 100+ artifacts
- Clear ownership by future AI agents
- Easy discoverability for users
- Aligned tooling for each domain

### Agent Domain Application

Every artifact (TVF, Metric View, Dashboard, Alert, ML Model, Monitor, Genie Space) must:
1. Be tagged with its Agent Domain
2. Use the domain's planning assets (Gold tables in acceleration; selected-layer tables in workshop)
3. Answer domain-specific questions
4. Be grouped with related domain artifacts in documentation

**Example Pattern:**
```markdown
## {Domain}: get_{metric}_by_{dimension}

**Agent Domain:** {Domain}
**Planning Assets:** `fact_{entity}`, `dim_{entity}`   # or `silver_{entity}` / `bronze_{entity}` in workshop drafts
**Business Questions:** "What are the top {metric} by {dimension}?"
```

See [Industry Domain Patterns](references/industry-domain-patterns.md) for domain templates by industry.

## Agent Layer Architecture Pattern

### Core Principle: Agents Use Genie Spaces as Query Interface

**AI Agents DO NOT query data assets directly.** Instead, they use Genie Spaces as their natural language query interface. Genie Spaces translate natural language to SQL and route to appropriate tools.

```
USERS (Natural Language)
    ↓
PHASE 2: AI AGENT LAYER (LangChain/LangGraph)
    ├── Orchestrator Agent (intent classification)
    └── Specialized Agents (1 per domain)
            ↓
PHASE 1.6: GENIE SPACES (NL Query Execution)
    ├── {Domain 1} Intelligence Genie Space
    ├── {Domain 2} Intelligence Genie Space
    └── Unified {Project} Monitor
            ↓
PHASE 1: DATA ASSETS (Agent Tools)
    ├── Metric Views (pre-aggregated - use FIRST)
    ├── TVFs (parameterized queries)
    ├── ML Predictions (ML-powered insights)
    └── Lakehouse Monitors (drift detection)
 

…(truncated)