Project Planning

Project Plan Methodology for Databricks Solutions

Planning Mode

Default: Data Product Acceleration — full breadth, all domains, all artifacts, Gold layer required as planning basis. This is the standard behavior described in this entire skill document below.

Workshop mode is available for Learning & Enablement scenarios with hard artifact caps and layer flexibility — it can plan from the best available source layer (Gold, Gold design YAML, Silver, Bronze, or source CSV). Workshop mode is NEVER activated unless the user includes the exact phrase planning_mode: workshop in their prompt.

Mode vs source layer: planning_mode (acceleration | workshop) controls artifact caps and validation strictness. planning_source.selected_layer (gold | gold_design | silver | bronze | source_csv) records which input the plan was derived from and is set automatically by Phase 0 below. Acceleration mode FORCES selected_layer = gold (or gold_design only if explicitly allowed). Workshop mode picks the best available source via the Phase 0 priority order and stamps it onto every manifest.

Mode Detection Rules

1. Default is ALWAYS acceleration. If the user does not explicitly declare workshop mode, use acceleration.
2. Workshop mode requires EXPLICIT opt-in. The user must include one of these EXACT phrases:
   • planning_mode: workshop
   • "workshop mode"
   • "use workshop mode"
3. Do NOT infer workshop mode from words like "small", "simple", "demo", "limited", "quick", "basic", "training", or "few". These are NOT triggers. A user may want a narrow-scope acceleration plan — that's still acceleration mode with fewer use cases.
4. When in doubt, ask. If the user's intent is ambiguous (e.g., "Create a plan for a workshop"), ask: "Would you like full Data Product Acceleration mode (default) or Workshop mode with limited artifacts? To use workshop mode, include planning_mode: workshop in your request."
5. Confirm mode at the start. The first line of any plan output should state the active mode:
   • **Planning Mode:** Data Product Acceleration (default)
   • **Planning Mode:** Workshop (explicit opt-in — artifact caps active)
6. When workshop mode is activated, read references/workshop-mode-profile.md for artifact caps, phase scope, and selection criteria. Do NOT read that reference otherwise.
7. Propagate mode to manifests. Add planning_mode: workshop or planning_mode: acceleration to all generated manifest YAML files. Downstream orchestrators seeing workshop MUST NOT expand beyond the listed artifacts via self-discovery.

Overview

Comprehensive methodology for creating multi-phase project plans for Databricks data platform solutions. This skill combines interactive project planning with architectural methodology, including templates, worked examples, and quality standards.

Key Assumption (mode-aware):

• Acceleration mode (default): Planning starts AFTER Bronze ingestion AND Gold layer design are complete. Gold is the required planning basis. These are prerequisites, not phases. Phase 0 will stop with a remediation message if Gold is missing.
• Workshop mode (planning_mode: workshop): Planning AND deployment are layer-agnostic. Phase 0 selects the highest-fidelity input present from: deployed Gold, Gold design YAML, deployed Silver, deployed Bronze, or a source schema CSV. The selected layer is stamped onto every manifest as planning_source.selected_layer. Workshop manifests built from Silver or Bronze are marked implementation_readiness: workshop_deployable — downstream stages (semantic-layer, observability, ml, genai-agents) deploy directly against the selected layer. Workshop manifests built from a source CSV are marked implementation_readiness: workshop_draft (planning contract only — no live tables to deploy against). requires_gold_promotion is an advisory field; it is recommended for production but never blocks deployment.

When to Use This Skill

Use this skill when:

• Creating architectural plans for Databricks data platform projects
• Building observability, analytics, or monitoring solutions
• Planning multi-artifact solutions (TVFs, Metric Views, Dashboards, Genie Spaces, Alerts, ML Models)
• Developing agent-based frameworks for platform management
• Creating frontend applications for data platform interaction
• Starting a new project after Gold layer is complete

Idempotency Guard (Run FIRST)

Before regenerating plans, detect existing artifacts to avoid clobbering work-in-progress. A common failure mode is regenerating plans/ wholesale on a re-run and overwriting user edits to manifests, addendums, or the Use Case Catalog.

from pathlib import Path

PLANS_DIR = Path("plans")
if PLANS_DIR.exists() and any(PLANS_DIR.iterdir()):
    existing = sorted(p.relative_to(".") for p in PLANS_DIR.rglob("*") if p.is_file())
    print("Existing plan artifacts detected:")
    for p in existing:
        print(f"  {p}  (mtime={Path(p).stat().st_mtime})")
    print(
        "\nHow would you like to proceed?\n"
        "  - regenerate  (DELETE and rebuild all plan files — destructive)\n"
        "  - incremental (keep existing files, only emit MISSING artifacts)\n"
        "  - skip        (exit this orchestrator — recommended default)\n"
    )

Rules:

• Default is skip. If the user is silent or ambiguous, assume skip and exit with a summary of existing files.
• regenerate must be explicit. Confirm the action ("I will delete N files under plans/ — proceed?") before doing anything destructive.
• incremental is the right choice when downstream orchestrators (semantic-layer, observability, ml, genai-agents) reported a missing manifest — only emit the missing manifest, not the whole tree.

Escape flag: Users can set planning_allow_overwrite: true in their prompt to skip the idempotency check (equivalent to choosing regenerate without interactive confirmation).

Quick Start (5 Minutes)

Fast Track: Create Your Project Plan

# 1. Verify prerequisites for your mode:
#    Acceleration (default):
#      - Bronze ingestion ✅
#      - Silver DLT streaming ✅
#      - Gold dimensional model ✅ (REQUIRED)
#    Workshop (planning_mode: workshop):
#      - At least ONE of: deployed Gold, gold_layer_design/yaml/, deployed Silver,
#        deployed Bronze, or data_product_accelerator/context/*.csv
#      - Phase 0 picks the highest-fidelity input automatically.

# 2. Run this prompt with your project info:
"Create a phased project plan for {project_name} with:
- Planning assets: {n} tables (Gold/Silver/Bronze depending on what is available)
- Use cases: {use_case_1, use_case_2, use_case_3, etc.}
- Target audience: {executives, analysts, data scientists}
- Agent domains: {domain1, domain2, domain3, domain4, domain5}"

# 3. Output: Complete plan structure in plans/ folder
#    - Acceleration emits gold-dependency-manifest.yaml.
#    - Workshop emits gold-dependency-manifest.yaml OR source-dependency-manifest.yaml
#      depending on the selected planning source layer.

Key Decisions (Answer These First)

Working Memory Management

This orchestrator spans 3 phases. To maintain coherence without context pollution:

After each phase, persist a brief summary note capturing:

• Phase 1: Domain list with Gold table mappings, addendum selections, business questions per domain, artifact count estimates
• Phase 2: Plan document file paths, cross-references verified, total artifact counts by type
• Phase 3: Manifest file paths (semantic-layer, observability, ml, genai-agents), validation results, summary counts

What to keep in working memory: Current phase's template, domain list + artifact inventory, and previous phase's summary. Discard intermediate outputs — they are on disk. Read templates from assets/templates/ and references just-in-time, not upfront.

Step-by-Step Workflow

Phase 0: Planning Source Discovery (MANDATORY, runs before Phase 1)

This phase decides WHICH layer the plan will be derived from and stamps the answer onto every emitted manifest as planning_source. It runs in both modes; the only difference is which selections are allowed.

Step 0.1 — Inventory available planning inputs

Detect each potential planning source. Record presence/absence in working memory.

from pathlib import Path
from databricks.sdk import WorkspaceClient

def detect_planning_sources(catalog: str, user_schema_prefix: str) -> dict:
    """Return a dict describing every potential planning source that exists.

    Priority order (highest fidelity first):
      1. deployed_gold     — live tables in <catalog>.<prefix>_gold
      2. gold_design       — gold_layer_design/yaml/*.yaml authored, deployment may or may not be done
      3. deployed_silver   — live tables in <catalog>.<prefix>_silver
      4. deployed_bronze   — live tables in <catalog>.<prefix>_bronze
      5. source_csv        — data_product_accelerator/context/*.csv (last resort)
    """
    w = WorkspaceClient()
    sources = {}
    for layer, schema in (
        ("deployed_gold",   f"{user_schema_prefix}_gold"),
        ("deployed_silver", f"{user_schema_prefix}_silver"),
        ("deployed_bronze", f"{user_schema_prefix}_bronze"),
    ):
        try:
            tables = list(w.tables.list(catalog_name=catalog, schema_name=schema))
            sources[layer] = {"schema": f"{catalog}.{schema}", "table_count": len(tables)} if tables else None
        except Exception:
            sources[layer] = None
    yaml_dir = Path("gold_layer_design/yaml")
    if yaml_dir.exists() and any(yaml_dir.glob("*.yaml")):
        sources["gold_design"] = {"path": str(yaml_dir), "yaml_count": len(list(yaml_dir.glob("*.yaml")))}
    else:
        sources["gold_design"] = None
    csvs = list(Path("data_product_accelerator/context").glob("*.csv"))
    sources["source_csv"] = {"paths": [str(c) for c in csvs]} if csvs else None
    return sources

Step 0.2 — Select the planning source by mode

Acceleration STOP message:

Planning in acceleration mode requires the Gold layer. Run the Gold Layer Design and Setup skills first, or re-run with planning_mode: workshop to plan from a lower layer.

Workshop selection log (must be printed):

Phase 0 — Planning source selected: <selected_layer>
  Available: deployed_gold=<bool>, gold_design=<bool>, deployed_silver=<bool>, deployed_bronze=<bool>, source_csv=<bool>
  Reason: highest-fidelity available input under workshop mode

Step 0.3 — Derive readiness markers

Compute the readiness fields that every emitted manifest must include. requires_gold_promotion is advisory only — it is a hint for production hardening, never a deployment gate.

def readiness_for(selected_layer: str, mode: str) -> dict:
    if selected_layer == "deployed_gold":
        # Production-deployable from Gold.
        return {"implementation_readiness": "gold_ready",
                "requires_gold_promotion": False}
    if selected_layer == "gold_design":
        # Deployable once the Gold layer is provisioned.
        return {"implementation_readiness": "gold_design_only",
                "requires_gold_promotion": False}
    # Silver, Bronze, source CSV — workshop only.
    if mode != "workshop":
        raise SystemExit("Non-Gold planning sources are only allowed in workshop mode.")
    if selected_layer in {"deployed_silver", "deployed_bronze"}:
        # Workshop builds the semantic layer directly on top of Silver/Bronze.
        # Gold promotion is recommended for production but not required to deploy.
        return {"implementation_readiness": "workshop_deployable",
                "requires_gold_promotion": False}
    if selected_layer == "source_csv":
        # No live tables — planning contract only; downstream stages will not
        # attempt to deploy until at least one live layer exists.
        return {"implementation_readiness": "workshop_draft",
                "requires_gold_promotion": False}
    raise SystemExit(f"Unknown selected_layer={selected_layer!r}")

Readiness state semantics:

Step 0.4 — Stamp planning_source onto every manifest

Every manifest emitted by Phases 1–3 (semantic-layer, observability, ml, genai-agents, gold-dependency, source-dependency) MUST carry a top-level block:

planning_source:
  selected_layer: deployed_gold | gold_design | deployed_silver | deployed_bronze | source_csv
  schema: "<catalog>.<schema>"            # e.g. main.acme_gold (omit/null for source_csv)
  source_yaml_dir: "gold_layer_design/yaml"   # only when selected_layer = gold_design
  source_csv_paths: ["data_product_accelerator/context/<file>.csv"]  # only for source_csv
  selected_at: "<ISO-8601 UTC>"
implementation_readiness: gold_ready | gold_design_only | workshop_deployable | workshop_draft
requires_gold_promotion: true | false   # advisory only; never a deployment gate

Downstream orchestrators (semantic-layer, observability, ml, genai-agents) read these fields:

• gold_ready / gold_design_only / workshop_deployable — proceed with deployment against the layer the manifest declares (gold_schema for Gold sources; silver_schema / bronze_schema for workshop deployments on Silver/Bronze).
• workshop_draft (only emitted when selected_layer = source_csv) — stop before deployment; the plan is a contract only.
• requires_gold_promotion is advisory; it influences messaging, not gating.

Step 0.5 — Mode-aware Phase 1 prerequisites table

The Prerequisites Status table in Phase 1 must reflect the selected source. Layers above the selected one are still valid; layers below it (or absent) are marked N/A or Planned only. See the updated assets/templates/prerequisites-template.md for the dynamic format.

Phase 1: Requirements Gathering

Project Information

Prerequisites Status (filled by Phase 0)

The status of each layer must reflect what Phase 0 detected. Layers above the selected planning source are "✅ Complete"; the selected layer itself is the source the plan was derived from; layers below or absent are N/A or Planned only. Use the dynamic format from assets/templates/prerequisites-template.md.

Define Agent Domains

Derive domains from your business questions and planning-source table groupings (see Artifact Rationalization Framework). Use Gold table groupings when planning_source.selected_layer is deployed_gold or gold_design; otherwise group on the selected source-layer tables (Silver / Bronze / source CSV entities). Do not force a fixed number — let the data model and use cases determine natural boundaries.

Required Reads (Before Proceeding)

Before defining domains and use cases, you MUST have read these references. Check each off in your Skill Usage Summary:

• references/rationalization-framework.md — domain sizing rules, TVF creation gate, Genie Space capacity planning, naming conventions
• references/worked-example-wanderbricks.md (if the project is WanderBricks or a hospitality/vacation-rental domain) OR references/industry-domain-patterns.md (otherwise) — domain naming, use case card depth, reference SQL patterns

If a worked example matches your project, treat it as the primary format reference for use case cards and artifact designs — adapt, don't reinvent. "MANDATORY" means read it; note in the Skill Usage Summary if the reference did not change your decisions, but still read it.

Workshop mode: references/workshop-mode-profile.md should already be loaded when planning_mode: workshop was detected — see its Document Scope section, since workshop mode changes artifact counts but NOT which documents to produce.

Sizing check: If a domain has < 3 business questions, consider merging it. If two domains share > 70% of their planning assets (Gold/Silver/Bronze tables, depending on planning_source.selected_layer), consolidate.

See Industry Domain Patterns for examples by industry.

Phase 1 Addendum Selection

Key Business Questions by Domain

List 5-10 key questions per domain that the solution must answer:

{Domain 1}:

1. {Question 1}
2. {Question 2}
3. {Question 3}
4. {Question 4}
5. {Question 5}

Use Case Catalog

After defining business questions and selecting addendums, consolidate into a Use Case Catalog — one entry per distinct analytical or operational problem the solution will address. Each use case ties business questions to the planning assets (Gold tables in acceleration; selected-layer tables in workshop) and artifacts that solve them. Use assets/templates/use-case-catalog-template.md for the full format.

Use Case Catalog Rules:

• Every use case MUST include 3-5 business questions phrased in natural language
• Every business question from the domain sections above MUST map to at least one use case
• Every artifact in the addendum summaries MUST trace back to at least one use case question
• Questions should be phrased as stakeholders would ask them (these become Genie benchmark candidates)
• Group related questions into a single use case when they share the same planning assets (Gold tables in acceleration; selected-layer tables in workshop) and grain

See Worked Example: Wanderbricks for 3 fully worked-out use case cards.

Stakeholder Checkpoint: After generating the use case catalog, present the Use Case Summary table to the user for review. Whether to block depends on prompt specificity:

• If the user's prompt listed explicit output steps (e.g., "analyze Gold layer, generate use-case plans, produce manifests"): treat as pre-approval. Include the summary table at the top of your response and proceed without blocking. Note: "Use cases derived from Gold layer analysis — let me know if adjustments are needed before the next pipeline stage."
• If the user's prompt was open-ended (e.g., "create a plan"): pause and ask for confirmation before proceeding to addendum generation.

If the user requests changes after seeing the summary, update the catalog and domain questions before continuing.

Phase 2: Plan Document Generation

Create plan documents using templates in the following order:

1. README — assets/templates/plans-readme-template.md (plan index)
2. Prerequisites — assets/templates/prerequisites-template.md (data layer summary)
3. Use Case Catalog — assets/templates/use-case-catalog-template.md (consolidated use case definitions)
4. Phase 1 Master — assets/templates/phase1-use-cases-template.md (analytics artifacts)
5. Addendums (selected in Phase 1):
   • TVFs — assets/templates/phase1-tvfs-template.md
   • Alerting — assets/templates/phase1-alerting-template.md
   • Genie Spaces — assets/templates/phase1-genie-spaces-template.md
6. Phase 2 — assets/templates/phase2-agent-framework-template.md (AI agents)
7. Phase 3 — assets/templates/phase3-frontend-template.md (user interface)

Phase 2 Completion Gate

Before proceeding to Phase 3 (Manifests), verify that ALL selected plan documents exist on disk.

CANONICAL NUMBERING REFERENCE. Every filename in the table below matches assets/addendum-numbering.md — the single source of truth for Phase 1 addendum numbers. If you are adding a new addendum, extend addendum-numbering.md first, then use the new name here. Never invent a number (e.g. the stale phase1-addendum-1.1-dashboards.md is forbidden — dashboards are 1.5-aibi-dashboards.md).

If any required document is missing, create it from its template before generating manifests. Manifests reference these files in generated_from.plan_addendums — they must exist on disk. Workshop mode does not waive this gate: artifact counts inside each document are capped, but the document set is unchanged.

Phase 2 Step 5 — Emit Source Dependency Manifest (MANDATORY, layer-aware)

Before Phase 3 manifest generation, extract every source-layer table/column referenced across all plan addendums into a single machine-readable manifest. This becomes the contract validated against the live catalog in the next step.

The manifest filename and shape depend on the Phase 0 selected layer:

Acceleration mode emits ONLY gold-dependency-manifest.yaml (existing behavior). Workshop mode emits whichever file matches its selected layer; when the selected layer is Gold, it emits gold-dependency-manifest.yaml for backward compatibility. Both shapes share the same referenced_by semantics.

# plans/manifests/gold-dependency-manifest.yaml  (acceleration, or workshop with Gold source)
planning_mode: acceleration  # or workshop — mirror the parent manifest's planning_mode
planning_source:
  selected_layer: deployed_gold   # or gold_design
  schema: "<catalog>.<gold_schema>"
implementation_readiness: gold_ready  # or gold_design_only
requires_gold_promotion: false
generated_from:
  plan_addendums:
    - plans/phase1-use-cases.md
    - plans/phase1-addendum-1.2-tvfs.md
    - plans/phase1-addendum-1.3-metric-views.md
    - plans/phase1-addendum-1.5-aibi-dashboards.md
    - plans/phase1-addendum-1.6-genie-spaces.md
gold_dependencies:
  - table: fact_booking_daily
    columns: [booking_key, property_key, booking_date, net_revenue, nights]
    referenced_by:
      - semantic-layer/metric_views/revenue_analytics_metrics.yaml
      - semantic-layer/tvfs/get_revenue_by_property
      - observability/dashboards/revenue_overview.lvdash.json
  - table: dim_property
    columns: [property_key, property_name, destination_id, is_current]
    referenced_by:
      - semantic-layer/metric_views/revenue_analytics_metrics.yaml
summary:
  total_tables: 12
  total_columns: 84
  total_referenced_by: 37

# plans/manifests/source-dependency-manifest.yaml  (workshop only; Bronze/Silver/source CSV)
planning_mode: workshop
planning_source:
  selected_layer: deployed_silver   # or deployed_bronze, source_csv
  schema: "<catalog>.<silver_schema>"   # null for source_csv
  source_csv_paths: []                  # populated for source_csv
implementation_readiness: workshop_deployable   # workshop_draft only when selected_layer=source_csv
requires_gold_promotion: false                  # advisory only; never gates deployment
generated_from:
  plan_addendums:
    - plans/phase1-use-cases.md
source_dependencies:
  - table: silver_bookings
    columns: [booking_id, property_id, booking_date, gross_amount]
    referenced_by:
      - planning/use_case_cards/revenue_overview.md
summary:
  total_tables: 4
  total_columns: 22
  total_referenced_by: 6

Rules:

• One entry per distinct source table; union all column references from all plan addendums.
• referenced_by uses relative artifact paths so downstream fixes can trace artifacts back to the missing column.
• Emit this manifest even when planning_mode: workshop — the workshop cap applies to artifact counts, not to manifest accuracy.
• The shape is identical to Gold's: only the filename, top-level key (source_dependencies vs gold_dependencies), and planning_source block change.

Phase 2 Step 6 — Live-Catalog Intersection (STOP / WARN Rule, MANDATORY, mode-aware)

Immediately after emitting the dependency manifest, query the live catalog and cross-reference every table/column reference. Downstream stages all assume the planning source is consistent with the live catalog — catching gaps HERE saves 5+ deploy cycles later.

The validation behavior depends on planning_mode and planning_source.selected_layer:

import yaml
from pathlib import Path
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
catalog = "<lakehouse_default_catalog>"

# Choose which manifest to load based on Phase 0 selection.
gold_path   = Path("plans/manifests/gold-dependency-manifest.yaml")
source_path = Path("plans/manifests/source-dependency-manifest.yaml")
manifest_path = gold_path if gold_path.exists() else source_path
manifest = yaml.safe_load(manifest_path.read_text())

planning_mode    = manifest.get("planning_mode", "acceleration")
selected_layer   = manifest.get("planning_source", {}).get("selected_layer", "deployed_gold")
target_schema    = manifest.get("planning_source", {}).get("schema")  # e.g. "<catalog>.<schema>"
deps_key         = "gold_dependencies" if "gold_dependencies" in manifest else "source_dependencies"

# Workshop + source_csv: nothing to intersect against, skip.
if selected_layer == "source_csv":
    print("ℹ Skipping live-catalog intersection — selected_layer=source_csv has no live schema.")
else:
    # Pull every table and column from the live target schema in one shot.
    schema_only = target_schema.split(".")[-1]
    live_cols = (
        spark.sql(f"""
          SELECT table_name, column_name, full_data_type
          FROM {catalog}.information_schema.columns
          WHERE table_schema = '{schema_only}'
        """).collect()
    )
    live_index = {}
    for row in live_cols:
        live_index.setdefault(row.table_name, {})[row.column_name] = row.full_data_type

    missing_tables, missing_columns = [], []
    for dep in manifest[deps_key]:
        tbl = dep["table"]
        if tbl not in live_index:
            missing_tables.append({"table": tbl, "referenced_by": dep["referenced_by"]})
            continue
        for col in dep["columns"]:
            if col not in live_index[tbl]:
                missing_columns.append({"table": tbl, "column": col, "referenced_by": dep["referenced_by"]})

    if missing_tables or missing_columns:
        # Pick remediation filename based on dependency kind.
        remediation_path = Path(
            "plans/gold-gap-remediation.md" if deps_key == "gold_dependencies"
            else "plans/source-gap-remediation.md"
        )
        remediation_path.write_text(
            f"# {('Gold' if deps_key == 'gold_dependencies' else 'Source')} Dependency Gap Remediation\n\n"
            f"The following references in plan addendums do not exist in `{target_schema}`.\n\n"
            "## Missing tables\n\n"
            + "\n".join(f"- **{m['table']}** — referenced by {m['referenced_by']}" for m in missing_tables)
            + "\n\n## Missing columns\n\n"
            + "\n".join(
                f"- **{m['table']}.{m['column']}** — referenced by {m['referenced_by']}"
                for m in missing_columns
            )
            + "\n\n## Next steps\n\n"
            "1. Acceleration / Gold source: add tables to `gold_layer_design/yaml/` and re-run "
            "`gold/01-gold-layer-setup`.\n"
            "2. Workshop / Silver or Bronze: extend the corresponding setup skill (silver/bronze) "
            "and re-run.\n"
            "3. Re-run this Planning skill to regenerate manifests.\n"
        )

        is_strict = (planning_mode == "acceleration" and selected_layer == "deployed_gold") \
                 or (planning_mode == "workshop"     and selected_layer == "deployed_gold")
        msg = (f"{deps_key} gap detected: {len(missing_tables)} missing tables, "
               f"{len(missing_columns)} missing columns. See {remediation_path}.")
        if is_strict:
            raise RuntimeError(msg + " STOP — downstream orchestrators cannot proceed.")
        else:
            print(f"⚠ {msg} Continuing under non-strict mode/layer; downstream manifests "
                  f"will carry implementation_readiness=" + manifest.get("implementation_readiness", ""))
    else:
        print(f"✅ {deps_key} intersected cleanly with live catalog `{target_schema}`.")

Escape flag: If the user has an out-of-band reason to bypass the gap (e.g., Gold is intentionally incomplete for a phased rollout), they can pass planning_allow_gold_gap: true in their prompt. In that case, still emit the remediation file as a warning, but proceed to Phase 3 with a prominent gold_gap_acknowledged: true marker in every downstream manifest. This flag does NOT relax mode-specific behavior; non-Gold workshop manifests already carry implementation_readiness: workshop_deployable (Silver/Bronze) or workshop_draft (source CSV).

Phase 3: Manifest Generation (Plan-as-Contract)

After creating plan documents, generate machine-readable YAML manifests that downstream orchestrators consume as implementation contracts.

Why manifests? The "Extract, Don't Generate" principle applies to the planning-to-implementation handoff. Manifests ensure downstream orchestrators implement exactly what was planned — no missed artifacts, no naming inconsistencies.

MANDATORY: Read the manifest generation guide:

Steps:

1. Review Gold layer YAML schemas in gold_layer_design/yaml/
2. For each plan addendum, extract the concrete artifact definitions
3. Generate 4 YAML manifests using templates from assets/templates/manifests/:
   • plans/manifests/semantic-layer-manifest.yaml — TVFs, Metric Views, Genie Spaces
   • plans/manifests/observability-manifest.yaml — Monitors, Dashboards, Alerts
   • plans/manifests/ml-manifest.yaml — Feature Tables, Models, Experiments
   • plans/manifests/genai-agents-manifest.yaml — Agents, Tools, Eval Datasets
4. For each artifact in a manifest, add use_case_refs listing the UC# it implements (from plans/use-case-catalog.md)
5. Validate all table/column references exist in Gold YAML
6. Verify summary counts match actual artifact counts
7. Run python scripts/validate_use_case_coverage.py plans/use-case-catalog.md to verify coverage
8. Commit manifests alongside plan documents

Key principle: Every artifact in a manifest MUST trace back to (a) a Gold layer table and (b) a business question from the plan addendum.

Output Structure:

plans/
├── use-case-catalog.md                    # Consolidated use case definitions
├── manifests/
│   ├── semantic-layer-manifest.yaml       # → consumed by semantic-layer/00-*
│   ├── observability-manifest.yaml        # → consumed by monitoring/00-*
│   ├── ml-manifest.yaml                   # → consumed by ml/00-*
│   └── genai-agents-manifest.yaml         # → consumed by genai-agents/00-*

Downstream consumption: Each downstream orchestrator (stages 6-9) has a Phase 0: Read Plan step that reads its manifest. If the manifest doesn't exist (e.g., user skipped Planning), the orchestrator falls back to self-discovery from Gold tables.

Plan Structure Framework

Standard Project Phases

plans/
├── README.md                              # Index and overview
├── use-case-catalog.md                    # Consolidated use case definitions
├── prerequisites.md                       # Bronze/Silver/Gold summary (optional)
├── phase1-use-cases.md                    # Analytics artifacts (master)
│   ├── phase1-addendum-1.1-ml-models.md
│   ├── phase1-addendum-1.2-tvfs.md
│   ├── phase1-addendum-1.3-metric-views.md
│   ├── phase1-addendum-1.4-lakehouse-monitoring.md
│   ├── phase1-addendum-1.5-aibi-dashboards.md
│   ├── phase1-addendum-1.6-genie-spaces.md
│   └── phase1-addendum-1.7-alerting.md
├── phase2-agent-framework.md              # AI Agents
├── phase3-frontend-app.md                 # User Interface
└── manifests/                             # Machine-readable contracts
    ├── semantic-layer-manifest.yaml       # → semantic-layer/00-*
    ├── observability-manifest.yaml        # → monitoring/00-*
    ├── ml-manifest.yaml                   # → ml/00-*
    └── genai-agents-manifest.yaml         # → genai-agents/00-*

Phase Dependencies

Prerequisites (Bronze → Silver → Gold) → Phase 1 (Use Cases) → Phase 2 (Agents) → Phase 3 (Frontend)
         [COMPLETE]                               ↓
                                           All Addendums

Agent Domain Framework

Core Principle

ALL artifacts across ALL phases MUST be organized by Agent Domain. This ensures:

• Consistent categorization across 100+ artifacts
• Clear ownership by future AI agents
• Easy discoverability for users
• Aligned tooling for each domain

Agent Domain Application

Every artifact (TVF, Metric View, Dashboard, Alert, ML Model, Monitor, Genie Space) must:

1. Be tagged with its Agent Domain
2. Use the domain's planning assets (Gold tables in acceleration; selected-layer tables in workshop)
3. Answer domain-specific questions
4. Be grouped with related domain artifacts in documentation

Example Pattern:

## {Domain}: get_{metric}_by_{dimension}

**Agent Domain:** {Domain}
**Planning Assets:** `fact_{entity}`, `dim_{entity}`   # or `silver_{entity}` / `bronze_{entity}` in workshop drafts
**Business Questions:** "What are the top {metric} by {dimension}?"

See Industry Domain Patterns for domain templates by industry.

Agent Layer Architecture Pattern

Core Principle: Agents Use Genie Spaces as Query Interface

AI Agents DO NOT query data assets directly. Instead, they use Genie Spaces as their natural language query interface. Genie Spaces translate natural language to SQL and route to appropriate tools.

USERS (Natural Language)
    ↓
PHASE 2: AI AGENT LAYER (LangChain/LangGraph)
    ├── Orchestrator Agent (intent classification)
    └── Specialized Agents (1 per domain)
            ↓
PHASE 1.6: GENIE SPACES (NL Query Execution)
    ├── {Domain 1} Intelligence Genie Space
    ├── {Domain 2} Intelligence Genie Space
    └── Unified {Project} Monitor
            ↓
PHASE 1: DATA ASSETS (Agent Tools)
    ├── Metric Views (pre-aggregated - use FIRST)
    ├── TVFs (parameterized queries)
    ├── ML Predictions (ML-powered insights)
    └── Lakehouse Monitors (drift detection)
 

…(truncated)