Sdp Meta

SDP-META — Metadata-Driven Lakeflow Pipelines

Overview

SDP-META (the framework in this repository) is a metadata-driven layer that builds Databricks Lakeflow Spark Declarative Pipelines (SDP) from configuration instead of per-table pipeline code. You describe each source→bronze→silver flow once as a dataflowspec (an onboarding JSON/YAML record); the framework generates and runs the bronze and silver SDP pipelines. Add a table by adding a metadata entry — not by writing another pipeline.

• Package: databricks-labs-sdp-meta (formerly dlt-meta).
• CLI: databricks labs sdp-meta <command>.
• Canonical imports: from databricks.labs.sdp_meta.dataflow_pipeline import DataflowPipeline (v0.0.10 from src.* still resolves via the bundled compat shim).
• Relationship to plain SDP: SDP-META generates SDP pipelines. Use this skill when the user wants config-driven fan-out across many tables/layers; hand-authored streaming tables / materialized views are plain SDP.

When to use: the user has (or wants) an onboarding file describing many tables, mentions sdp-meta/dlt-meta, or wants to onboard/deploy/scaffold a metadata-driven pipeline. When NOT to use: a one-off pipeline with a couple of hand-written tables.

The mental model (3 steps)

1. Author an onboarding file: a list of dataflowspec records, one per source table, describing bronze (and optionally silver) targets, reader options, data-quality expectations, and transformations.
2. Onboard (sdp-meta onboard): reads the onboarding file and writes the dataflowspec into Delta tables (the bronze/silver dataflowspec tables). Metadata only — no data pipeline runs yet.
3. Deploy (sdp-meta deploy): creates a Lakeflow SDP pipeline whose runner notebook reads the dataflowspec tables and materializes bronze/silver tables.

For infrastructure-as-code, bundle-init scaffolds a Databricks Asset Bundle (DAB) wiring the onboarding job + the SDP pipeline + variables together.

Quick start

# Author conf/onboarding.json (see references/onboarding-spec.md), then:
databricks labs sdp-meta onboard --profile <profile>   # write dataflowspec Delta tables
databricks labs sdp-meta deploy  --profile <profile>   # create + run the SDP pipeline

Prefer a bundle for anything beyond a one-off:

databricks labs sdp-meta bundle-init --quickstart=true --output-dir=.
# edit resources/variables.yml (catalog / schema / sdp_meta_dependency), then:
databricks labs sdp-meta bundle-validate
databricks bundle deploy && databricks bundle run

Use the = syntax on boolean flags (--quickstart=true, --build-and-upload-whl=true) — the Labs CLI string-flag parser otherwise consumes the next token as the value.

Real, working configs live in this repo under examples/ (json/, yml/, and the sdp_meta_pipeline*.ipynb notebooks) and in tests/resources/ (onboarding*.json, silver_transformations*.json, dqe/). Copy from these rather than inventing field names.

How agents drive SDP-META

Two complementary surfaces:

• MCP tools (scaffolding + inspection, no live workspace needed): the framework ships its own stdio MCP server (databricks labs sdp-meta mcp, requires the mcp extra) with 5 tools — sdp_meta_bundle_init, sdp_meta_bundle_add_flow, sdp_meta_bundle_validate, sdp_meta_list_templates, sdp_meta_get_onboarding_template. See references/mcp-tools.md.
• CLI (onboard/deploy/bundle-*): the operational path that touches a live workspace. See references/cli-and-bundles.md.

Recommended flow: scaffold + add flows + validate locally with the MCP tools (or bundle-init), then hand off to CLI deploy for the live run.

Onboarding file essentials

A dataflowspec record keys targets by layer and environment. Environment-suffixed fields (_dev / _staging / _prd) let one spec serve all environments.

{
  "data_flow_id": "100",
  "data_flow_group": "A1",              // groups flows into one pipeline
  "source_format": "cloudFiles",        // cloudFiles | delta | eventhub | kafka | snapshot
  "source_details": { "source_database": "APP", "source_table": "CUSTOMERS",
                       "source_path_dev": "/Volumes/.../customers" },
  "bronze_catalog_dev": "bronze_cl", "bronze_database_dev": "bronze",
  "bronze_table": "customers",
  "bronze_reader_options": { "cloudFiles.format": "json", "cloudFiles.inferColumnTypes": "true" },
  "bronze_data_quality_expectations_json_dev": "/path/bronze_dqe.json",
  "silver_catalog_dev": "silver_cl", "silver_database_dev": "silver",
  "silver_table": "customers",
  "silver_transformation_json_dev": "/path/silver_transformations.json"
}

• Data quality (*_data_quality_expectations_json_*): expect_or_drop (quarantine bad rows), expect_or_fail (halt), expect (warn).
• Silver transformations (silver_transformation_json_*): a list of { target_table, select_exp[], where_clause[] }.
• CDC / SCD Type 2: silver_cdc_apply_changes (and bronze CDC options).

Full field reference: references/onboarding-spec.md.

Reference files

Common issues