Databricks Lakeflow Pipeline Engineering

Use this skill to design Lakeflow Spark Declarative Pipelines: medallion layering, Lakeflow Jobs orchestration and task dependencies, Delta table layout (liquid clustering, deletion vectors, Predictive Optimization), Auto Loader ingestion, schema evolution and `_rescued_data`, materialized views versus streaming tables, and backfill strategy. Reads pipeline source, table metadata, and job definitions only; never executes pipelines and never assumes feature availability without checking the documentation.

VincentChuWaiChow Updated 24 repo stars

File contents

VincentChuWaiChow/vanguard-frontier-agentic commit 5efc752551

Frequently asked questions

npx get-skill add vincentchuwaichow/databricks-lakeflow-pipeline-engineering