How do I install the Databricks Core Workflow A skill?

Run npx skillmds@latest add micsapp/databricks-core-workflow-a in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Databricks Core Workflow A safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Databricks Core Workflow A?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Databricks Core Workflow A free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under MIT.

Who published Databricks Core Workflow A?

micsapp (@micsapp) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Databricks Core Workflow A by micsapp

Name: Databricks Core Workflow A
Rating: 4.666666666666667 (3 reviews)
Author: micsapp

Databricks Core Workflow A

Execute Databricks primary workflow: Delta Lake ETL pipelines. Use when building data ingestion pipelines, implementing medallion architecture, or creating Delta Lake transformations. Trigger with phrases like "databricks ETL", "delta lake pipeline", "medallion architecture", "databricks data pipeline", "bronze silver gold".

micsapp Updated Sep 8, 2026 3 repo stars

File contents

Databricks Core Workflow A: Delta Lake ETL

Overview
Prerequisites
Instructions
Output
Error Handling
Examples
Resources

Overview

Build production Delta Lake ETL pipelines using medallion architecture (Bronze -> Silver -> Gold).

Prerequisites

Completed databricks-install-auth setup
Understanding of Delta Lake concepts
Unity Catalog configured (recommended)

Medallion Architecture

Raw Sources -> Bronze (Raw/Landing) -> Silver (Cleaned/Business Logic) -> Gold (Aggregated/Analytics Ready)

Instructions

Step 1: Bronze Layer - Raw Ingestion

Ingest raw data with metadata columns (_ingested_at, _source_file). Use mergeSchema for schema evolution. Use Auto Loader (cloudFiles) for streaming ingestion with schema inference.

Step 2: Silver Layer - Data Cleansing

Read from Bronze using Change Data Feed. Apply transformations: trim/lowercase strings, parse timestamps, hash PII, filter nulls, generate surrogate keys. Merge into Silver with upsert pattern.

Step 3: Gold Layer - Business Aggregations

Aggregate Silver data by business dimensions and time grain. Use partition-level overwrites for efficient updates.

Step 4: Delta Live Tables (Optional)

Declarative pipeline with @dlt.table decorators and data quality expectations (@dlt.expect_or_drop).

See detailed implementation for complete Bronze/Silver/Gold pipeline code, Auto Loader config, DLT pipeline, and orchestration example.

Output

Bronze layer with raw data and ingestion metadata
Silver layer with cleansed, deduplicated data
Gold layer with business aggregations
Delta Lake tables with ACID transactions

Error Handling

Error	Cause	Solution
Schema mismatch	Source schema changed	Use `mergeSchema` option
Duplicate records	Missing deduplication	Add merge logic with primary keys
Null values	Data quality issues	Add expectations/filters in Silver
Memory errors	Large aggregations	Increase cluster size or partition data

Examples

Quick Pipeline Run

# Full medallion pipeline
bronze.ingest_to_bronze(spark, "/mnt/landing/orders/", "catalog.bronze.orders")
silver.transform_to_silver(spark, "catalog.bronze.orders", "catalog.silver.orders", primary_keys=["order_id"])
gold.aggregate_to_gold(spark, "catalog.silver.orders", "catalog.gold.metrics", group_by_columns=["region"])

Resources

Next Steps

For ML workflows, see databricks-core-workflow-b.

---name: databricks-core-workflow-adescription: Execute Databricks primary workflow: Delta Lake ETL pipelines. Use when building data ingestion pipelines, implementing medallion architecture, or creating Delta Lake transformations. Trigger with phrases like "databricks ETL", "delta lake pipeline", "medallion architecture", "databricks data pipeline", "bronze silver gold".license: MIT---# Databricks Core Workflow A: Delta Lake ETL## Contents- [Overview](#overview)- [Prerequisites](#prerequisites)- [Instructions](#instructions)- [Output](#output)- [Error Handling](#error-handling)- [Examples](#examples)- [Resources](#resources)## OverviewBuild production Delta Lake ETL pipelines using medallion architecture (Bronze -> Silver -> Gold).## Prerequisites- Completed `databricks-install-auth` setup- Understanding of Delta Lake concepts- Unity Catalog configured (recommended)## Medallion Architecture```Raw Sources -> Bronze (Raw/Landing) -> Silver (Cleaned/Business Logic) -> Gold (Aggregated/Analytics Ready)```## Instructions### Step 1: Bronze Layer - Raw IngestionIngest raw data with metadata columns (`_ingested_at`, `_source_file`). Use `mergeSchema` for schema evolution. Use Auto Loader (`cloudFiles`) for streaming ingestion with schema inference.### Step 2: Silver Layer - Data CleansingRead from Bronze using Change Data Feed. Apply transformations: trim/lowercase strings, parse timestamps, hash PII, filter nulls, generate surrogate keys. Merge into Silver with upsert pattern.### Step 3: Gold Layer - Business AggregationsAggregate Silver data by business dimensions and time grain. Use partition-level overwrites for efficient updates.### Step 4: Delta Live Tables (Optional)Declarative pipeline with `@dlt.table` decorators and data quality expectations (`@dlt.expect_or_drop`).See [detailed implementation](${CLAUDE_SKILL_DIR}/references/implementation.md) for complete Bronze/Silver/Gold pipeline code, Auto Loader config, DLT pipeline, and orchestration example.## Output- Bronze layer with raw data and ingestion metadata- Silver layer with cleansed, deduplicated data- Gold layer with business aggregations- Delta Lake tables with ACID transactions## Error Handling| Error | Cause | Solution ||-------|-------|----------|| Schema mismatch | Source schema changed | Use `mergeSchema` option || Duplicate records | Missing deduplication | Add merge logic with primary keys || Null values | Data quality issues | Add expectations/filters in Silver || Memory errors | Large aggregations | Increase cluster size or partition data |## Examples### Quick Pipeline Run```python# Full medallion pipelinebronze.ingest_to_bronze(spark, "/mnt/landing/orders/", "catalog.bronze.orders")silver.transform_to_silver(spark, "catalog.bronze.orders", "catalog.silver.orders", primary_keys=["order_id"])gold.aggregate_to_gold(spark, "catalog.silver.orders", "catalog.gold.metrics", group_by_columns=["region"])```## Resources- [Delta Lake Guide](https://docs.databricks.com/delta/index.html)- [Medallion Architecture](https://www.databricks.com/glossary/medallion-architecture)- [Delta Live Tables](https://docs.databricks.com/delta-live-tables/index.html)- [Auto Loader](https://docs.databricks.com/ingestion/auto-loader/index.html)## Next StepsFor ML workflows, see `databricks-core-workflow-b`.

Frequently asked questions