How do I install the Azure Databricks skill?

Run npx skillmds@latest add luokai0/azure-databricks in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Azure Databricks skill do?

Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Lakehouse/Lakebase, Lakeflow pipelines, Vector Search/RAG, or model serving, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory). It is listed under DevOps & Infra on SkillMD.

Is Azure Databricks safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Azure Databricks?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Azure Databricks free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Azure Databricks?

luokai0 (@luokai0) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Azure Databricks by luokai0 — Agent Skill

Name: Azure Databricks
Rating: 4.2 (5 reviews)
Author: luokai0

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Location	Description
Troubleshooting	L37-L138	Diagnosing and fixing Databricks errors, job and compute failures, connector/ingestion issues, SQL error codes, and performance/debugging problems across Spark, AI, Lakeflow, and tooling.
Best Practices	L139-L313	End-to-end Databricks best practices for performance, cost, governance, streaming, ML/LLM/RAG, BI, Lakeflow, Vector Search, and operational reliability across Azure Databricks workloads.
Decision Making	L314-L401	Guides for choosing Azure Databricks architectures, compute, runtimes, ML/LLM options, and detailed migration paths (Unity Catalog, Delta, SQL, Connect, MLflow, serverless, Lakebase, and networking).
Architecture & Design Patterns	L402-L444	Architectural blueprints and patterns for Databricks: lakehouse, networking, storage, HA/DR, governance, performance, ML/MLOps, RAG/agents, Lakebase, streaming, and external data access.
Limits & Quotas	limits-quotas.md	Limits, quotas, and constraints for Databricks compute, AI/BI, connectors, Lakeflow, Lakebase, model serving, tokens, data types, and Unity Catalog resources, plus related workarounds.
Security	security.md	Identity, access control, encryption, networking, compliance, and secure integrations for Azure Databricks, Unity Catalog, Lakeflow, Lakebase, apps, and external data sources.
Configuration	configuration.md	Configuring and administering Azure Databricks: accounts, workspaces, security, networking, compute, storage, jobs, ML/serving, Lakehouse/Unity Catalog, Lakeflow, apps, and system-table–based monitoring.
Integrations & Coding Patterns	integrations.md	Patterns and APIs for integrating Databricks with external data systems, tools, and AI/ML frameworks, plus detailed PySpark/SQL function references and Lakehouse Federation/streaming examples.
Deployment	deployment.md	Deploying and operating Databricks apps, agents, models, jobs, and infrastructure using CI/CD, IaC, bundles, serving, Terraform, Git, and region/release planning.

Troubleshooting

Topic	URL
Monitor Genie space activity with audit log queries	https://learn.microsoft.com/en-us/azure/databricks/ai-bi/admin/audit
Troubleshoot Azure Databricks compute startup issues	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Resolve Databricks classic compute termination error codes	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UI	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot Apache Kafka usage on Databricks	https://learn.microsoft.com/en-us/azure/databricks/connect/streaming/kafka/faq
Audit and monitor Delta Sharing activity with Databricks logs	https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/audit-logs
Troubleshoot common Delta Sharing access errors	https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
Troubleshoot common Databricks CLI errors and issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Use Databricks app details for monitoring and troubleshooting	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/view-app-details
Troubleshoot Databricks Connect for Python issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Troubleshoot Databricks Connect for Scala problems	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot common Databricks Terraform provider errors	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Resolve common issues with Databricks VS Code extension	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/faqs
Troubleshoot Databricks VS Code extension errors	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Resolve ARITHMETIC_OVERFLOW errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Handle CAST_INVALID_INPUT errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Diagnose DC_GA4_RAW_DATA_ERROR in GA4 connector	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Diagnose DC_SQLSERVER_ERROR in SQL Server connector	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Understand DELTA_ICEBERG_COMPAT_V1_VIOLATION errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Handle DIVIDE_BY_ZERO errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Handle Azure Databricks named error conditions	https://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Fix EWKB_PARSE_ERROR geometry parsing issues	https://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Fix EWKT_PARSE_ERROR geometry parsing issues	https://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Resolve GEOJSON_PARSE_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Address GROUP_BY_AGGREGATE errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Interpret and resolve H3_INVALID_GRID_DISTANCE_VALUE in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Handle H3_INVALID_RESOLUTION_VALUE errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Resolve H3_NOT_ENABLED errors and tier requirements	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Fix INSUFFICIENT_TABLE_PROPERTY errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Troubleshoot INVALID_ARRAY_INDEX errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Troubleshoot INVALID_ARRAY_INDEX_IN_ELEMENT_AT in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Resolve MISSING_AGGREGATION errors in Databricks queries	https://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Diagnose ROW_COLUMN_ACCESS errors for filters and masks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Interpret Azure Databricks SQLSTATE error codes	https://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Fix TABLE_OR_VIEW_NOT_FOUND errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Resolve UNRESOLVED_ROUTINE function resolution errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Understand UNSUPPORTED_TABLE_OPERATION errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Understand UNSUPPORTED_VIEW_OPERATION errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR for geometry parsing	https://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR for geometry parsing	https://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Troubleshoot Mosaic AI Agent Evaluation issues	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/troubleshooting
Troubleshoot and debug Databricks AI agent deployments	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/debug-agent
Troubleshoot common Azure Databricks Genie issues	https://learn.microsoft.com/en-us/azure/databricks/genie/troubleshooting
Resolve common Databricks Auto Loader questions and issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/faq
Diagnose and fix Databricks Confluence ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot Dynamics 365 data ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Troubleshoot Google Ads connector ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot Google Analytics raw data ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Troubleshoot HubSpot connector ingestion problems	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Troubleshoot Jira Lakeflow ingestion errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Meta Ads Lakeflow ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Diagnose and fix MySQL Lakeflow Connect ingestion errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Resolve common PostgreSQL Lakeflow Connect connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-faq
Troubleshoot PostgreSQL ingestion with Lakeflow Connect	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot Lakeflow Connect query-based connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/query-based-troubleshoot
Troubleshoot Salesforce Lakeflow ingestion problems	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Diagnose and fix Databricks ServiceNow connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Diagnose and fix Lakeflow SharePoint connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Answer common SQL Server Lakeflow Connect connector questions	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-faq
Troubleshoot SQL Server ingestion with Lakeflow Connect	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Troubleshoot TikTok Ads connector in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Troubleshoot Workday HCM connector in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Diagnose and fix Databricks Workday connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Troubleshoot Databricks Zendesk Support connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Handle Zerobus Ingest errors and retries	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Use logging to troubleshoot Databricks init scripts	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Test and validate legacy Databricks JDBC driver connections	https://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc/testing
Test and validate Databricks ODBC driver connections	https://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Configure and troubleshoot Lakeflow Jobs with many tasks	https://learn.microsoft.com/en-us/azure/databricks/jobs/large-jobs
Troubleshoot and repair Azure Databricks job failures	https://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Monitor and troubleshoot Lakeflow Spark pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/observability
Use pipeline query history for debugging and tuning	https://learn.microsoft.com/en-us/azure/databricks/ldp/query-history
Recover Databricks pipelines from checkpoint failures	https://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
User guides, migration, and troubleshooting for AI Runtime	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/guides
Troubleshoot Databricks Feature Store issues and limits	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Debug common issues in Databricks Model Serving endpoints	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose Databricks model serving issues with Genie Code	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-genie-code
Debug Python code in Databricks notebooks	https://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger
Troubleshoot failing Spark jobs and executors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Use Databricks Spark jobs timeline for debugging	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/jobs-timeline
Diagnose long-running Spark stages in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage
Investigate high I/O Spark stages in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-io
Debug slow low-I/O Spark stages in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/slow-spark-stage-low-io
Identify expensive reads in Spark DAG on Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-dag-expensive-read
Diagnose gaps between Spark jobs in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-job-gaps
Diagnose and fix Spark memory issues on Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Troubleshoot Azure Databricks Partner Connect issues	https://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Troubleshoot Databricks Git folder errors	https://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Fetch cursor rows and handle SQLSTATE in Databricks	https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/fetch-stmt
Use GET DIAGNOSTICS for SQL error handling in Databricks	https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/get-diagnostics-stmt
Open cursors and handle errors with OPEN in Databricks	https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/open-stmt
Validate UTF-8 strings and handle INVALID_UTF8_STRING	https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/validate_utf8
Understand Databricks SQL query performance insights	https://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/performance-insights
Use Databricks query history to debug performance	https://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Interpret Databricks query profiles for performance tuning	https://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile
Identify and clean up unused Vector Search endpoints	https://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-unused-endpoints

Best Practices

Topic	URL
Use Databricks default compute policy families effectively	https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply identity best practices and migrate to federation	https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Apply best practices for Azure Databricks serverless workspaces	https://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Migrate Databricks library installs from init scripts	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply best practices for Databricks compute policies	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databricks	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hints	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelines	https://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Model business data with Unity Catalog metric views	https://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/basic-modeling
Apply Azure Databricks platform administration best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI performance with Databricks SQL warehouses	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Prepare and model data for high-performance BI on Databricks	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Configure Databricks SQL warehouses for optimal BI serving	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Apply Azure Databricks compute creation best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement Azure Databricks production job scheduling best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Best practices for Power BI dashboards on Databricks	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Apply Databricks compute configuration best practices	https://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Use flexible node types for reliable Databricks compute	https://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks pools	https://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Apply serverless compute best practices in Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Tune Databricks SQL warehouses for BI workloads	https://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Use system table queries to monitor Databricks SQL warehouses	https://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/monitor/queries
Control large interactive queries with Query Watchdog	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Apply observability best practices for Databricks jobs and pipelines	https://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Write efficient UDFs for Unity Catalog ABAC policies	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/udf-best-practices
Apply Unity Catalog data governance best practices	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Work with legacy Hive metastore database objects	https://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Follow DBFS root storage recommendations in Databricks	https://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Migrate from DBFS mounts to Unity Catalog external locations	https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts
Apply DBFS and Unity Catalog usage best practices	https://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Optimize Delta Sharing to reduce cloud egress costs	https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/manage-egress
Apply Delta Lake best practices on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Optimize Databricks tables with liquid clustering	https://learn.microsoft.com/en-us/azure/databricks/delta/clustering
Tune Azure Databricks data skipping with stats and Z-order	https://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping
Use deletion vectors to accelerate Delta table modifications on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/deletion-vectors
Optimize Delta table file layout on Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/optimize
Handle Delta Lake limitations on Amazon S3	https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Apply selective overwrite patterns in Delta Lake	https://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Control Delta table data file size on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size
Vacuum Delta tables safely and efficiently on Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/vacuum
Optimize VARIANT data performance with shredding on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/variant-shredding
Apply MLOps Stack best practices with bundles	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/mlops-stacks
Apply Databricks-recommended CI/CD workflows and patterns	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/best-practices
View Databricks cluster policy families via CLI	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/policy-families-commands
Apply security and performance best practices for Databricks apps	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Test Databricks Connect for Python code with pytest	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/testing
Handle async queries and interruptions in Databricks Connect	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/queries
Choose between Databricks volumes and workspace files	https://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Customize MLflow 2 AI judges for your agents	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/advanced-agent-eval
Apply best practices for MLflow 2 evaluation sets	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/evaluation-set
Measure RAG performance with Databricks metrics	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-assess-performance
Create evaluation sets for Databricks RAG apps	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-define-quality
Evaluate and monitor RAG apps on Databricks	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Optimize Databricks RAG application quality	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-overview
Improve Databricks RAG chain quality	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain
Configure Genie Code custom instructions effectively	https://learn.microsoft.com/en-us/azure/databricks/genie-code/instructions
Apply practical tips to improve Genie Code responses	https://learn.microsoft.com/en-us/azure/databricks/genie-code/tips
Apply best practices for curating Genie spaces	https://learn.microsoft.com/en-us/azure/databricks/genie/best-practices
Migrate existing Auto Loader streams to file events	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/migrating-to-file-events
Apply common Auto Loader data ingestion patterns	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/patterns
Configure Databricks Auto Loader for production	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Configure Auto Loader with Unity Catalog for secure ingestion	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/unity-catalog
Apply common COPY INTO data loading patterns	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Use the _metadata file column in Databricks	https://learn.microsoft.com/en-us/azure/databricks/ingestion/file-metadata-column
Apply common patterns for Lakeflow managed ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/common-patterns
Fully refresh Lakeflow Connect managed ingestion target tables	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/full-refresh
Query system.billing.usage to monitor Lakeflow ingestion costs	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monitor-costs
Perform ongoing maintenance for Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pipeline-maintenance
Maintain and operate PostgreSQL ingestion pipelines in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
Enable incremental ingestion for Salesforce formula fields	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-formula-fields
Use Databricks init scripts for cluster customization	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files safely in Databricks init scripts	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Configure compute for Lakeflow Jobs with recommended patterns	https://learn.microsoft.com/en-us/azure/databricks/jobs/compute
Build metadata-driven For each jobs with control tables	https://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/foreach-sql-lookup-tutorial
Apply best practices for configuring classic Lakeflow Jobs	https://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Apply cost optimization best practices on Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
Implement best practices for Databricks data and AI governance	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
Design observability and monitoring strategy for Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/observability
Apply interoperability and usability best practices on Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/best-practices
Implement operational excellence best practices on Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/best-practices
Apply performance best practices for Azure Databricks lakehouse	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/best-practices
Apply reliability best practices on Databricks lakehouse	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/best-practices
Optimize Lakeflow pipelines with enhanced autoscaling	https://learn.microsoft.com/en-us/azure/databricks/ldp/auto-scaling
Apply best practices for Lakeflow Spark Declarative Pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices
Use advanced AUTO CDC features and monitor processing metrics	https://learn.microsoft.com/en-us/azure/databricks/ldp/cdc-advanced
Apply development best practices to Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/develop
Manage Python dependencies safely in Databricks pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/developer/external-dependencies
Implement advanced expectation patterns at scale	https://learn.microsoft.com/en-us/azure/databricks/ldp/expectation-patterns
Reduce Lakeflow pipeline initialization latency	https://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Backfill historical data with Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/flows-backfill
Run full refresh operations for Databricks streaming tables safely	https://learn.microsoft.com/en-us/azure/databricks/ldp/full-refresh-st
Optimize stateful stream processing with watermarks	https://learn.microsoft.com/en-us/azure/databricks/ldp/stateful-processing
Design CDC and snapshot patterns in Databricks	https://learn.microsoft.com/en-us/azure/databricks/ldp/what-is-change-data-capture
Restart the Python process to refresh Databricks libraries	https://learn.microsoft.com/en-us/azure/databricks/libraries/restart-python-process
Apply data loading best practices on Databricks AI Runtime	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/dataloading
Track experiments and monitor GPU health on AI Runtime	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/tracking-observability
Apply Hyperopt best practices and troubleshooting on Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/hyperopt-best-practices
Implement point-in-time correct feature joins for time series	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series
Benchmark Databricks LLM endpoints for latency and throughput	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/prov-throughput-run-benchmark
Load and prepare data for ML on Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/load-data/
Implement LLMOps workflows on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/llmops
Configure Locust-based load tests for Databricks serving	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/configure-load-test
Validate models before Databricks Model Serving deployment	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-pre-deployment-validation
Optimize Databricks Model Serving endpoints for production	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/production-optimization
Plan and execute load testing for Databricks endpoints	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/what-is-load-test
Tune and scale Ray clusters on Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/scale-ray
Implement distributed image inference on Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/reference-solutions/images-etl-inference
Follow deep learning best practices on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
Fine-tune Hugging Face models on a single GPU in Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/fine-tune-model
Prepare datasets for Hugging Face fine-tuning on Databricks	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/load-data
Adapt Apache Spark workloads for Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/migration/spark
Align MLflow LLM judges with human evaluators	https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/align-judges
Optimize prompts using MLflow GEPA optimizer	https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/automatically-optimize-prompts
Evaluate and compare MLflow prompt versions effectively	https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-prompts
Use manual MLflow tracing for production GenAI apps	https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/app-instrumentation/manual-tracing/
Analyze GenAI traces for errors and performance	https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/observe-with-traces/analyze-traces
Apply software engineering practices to Databricks notebooks	https://learn.microsoft.com/en-us/azure/databricks/notebooks/best-practices
Apply Genie Code effectively in Databricks notebooks	https://learn.microsoft.com/en-us/azure/databricks/notebooks/code-assistant
Run Databricks notebooks safely and efficiently	https://learn.microsoft.com/en-us/azure/databricks/notebooks/run-notebook
Test and schedule Databricks notebook code	https://learn.microsoft.com/en-us/azure/databricks/notebooks/test-notebooks
Configure and optimize Lakebase Postgres computes	https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/manage-computes
Create and manage Lakebase read replicas	https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/manage-read-replicas
Monitor Lakebase queries using pg_stat_statements	https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/pg-stat-statements
Apply performance optimization recommendations on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/
Use adaptive query execution on Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/aqe
Leverage cost-based optimizer in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/optimizations/cbo
Improve read performance with Databricks disk cache	https://learn.microsoft.com/en-us/azure/databricks/optimizations/disk-cache
Improve Delta query performance with dynamic file pruning on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/dynamic-file-pruning
Accelerate data access with predictive I/O	https://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-io
Use predictive optimization for Unity Catalog tables	https://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-optimization
Tune Azure Databricks range join performance	https://learn.microsoft.com/en-us/azure/databricks/optimizations/range-join
Diagnose Databricks Spark cost and performance in UI	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/
Debug skew and spill in Databricks Spark stages	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-page
Handle Databricks spot instance losses effectively	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/losing-spot-instances
Resolve long Spark stages with a single task	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/one-spark-task
Optimize many small Spark jobs on Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/small-spark-jobs
Mitigate overloaded Spark driver on Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-driver-overloaded
Detect unnecessary data rewriting in Databricks Spark writes	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-rewriting-data
Best practices for setting up Databricks Partner Connect	https://learn.microsoft.com/en-us/azure/databricks/partner-connect/best-practice
Configure networking for Databricks Lakehouse Federation data sources	https://learn.microsoft.com/en-us/azure/databricks/query-federation/networking
Optimize performance of Databricks Lakehouse Federation queries	https://learn.microsoft.com/en-us/azure/databricks/query-federation/performance-recommendations
Encrypt inter-node traffic in Databricks clusters	https://learn.microsoft.com/en-us/azure/databricks/security/keys/encrypt-otw
Transform complex and nested data types in Databricks	https://learn.microsoft.com/en-us/azure/databricks/semi-structured/complex-types
Use higher-order functions on arrays in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/semi-structured/higher-order-functions
Query semi-structured data using VARIANT in Databr

…(truncated)

Azure Databricks

File contents

Azure Databricks Skill

How to Use This Skill

Category Index

Troubleshooting

Best Practices

Frequently asked questions

Azure Databricks

File contents

Azure Databricks Skill

How to Use This Skill

Category Index

Troubleshooting

Best Practices

Related skills

Frequently asked questions

How do I install the Azure Databricks skill?

What does the Azure Databricks skill do?

Is Azure Databricks safe to use?

Which AI agents work with Azure Databricks?

Is Azure Databricks free to use?

Who published Azure Databricks?