How do I install the Azure Databricks skill?

Run npx skillmds@latest add microsoftdocs/azure-databricks in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Azure Databricks skill do?

Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Lakeflow pipelines, Genie/AI Runtime, Delta Lake/streaming, or model serving, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory). It is listed under DevOps & Infra on SkillMD.

Is Azure Databricks safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: makes network calls, reads secrets. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Azure Databricks?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Azure Databricks free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Azure Databricks?

microsoftdocs (@microsoftdocs) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Azure Databricks by microsoftdocs — Agent Skill

Name: Azure Databricks
Rating: 4.5 (4 reviews)
Author: microsoftdocs

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Location	Description
Troubleshooting	L37-L171	Diagnosing and fixing Azure Databricks issues: logs, compute and Spark failures, SQL/runtime errors, connectors/Lakeflow ingestion, model serving, Feature Store, and performance troubleshooting.
Best Practices	L172-L375	Best-practice guidance for Databricks: cost/governance, compute and SQL tuning, Delta/streaming, Lakeflow ingestion, RAG/AI Search/GenAI, ML & serving, notebooks, networking, and admin operations.
Decision Making	L376-L488	Guides for choosing Azure Databricks tiers, compute, serverless, Unity Catalog, Lakeflow, ML/AI options, and detailed migration/upgrade paths between runtimes, tools, and data architectures.
Architecture & Design Patterns	architecture-patterns.md	Architectural and design patterns for Databricks: agent systems and memory, Lakehouse/medallion, Lakeflow pipelines, networking, security, storage, CDC, Lakebase, ML/MLOps, and dashboard/data modeling.
Limits & Quotas	limits-quotas.md	Limits, quotas, and constraints for Azure Databricks features: compute/serverless, AI/Genie, connectors/Lakeflow, SQL/Unity Catalog, real-time mode, jobs, dashboards, and resource usage.
Security	security.md	Identity, access control, encryption, networking, compliance, secrets, and secure integrations for Azure Databricks, Unity Catalog, Lakeflow, Genie, OpenSharing, and AI/ML services.
Configuration	configuration.md	Configuring Azure Databricks and Unity Catalog: account/workspace settings, compute, networking, security, AI/ML/GenAI services, Lakeflow pipelines, connectors, SQL behavior, and system-table observability.
Integrations & Coding Patterns	integrations.md	Patterns and APIs for integrating Databricks with external systems and tools: agents, AI Search, Lakeflow, AI Runtime, JDBC/ODBC, BI/ETL partners, Lakehouse Federation, SQL/PySpark data sources, and SDK/CLI automation.
Deployment	deployment.md	Deploying and managing Azure Databricks workspaces, apps, agents, dashboards, Lakeflow/Lakebase projects, CI/CD pipelines, and model serving endpoints across tools like ARM, CLI, Terraform, GitHub, and Azure DevOps.

Troubleshooting

Topic	URL
Interpret Azure Databricks diagnostic and audit logs	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/audit-logs
Troubleshoot Databricks MLflow Agent Evaluation issues	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-evaluation/troubleshooting
Debug custom code agents on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/debug-agent
Diagnose and resolve Azure Databricks compute startup issues	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Diagnose Azure Databricks classic compute termination errors	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UI	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot file events for Unity Catalog external locations	https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/file-events-faq
Troubleshoot common Databricks CLI issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Diagnose and fix Databricks Connect Python issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Diagnose and fix Databricks Connect Scala issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot common Databricks Terraform provider errors	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Troubleshoot common Databricks IDE extension errors	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Resolve ARITHMETIC_OVERFLOW errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Handle CAST_INVALID_INPUT errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Diagnose DC_GA4_RAW_DATA_ERROR in GA4 connector	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Diagnose DC_SQLSERVER_ERROR in SQL Server connector	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Understand DELTA_ICEBERG_COMPAT_V1_VIOLATION errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Resolve DIVIDE_BY_ZERO error in Azure Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Interpret Azure Databricks error condition strings	https://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Fix EWKB_PARSE_ERROR geometry parsing issues	https://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Fix EWKT_PARSE_ERROR geometry parsing issues	https://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Resolve GEOJSON_PARSE_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Address GROUP_BY_AGGREGATE errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Interpret and resolve H3_INVALID_GRID_DISTANCE_VALUE in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Handle H3_INVALID_RESOLUTION_VALUE errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Resolve H3_NOT_ENABLED errors and tier requirements	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Fix INSUFFICIENT_TABLE_PROPERTY errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Troubleshoot INVALID_ARRAY_INDEX errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Troubleshoot INVALID_ARRAY_INDEX_IN_ELEMENT_AT in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Resolve MISSING_AGGREGATION errors in Databricks queries	https://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Diagnose ROW_COLUMN_ACCESS errors for filters and masks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Map Azure Databricks errors to SQLSTATE codes	https://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Fix TABLE_OR_VIEW_NOT_FOUND errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Resolve UNRESOLVED_ROUTINE function resolution errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Understand UNSUPPORTED_TABLE_OPERATION errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Understand UNSUPPORTED_VIEW_OPERATION errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR for geometry parsing	https://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR for geometry parsing	https://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Diagnose and fix common Genie Agent issues	https://learn.microsoft.com/en-us/azure/databricks/genie-agents/troubleshooting
Monitor and troubleshoot Auto Loader ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/observability
Troubleshoot Databricks Aha! ingestion connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/aha-troubleshoot
Troubleshoot common Amplitude connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/amplitude-troubleshoot
Troubleshoot Databricks Anthropic connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/anthropic-troubleshoot
Resolve common Confluence connector ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-faq
Resolve common Databricks Confluence connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot Dynamics 365 ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Resolve common issues with Lakeflow managed connectors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/faq
Diagnose and fix Glean connector authentication and rate errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/glean-troubleshoot
Troubleshoot Databricks Gmail ingestion connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/gmail-troubleshoot
Troubleshoot Google Ads Lakeflow connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot Google Analytics Raw Data ingestion	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Troubleshoot Databricks Google Drive connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-drive-faq
Troubleshoot Databricks Google Drive ingestion failures	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-drive-troubleshoot
Troubleshoot Databricks Google Search Console connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-search-console-troubleshoot
Troubleshoot HubSpot Lakeflow connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Diagnose and fix Jira Lakeflow Connect issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Databricks Kafka connector ingestion errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/kafka-troubleshoot
Troubleshoot common LinkedIn Ads connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/linkedin-ads-troubleshoot
Troubleshoot Databricks Marketo connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/marketo-troubleshoot
Troubleshoot Meta Ads ingestion connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Troubleshoot Monday.com Lakeflow Connect ingestion errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monday-com-troubleshoot
Troubleshoot MySQL ingestion issues in Databricks	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Troubleshoot Netskope Logs Lakeflow Connect errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/netskope-logs-troubleshoot
Troubleshoot Databricks Notion connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/notion-troubleshoot
Resolve common issues with the Databricks OpenAI connector	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/openai-faq
Troubleshoot Databricks Lakeflow OpenAI connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/openai-troubleshoot
Troubleshoot Oracle integrated CDC ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/oracle-troubleshoot
Troubleshoot Databricks Outlook ingestion connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-troubleshoot
Troubleshoot Databricks PagerDuty ingestion connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pagerduty-troubleshoot
Troubleshoot Databricks Pendo connector ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pendo-troubleshoot
Troubleshoot PostgreSQL ingestion issues in Databricks	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot Databricks query-based connectors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/query-based-troubleshoot
Troubleshoot Databricks RabbitMQ connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/rabbitmq-troubleshoot
Resolve common Reddit Ads connector errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/reddit-ads-troubleshoot
Troubleshoot Databricks Salesforce ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Troubleshoot SendGrid connector errors and rate limits	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sendgrid-troubleshoot
Diagnose and fix Databricks ServiceNow connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Troubleshoot Salesforce Marketing Cloud connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sfmc-troubleshoot
Resolve common issues with the SharePoint connector	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-faq
Troubleshoot Microsoft SharePoint connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Troubleshoot Databricks Smartsheet connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/smartsheet-troubleshoot
Answer common SQL Server Lakeflow Connect connector questions	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-faq
Troubleshoot SQL Server ingestion in Lakeflow Connect	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Use SQL Server utility script for Databricks ingestion	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-utility-reference
Troubleshoot Square connector authentication and rate limit errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/square-troubleshoot
Troubleshoot common Strac connector errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/strac-troubleshoot
Troubleshoot TikTok Ads connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Troubleshoot Databricks Lakeflow managed ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/troubleshoot
Diagnose and fix UNITY_CATALOG_INITIALIZATION_FAILED in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/uc-initialization-troubleshoot
Troubleshoot Veeva Vault connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/veeva-vault-troubleshoot
Troubleshoot Wiz Audit Logs connector problems	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/wiz-audit-logs-troubleshoot
Troubleshoot Workday HCM connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Troubleshoot Workday Reports ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Resolve common issues with the Workiva connector	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workiva-faq
Troubleshoot Databricks Workiva connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workiva-troubleshoot
Troubleshoot Azure Databricks Zendesk Support connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Troubleshoot Databricks Zip connector authentication and rate limits	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zip-troubleshoot
Troubleshoot Zoho Books connector authentication and rate limits	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zoho-books-troubleshoot
Interpret Zerobus Ingest error codes and handling	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Inspect logs for Databricks init script execution	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Test and validate Databricks ODBC driver connections	https://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Diagnose and improve Azure Databricks Lakeflow job performance	https://learn.microsoft.com/en-us/azure/databricks/jobs/diagnose-job-performance
Troubleshoot and repair Azure Databricks job failures	https://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Monitor and troubleshoot Databricks materialized views	https://learn.microsoft.com/en-us/azure/databricks/ldp/dbsql/materialized-monitor
Recover Lakeflow pipelines from checkpoint failures	https://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
Troubleshoot AI Runtime GPU workloads with Genie Code	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/genie-code
Inspect and debug Databricks Feature Views in Unity Catalog	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/explore-feature-views
Troubleshoot Databricks Feature Store errors and limits	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Debug Azure Databricks model serving endpoints	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose Databricks model serving with Genie Code	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-genie-code
Use Insights to detect and resolve Lakebase issues	https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/observability-ai-insights
Diagnose and fix Lakebase Postgres issues with Genie	https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/observability-genie
Resolve common OpenSharing data access errors	https://learn.microsoft.com/en-us/azure/databricks/opensharing/troubleshooting
Troubleshoot failing Spark jobs and executors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Diagnose long-running Spark stages in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage
Diagnose skew and spill using Databricks Spark UI	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-page
Troubleshoot slow low-I/O Spark stages in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/slow-spark-stage-low-io
Identify expensive reads in Spark DAG on Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-dag-expensive-read
Diagnose gaps between Spark jobs in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-job-gaps
Diagnose and fix Spark memory issues on Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Diagnose and fix Azure Databricks Partner Connect issues	https://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Troubleshoot Azure Databricks publishing to Power BI	https://learn.microsoft.com/en-us/azure/databricks/partners/bi/power-bi/troubleshooting
Retrieve exceptions from terminated StreamingQuery	https://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/classes/streamingquery/exception
Debug streaming queries with explain plans	https://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/classes/streamingquery/explain
Troubleshoot Databricks Git folder sync errors	https://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Detect and repair Delta table metadata and file issues	https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-fsck
Use Databricks SQL query history for troubleshooting	https://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Use Databricks SQL query profiles to diagnose performance	https://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile
Inspect and debug Structured Streaming state on Databricks	https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/read-state

Best Practices

Topic	URL
Tag Databricks resources for cost attribution	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags
Use default Databricks policy families to enforce compute best practices	https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Use managed disaster recovery for Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/admin/managed-disaster-recovery
Apply identity federation best practices in Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Apply best practices for serverless Databricks workspaces	https://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Build and refine Knowledge Assistant chatbots on Databricks	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-bricks/knowledge-assistant
Design effective evaluation sets for Databricks agents	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-evaluation/evaluation-set
Load test Databricks Apps agents for QPS limits	https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/load-test-agent-app
Measure RAG performance with retrieval and response metrics	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-assess-performance
Define RAG application quality with evaluation sets	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-define-quality
Set up Databricks infrastructure to measure RAG quality	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-enable-measurement
Evaluate and monitor RAG applications for quality, cost, latency	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Design and optimize RAG inference chains on Databricks	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/fundamentals-inference-chain-rag
Build and tune Databricks RAG data pipelines	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/quality-data-pipeline-rag
Improve RAG application quality via key tuning knobs	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/quality-overview
Optimize RAG chain components for better responses	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/quality-rag-chain
Optimize Databricks AI Search performance	https://learn.microsoft.com/en-us/azure/databricks/ai-search/best-practices
Load test Databricks AI Search endpoints for sizing	https://learn.microsoft.com/en-us/azure/databricks/ai-search/endpoint-load-test
Apply Databricks AI Search filter expressions effectively	https://learn.microsoft.com/en-us/azure/databricks/ai-search/filtering-guide
Improve Databricks AI Search retrieval quality	https://learn.microsoft.com/en-us/azure/databricks/ai-search/retrieval-quality
Evaluate Databricks AI Search retrieval strategies	https://learn.microsoft.com/en-us/azure/databricks/ai-search/retrieval-quality-eval
Detect and clean up unused Databricks AI Search endpoints	https://learn.microsoft.com/en-us/azure/databricks/ai-search/unused-endpoints
Migrate Databricks library installs from init scripts	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply compute policy best practices in Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databricks	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hints	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelines	https://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Apply Azure Databricks administration best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI performance with Databricks SQL warehouses	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Optimize BI performance with Databricks data preparation	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Configure Databricks SQL warehouses for optimal BI serving	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Apply Azure Databricks compute creation best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement Azure Databricks production job scheduling best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Apply Power BI performance best practices with Databricks data	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Apply classic compute configuration best practices in Databricks	https://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Use flexible node types for reliable Databricks compute	https://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks pools	https://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Use serverless compute effectively on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Tune Databricks SQL warehouses for BI workloads	https://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Control large interactive queries with Query Watchdog	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Optimize Azure Databricks dashboard caching and datasets	https://learn.microsoft.com/en-us/azure/databricks/dashboards/caching
Apply observability best practices for Databricks jobs and pipelines	https://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Apply best practices for Unity Catalog ABAC policies	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/best-practices
Use common ABAC patterns for filtering and masking	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/common-patterns
Optimize performance of ABAC row filter and mask policies	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/performance
Apply Unity Catalog governance best practices	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Configure anomaly detection for data quality	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/
Manage Unity Catalog object storage lifecycle and recovery	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/object-storage-lifecycle
Work with legacy Hive metastore objects in Databricks	https://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Follow DBFS root storage recommendations in Databricks	https://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Apply DBFS and Unity Catalog usage best practices	https://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Apply Delta Lake best practices on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Handle Delta Lake limitations and risks on Amazon S3	https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Selectively overwrite data in Delta Lake tables	https://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Apply MLOps Stack best practices with bundles	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/mlops-stacks
Apply CI/CD workflow best practices on Databricks	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/flows
Apply security and performance best practices for Databricks apps	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Implement horizontal scaling for Azure Databricks apps	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/horizontal-scaling
Test Databricks Connect for Python code with pytest	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/testing
Handle async queries and interruptions in Databricks Connect	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/queries
Apply performance best practices for Databricks REST API	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/rest-api
Apply development and CI/CD best practices on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/developers/best-practices
Explore Unity Catalog volumes and storage files in Databricks	https://learn.microsoft.com/en-us/azure/databricks/discover/files
Choose between Databricks volumes and workspace files	https://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Manage and use workspace files in Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/files/workspace
Curate effective Genie Agents for accuracy	https://learn.microsoft.com/en-us/azure/databricks/genie-agents/best-practices
Apply Auto Loader best practices for reliable ingestion	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/best-practices
Optimize Databricks Auto Loader directory listing mode	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/directory-listing-mode
Configure Databricks Auto Loader for production ingestion	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Configure Auto Loader automatic type widening	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/type-widening
Apply common COPY INTO data loading patterns	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Incrementally clone Parquet and Iceberg tables to Delta	https://learn.microsoft.com/en-us/azure/databricks/ingestion/data-migration/clone-parquet
Use the Anthropic connector effectively in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/anthropic-faq
Monitor Lakeflow ingestion gateways with event logs	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/gateway-event-logs
Apply Marketo connector FAQs and usage guidance	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/marketo-faq
Query system.billing.usage to monitor Lakeflow costs	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monitor-costs
Apply Notion connector FAQs and usage tips	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/notion-faq
Apply Outlook connector FAQs and guidance	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-faq
Maintain Databricks Lakeflow managed ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pipeline-maintenance
Maintain and operate PostgreSQL Lakeflow ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
Filter rows during Lakeflow Connect ingestion	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/row-filtering
Optimize Databricks smart closure for CDC pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/smart-closure
Design asynchronous Zerobus stream communication	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-async-communication
Use Zerobus acknowledgment callbacks effectively	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-callbacks
Choose Zerobus ingestion methods and blocking	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-message-blocking
Implement Zerobus recovery and retry patterns	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-recovery
Capture schema mismatches with Zerobus rescue column	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-rescue-column
Design Zerobus schemas for evolving data	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-schema-management
Use and configure init scripts on Azure Databricks clusters	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files safely in Databricks init scripts	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Implement recurring Lakeflow Jobs with backfill parameters	https://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/create-recurring-job
Incremental table copy with Lakeflow watermark control	https://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/foreach-watermark-tutorial
Apply classic compute best practices for Databricks jobs	https://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Design compute and workspace configuration for Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/compute
Design observability and monitoring for Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/observability
Implement API-based ingestion patterns in Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/api-ingestion
Optimize Lakeflow clusters with enhanced and vertical autoscaling	https://learn.microsoft.com/en-us/azure/databricks/ldp/auto-scaling
Apply Lakeflow pipeline design best practices	https://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices/
Organize datasets across Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices/organize-datasets
Ensure processing guarantees in Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices/processing-guarantees
Validate production readiness of Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices/production-readiness
Design Lakeflow flows for incremental data processing	https://learn.microsoft.com/en-us/azure/databricks/ldp/concepts/flows
Handle compatibility limits for Lakeflow environments	https://learn.microsoft.com/en-us/azure/databricks/ldp/developer/environment-version-compatibility
Apply data quality expectations in pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/developer/ldp-python-ref-expectations
Apply advanced expectation patterns in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ldp/expectation-patterns
Reduce high initialization times in Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Use from_json for schema inference in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ldp/from-json-schema-evolution
Run full refreshes on Lakeflow streaming tables safely	https://learn.microsoft.com/en-us/azure/databricks/ldp/full-refresh-st
Optimize sta

…(truncated)

Azure Databricks

File contents

Azure Databricks Skill

How to Use This Skill

Category Index

Troubleshooting

Best Practices

Frequently asked questions

Azure Databricks

File contents

Azure Databricks Skill

How to Use This Skill

Category Index

Troubleshooting

Best Practices

Related skills

Frequently asked questions

How do I install the Azure Databricks skill?

What does the Azure Databricks skill do?

Is Azure Databricks safe to use?

Which AI agents work with Azure Databricks?

Is Azure Databricks free to use?

Who published Azure Databricks?