Lakeflow Connect: SaaS connectors
FIRST: use the parent databricks-core skill for CLI install, authentication, and profile selection.
| Source | Skill |
|---|---|
| A SaaS application, a RabbitMQ queue, or a Kafka topic with the managed connector (Beta) | this skill |
| A Kafka topic whose messages need more than the key and value formats the Kafka pipeline page lists | read_kafka in databricks-pipelines, or databricks-spark-structured-streaming |
| A database (SQL Server, PostgreSQL, MySQL, Oracle, query-based) | databricks-lakeflow-connect |
| An app or device that pushes events | databricks-zerobus-ingest |
| Files on cloud object storage | Auto Loader, databricks-pipelines |
| Files on an SFTP server | SFTP file connector on Auto Loader: file-connectors.md |
| A stream with no managed connector (Kinesis, Pub/Sub, Pulsar) | databricks-spark-structured-streaming |
| A source with no managed connector | a custom connector (Beta) or a community connector |
Find the connector's pages
Connectors and options change faster than this skill: read the connector's docs pages instead of answering from memory. When a page and this skill disagree, the page wins.
- Connector keys: the links on SaaS connectors, streaming connectors (Kafka, RabbitMQ) and file connectors. Say a connector isn't available only when none of them lists it.
- Keys on those pages as of 2026-11 (newer ones may be there): 1password-event-logs, aha, akamai-waf, amplitude, anaplan, anthropic, anysphere-audit-logs, anysphere-organization, asana, atlassian-audit-logs, azure-devops, box-audit-logs, celigo, confluence, cribl, crowdstrike-falcon-event-stream, d365, github, gitlab, glean, gmail, google-ads, google-analytics, google-analytics-aggregated, google-drive, google-search-console, google-workspace, hubspot, jira, kafka, linkedin-ads, marketo, meta-ads, microsoft-365, microsoft-entra-id, microsoft-graph-logs, monday-com, netskope-logs, netsuite, notion, okta, okta-system-logs, onedrive, openai, oracle-eloqua, outlook, pagerduty, paypal, pendo, productboard, quickbooks, rabbitmq, reddit-ads, salesforce, sendgrid, servicenow, sfmc, sharepoint, shopify, smartsheet, square, strac, surveymonkey, tiktok-ads, veeva-vault, verkada, vimeo, wiz-audit-logs, workday-activity-logging, workday-hcm, workday-reports, workiva, xero, zendesk-support, zip, zoho-books, zoho-crm.
- Pages:
https://docs.databricks.com/ingestion/lakeflow-connect/<key>(or<key>-overview), then-source-setup,-connection,-pipeline,-reference,-limits,-troubleshoot(not every connector has each). - Read pages with the web-fetch tool or the databricks-docs skill. If neither is available, give the user the page links and ask what they show; don't fetch another way.
- Read
-referencebefore naming a source table or saying which tables are incremental;-limitsbefore sizing. - Notes to read before the steps: Google, Meta, LinkedIn, Reddit, TikTok Ads → ads.md; SharePoint, Google Drive, OneDrive, SFTP → file-connectors.md; Kafka, RabbitMQ → streaming.md; the rest → connector-notes.md.
Workflow
Ask for every missing input in one message (connector, objects, destination catalog and schema, schedule), list what you'll create, and wait for one "yes".
- Previews toggle. When the page's Beta note mentions the Previews page, a workspace admin turns that preview on first. Beta has no production SLA: say so.
- Source setup. Follow
-source-setupon the SaaS side (app registration, API user, scopes). Connectors with several auth methods have one setup page each. - Connection. Created in Catalog Explorer (the
-connectionpage has the exact path), often with a browser sign-in. Ask the user to create it from the-connectionpage and give you its name; don't guessconnection_typeor option names fordatabricks connections create. Grants: databricks-unity-catalog. - Pipeline. Copy the object shape from the
-pipelinepage examples.source_schemais usuallydefault, the account ID for ads,objectsfor Salesforce; never a placeholder. - Run and schedule. Start one update and check it, then schedule a Jobs
pipeline_task(databricks-pipelines, databricks-jobs). Anything kept: a bundlepipelinesresource with thisingestion_definitionplus a job with apipeline_task(databricks-dabs). Kafka and RabbitMQ run continuously, with no job.
{
"name": "jira_ingest",
"catalog": "main",
"schema": "jira_raw",
"ingestion_definition": {
"connection_name": "<jira-connection>",
"objects": [
{"table": {"source_schema": "default", "source_table": "issues",
"destination_catalog": "main", "destination_schema": "jira_raw"}}
]
}
}
connection_nameis the UC connection's name, not its ID.objects[].tableper table,objects[].schemafor every table of a source schema, orobjects[].reportwithsource_url(Workday Reports).connector_optionscarries connector settings (lookback windows, report levels, URLs); the keys don't follow the connector key (Google Drive isgdrive_options): take them fromdatabricks bundle schema(pipelines.ConnectorOptions).table_configurationcarries per-table settings such asscd_type: SCD_TYPE_2, where the overview's feature table allows it.channel: PREVIEWonly where the docs examples set it: SharePoint (GA), Google Drive, OneDrive, Kafka, RabbitMQ.
Important
- An ingestion pipeline has
ingestion_definition, neverlibraries. - Ask before reusing a destination schema: a source table whose name matches an existing destination table fails the update.
- Two tables with the same name can't share a pipeline, even from different source schemas: set
destination_tableon one, or split. A renamed table, or several destination catalogs or schemas, make the pipeline API-only. - A column selected after the pipeline started isn't backfilled. A full refresh backfills it but clears the table and its state and reloads from the source: SCD type 2 history, rows deleted at the source and data the source no longer keeps are lost. Say so and ask before running one.
- A table deleted at the source keeps its destination table.
Troubleshooting
Read the pipeline's event log (databricks-pipelines), then the connector's -troubleshoot page.
| Symptom | Cause | Fix |
|---|---|---|
DUPLICATE_KEY_VIOLATION |
The key the source reports isn't unique per row (NetSuite's line-level tables report transactionid) |
Set table_configuration.primary_keys to a key that is unique per row, as the connector's -pipeline page names it (NetSuite: transactionid and line), then full refresh that table (ask first) |
| A working connection fails auth | Refresh token revoked or expired | Reauthorize the connection in Catalog Explorer |
| Create fails or the connector isn't found | Previews toggle off, or channel: PREVIEW missing |
Check the page's Beta note and examples |
| Every update re-reads a whole table | No cursor (entity tables, formula fields, report snapshots) | Expected; Salesforce formula fields can be ingested incrementally (Beta) in a new pipeline, API only: Ingest formula fields incrementally |
| Too many tables | Per-pipeline cap on -limits (250 where stated, NetSuite 200, Outlook 1) |
Split the pipeline |
Reference Files
- ads.md: ads entity vs report tables, account-ID schemas, options.
- file-connectors.md: SharePoint, Google Drive, OneDrive, SFTP.
- streaming.md: Kafka and RabbitMQ.
- connector-notes.md: traps for other connectors, A to Z.