Microsoft Fabric Integration with Azure Data Factory (2025)
Overview
Microsoft Fabric is a unified SaaS analytics platform combining Power BI, Azure Synapse Analytics, and Azure Data Factory capabilities. ADF provides native connectors for Fabric Lakehouse and Fabric Warehouse, enabling seamless data movement between ADF and Fabric workspaces.
Microsoft Fabric Lakehouse Connector
The Fabric Lakehouse connector enables read and write operations to Microsoft Fabric Lakehouse for tables and files.
Supported Activities
| Activity |
Supported |
| Copy Activity (source and sink) |
Yes |
| Lookup Activity |
Yes |
| Get Metadata Activity |
Yes |
| Delete Activity |
Yes |
Quick Reference
- Linked service type:
Lakehouse
- Authentication: Managed Identity (preferred) or Service Principal
- Required IDs:
workspaceId and artifactId
- Sink types:
LakehouseTableSink (tables), LakehouseFileSink (files)
- Source type:
LakehouseTableSource
- Table action options:
append or overwrite
Finding Workspace and Artifact IDs:
- Navigate to Fabric workspace in browser
- Copy workspace ID from URL:
https://app.powerbi.com/groups/<workspaceId>/...
- Open Lakehouse settings to find artifact ID
- Or use Fabric REST API to enumerate workspace items
For complete linked service, dataset, and copy activity JSON examples, see references/lakehouse-examples.md.
Microsoft Fabric Warehouse Connector
The Fabric Warehouse connector provides T-SQL based data warehousing capabilities within the Fabric ecosystem.
Supported Activities
| Activity |
Supported |
| Copy Activity (source and sink) |
Yes |
| Lookup Activity |
Yes |
| Get Metadata Activity |
Yes |
| Script Activity |
Yes |
| Stored Procedure Activity |
Yes |
Quick Reference
- Linked service type:
Warehouse
- Authentication: System-Assigned Managed Identity (recommended), User-Assigned MI, or Service Principal
- Required properties:
endpoint, warehouse
- Sink type:
WarehouseSink
- Write behaviors:
insert or upsert
- Table option:
autoCreate (creates table if missing)
For complete linked service, copy activity, stored procedure, and script activity JSON examples, see references/warehouse-examples.md.
OneLake Integration Patterns
ADF supports three integration patterns with OneLake:
| Pattern |
Description |
Key Benefit |
| ADLS Gen2 Shortcuts |
Reference ADLS data via OneLake shortcuts (zero-copy) |
No data duplication |
| Incremental Load |
Watermark-based incremental copy to Lakehouse |
Efficient updates |
| Cross-Platform Invoke |
Use InvokePipeline activity to call Fabric pipelines |
Hybrid orchestration |
OneLake Shortcuts are the preferred approach when data already exists in ADLS Gen2 -- they provide instant zero-copy access without data movement. Use ADF Copy Activity only when data transformation or format conversion is needed.
For complete pipeline JSON examples for all three patterns, see references/onelake-patterns.md.
Permission Configuration
Azure Data Factory Managed Identity Permissions in Fabric
For Fabric Lakehouse:
- Open Fabric workspace
- Go to Workspace settings -> Manage access
- Add ADF managed identity with Contributor role
- Or assign Workspace Admin for full access
For Fabric Warehouse:
- Navigate to Warehouse SQL endpoint
- Execute SQL to create user:
CREATE USER [your-adf-name] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [your-adf-name];
ALTER ROLE db_datawriter ADD MEMBER [your-adf-name];
Service Principal Permissions
App Registration Setup:
- Register app in Microsoft Entra ID
- Create client secret (store in Key Vault)
- Add app to Fabric workspace with Contributor role
- For Warehouse, create SQL user as shown above
Best Practices (2025)
Use Managed Identity -- System-assigned for single ADF, user-assigned for multiple. Avoid service principal keys when possible. Store any secrets in Key Vault.
Enable Staging for Large Loads -- Use staging with compression for data volumes > 1 GB, complex transformations, or Fabric Warehouse loads.
Leverage OneLake Shortcuts -- Use ADLS Gen2 -> OneLake Shortcut -> Direct Access instead of ADLS Gen2 -> Copy Activity -> Lakehouse. No data movement, instant availability, reduced costs.
Monitor Fabric Capacity Units (CU) -- Track CU consumption per pipeline run, peak usage, and throttling. Optimize with incremental loads, off-peak scheduling, and right-sized parallelism.
Use Table Option AutoCreate -- Set tableOption: "autoCreate" on WarehouseSink for automatic schema management and faster development.
Implement Error Handling -- Configure retry policies on Copy activities and add WebActivity-based failure logging with dependencyConditions: ["Failed"].
Common Issues and Solutions
| Issue |
Error Message |
Solution |
| Permission Denied |
"User does not have permission to access Fabric workspace" |
Add ADF managed identity as Contributor; for Warehouse, create SQL user; allow 5 min propagation |
| Endpoint Not Found |
"Unable to connect to endpoint" |
Verify workspaceId/artifactId; check workspace URL; ensure Lakehouse/Warehouse is not paused |
| Schema Mismatch |
"Column types do not match" |
Use tableOption: "autoCreate" or explicit column mappings in translator |
| Performance Degradation |
Slow copy performance |
Enable staging, increase parallelCopies (4-8), increase DIUs (8-32), check CU throttling |
Resources
Progressive Disclosure References
- Lakehouse Examples:
references/lakehouse-examples.md - Complete linked service, dataset, copy activity, and lookup JSON examples
- Warehouse Examples:
references/warehouse-examples.md - Complete linked service, copy activity, stored procedure, and script activity JSON examples
- OneLake Patterns:
references/onelake-patterns.md - Pipeline patterns for shortcuts, incremental loads, and cross-platform Invoke Pipeline
1---
2name: fabric-onelake-2025
3description: |
4 ADF + Microsoft Fabric / OneLake 2025 integration.
5 PROACTIVELY activate for: (1) Fabric Lakehouse connector in ADF, (2) Fabric Warehouse connector in ADF, (3) OneLake shortcuts and cross-workspace data, (4) Invoke Pipeline activity for cross-platform orchestration (ADF -> Fabric, Fabric -> ADF), (5) copying data between ADF and Microsoft Fabric workspaces, (6) authenticating with Fabric workspace identity, (7) cross-platform parameter passing, (8) hybrid ADF + Fabric pipeline patterns.
6 Provides: Fabric connector setup, Invoke Pipeline templates, OneLake shortcut patterns, and ADF-to-Fabric migration guidance.
7metadata:
8 source: https://skillmd.com/skills/josiahsiegel/fabric-onelake-2025
9---
10
11# Microsoft Fabric Integration with Azure Data Factory (2025)
12
13## Overview
14
15Microsoft Fabric is a unified SaaS analytics platform combining Power BI, Azure Synapse Analytics, and Azure Data Factory capabilities. ADF provides native connectors for Fabric Lakehouse and Fabric Warehouse, enabling seamless data movement between ADF and Fabric workspaces.
16
17## Microsoft Fabric Lakehouse Connector
18
19The Fabric Lakehouse connector enables read and write operations to Microsoft Fabric Lakehouse for tables and files.
20
21### Supported Activities
22
23| Activity | Supported |
24|----------|-----------|
25| Copy Activity (source and sink) | Yes |
26| Lookup Activity | Yes |
27| Get Metadata Activity | Yes |
28| Delete Activity | Yes |
29
30### Quick Reference
31
32- **Linked service type:** `Lakehouse`
33- **Authentication:** Managed Identity (preferred) or Service Principal
34- **Required IDs:** `workspaceId` and `artifactId`
35- **Sink types:** `LakehouseTableSink` (tables), `LakehouseFileSink` (files)
36- **Source type:** `LakehouseTableSource`
37- **Table action options:** `append` or `overwrite`
38
39**Finding Workspace and Artifact IDs:**
401. Navigate to Fabric workspace in browser
412. Copy workspace ID from URL: `https://app.powerbi.com/groups/<workspaceId>/...`
423. Open Lakehouse settings to find artifact ID
434. Or use Fabric REST API to enumerate workspace items
44
45For complete linked service, dataset, and copy activity JSON examples, see `references/lakehouse-examples.md`.
46
47## Microsoft Fabric Warehouse Connector
48
49The Fabric Warehouse connector provides T-SQL based data warehousing capabilities within the Fabric ecosystem.
50
51### Supported Activities
52
53| Activity | Supported |
54|----------|-----------|
55| Copy Activity (source and sink) | Yes |
56| Lookup Activity | Yes |
57| Get Metadata Activity | Yes |
58| Script Activity | Yes |
59| Stored Procedure Activity | Yes |
60
61### Quick Reference
62
63- **Linked service type:** `Warehouse`
64- **Authentication:** System-Assigned Managed Identity (recommended), User-Assigned MI, or Service Principal
65- **Required properties:** `endpoint`, `warehouse`
66- **Sink type:** `WarehouseSink`
67- **Write behaviors:** `insert` or `upsert`
68- **Table option:** `autoCreate` (creates table if missing)
69
70For complete linked service, copy activity, stored procedure, and script activity JSON examples, see `references/warehouse-examples.md`.
71
72## OneLake Integration Patterns
73
74ADF supports three integration patterns with OneLake:
75
76| Pattern | Description | Key Benefit |
77|---------|-------------|-------------|
78| ADLS Gen2 Shortcuts | Reference ADLS data via OneLake shortcuts (zero-copy) | No data duplication |
79| Incremental Load | Watermark-based incremental copy to Lakehouse | Efficient updates |
80| Cross-Platform Invoke | Use InvokePipeline activity to call Fabric pipelines | Hybrid orchestration |
81
82**OneLake Shortcuts** are the preferred approach when data already exists in ADLS Gen2 -- they provide instant zero-copy access without data movement. Use ADF Copy Activity only when data transformation or format conversion is needed.
83
84For complete pipeline JSON examples for all three patterns, see `references/onelake-patterns.md`.
85
86## Permission Configuration
87
88### Azure Data Factory Managed Identity Permissions in Fabric
89
90**For Fabric Lakehouse:**
911. Open Fabric workspace
922. Go to Workspace settings -> Manage access
933. Add ADF managed identity with **Contributor** role
944. Or assign **Workspace Admin** for full access
95
96**For Fabric Warehouse:**
971. Navigate to Warehouse SQL endpoint
982. Execute SQL to create user:
99```sql
100CREATE USER [your-adf-name] FROM EXTERNAL PROVIDER;
101ALTER ROLE db_datareader ADD MEMBER [your-adf-name];
102ALTER ROLE db_datawriter ADD MEMBER [your-adf-name];
103```
104
105### Service Principal Permissions
106
107**App Registration Setup:**
1081. Register app in Microsoft Entra ID
1092. Create client secret (store in Key Vault)
1103. Add app to Fabric workspace with Contributor role
1114. For Warehouse, create SQL user as shown above
112
113## Best Practices (2025)
114
1151. **Use Managed Identity** -- System-assigned for single ADF, user-assigned for multiple. Avoid service principal keys when possible. Store any secrets in Key Vault.
116
1172. **Enable Staging for Large Loads** -- Use staging with compression for data volumes > 1 GB, complex transformations, or Fabric Warehouse loads.
118
1193. **Leverage OneLake Shortcuts** -- Use `ADLS Gen2 -> OneLake Shortcut -> Direct Access` instead of `ADLS Gen2 -> Copy Activity -> Lakehouse`. No data movement, instant availability, reduced costs.
120
1214. **Monitor Fabric Capacity Units (CU)** -- Track CU consumption per pipeline run, peak usage, and throttling. Optimize with incremental loads, off-peak scheduling, and right-sized parallelism.
122
1235. **Use Table Option AutoCreate** -- Set `tableOption: "autoCreate"` on WarehouseSink for automatic schema management and faster development.
124
1256. **Implement Error Handling** -- Configure retry policies on Copy activities and add WebActivity-based failure logging with `dependencyConditions: ["Failed"]`.
126
127## Common Issues and Solutions
128
129| Issue | Error Message | Solution |
130|-------|--------------|----------|
131| Permission Denied | "User does not have permission to access Fabric workspace" | Add ADF managed identity as Contributor; for Warehouse, create SQL user; allow 5 min propagation |
132| Endpoint Not Found | "Unable to connect to endpoint" | Verify workspaceId/artifactId; check workspace URL; ensure Lakehouse/Warehouse is not paused |
133| Schema Mismatch | "Column types do not match" | Use `tableOption: "autoCreate"` or explicit column mappings in translator |
134| Performance Degradation | Slow copy performance | Enable staging, increase parallelCopies (4-8), increase DIUs (8-32), check CU throttling |
135
136## Resources
137
138- [Fabric Lakehouse Connector](https://learn.microsoft.com/azure/data-factory/connector-microsoft-fabric-lakehouse)
139- [Fabric Warehouse Connector](https://learn.microsoft.com/azure/data-factory/connector-microsoft-fabric-warehouse)
140- [OneLake Documentation](https://learn.microsoft.com/fabric/onelake/)
141- [Fabric Capacity Management](https://learn.microsoft.com/fabric/enterprise/licenses)
142- [ADF to Fabric Integration Guide](https://learn.microsoft.com/fabric/data-factory/how-to-ingest-data-into-fabric-from-azure-data-factory)
143
144## Progressive Disclosure References
145
146- **Lakehouse Examples**: `references/lakehouse-examples.md` - Complete linked service, dataset, copy activity, and lookup JSON examples
147- **Warehouse Examples**: `references/warehouse-examples.md` - Complete linked service, copy activity, stored procedure, and script activity JSON examples
148- **OneLake Patterns**: `references/onelake-patterns.md` - Pipeline patterns for shortcuts, incremental loads, and cross-platform Invoke Pipeline