Azure Data Factory Master Knowledge Base
Deprecated Features
Apache Airflow Workflow Orchestration Manager - DEPRECATED
Status: Deprecated since early 2025. Available only for existing customers.
Retirement Date: Not yet announced, but no new deployments permitted.
Impact: New customers cannot provision Apache Airflow in Azure Data Factory.
Deprecation Details:
- Apache Airflow Workflow Orchestration Manager is deprecated with no retirement date set
- Only existing deployments can continue using this feature
- No new Airflow integrations can be created in ADF
Migration Path:
- Recommended: Migrate to Fabric Data Factory with native Airflow support
- Alternative: Use standalone Apache Airflow deployments (Azure Container Instances, AKS, or VM-based)
- Alternative: Migrate orchestration logic to native ADF pipelines with control flow activities
Why Deprecated:
- Microsoft focus shifted to Fabric Data Factory as the unified data integration platform
- Fabric provides modern orchestration capabilities superseding Airflow integration
- Limited adoption and maintenance burden for standalone Airflow feature in ADF
Action Required:
- If using Airflow in ADF: Migrate to Fabric Data Factory, standalone Airflow, or native ADF patterns
- For new projects: Do NOT use Airflow in ADF
- Monitor Microsoft announcements for official retirement timeline
Reference:
Feature Updates (2025-2026)
Microsoft Fabric Integration (GA)
ADF Mounting in Fabric:
- Bring existing ADF pipelines into Fabric workspaces without rebuilding
- Generally Available since June 2025
- Seamless integration enables hybrid ADF + Fabric workflows
Cross-Workspace Pipeline Orchestration:
- New Invoke Pipeline activity supports cross-platform calls
- Invoke pipelines across Fabric, Azure Data Factory, and Synapse
- Managed VNet support for secure cross-workspace communication
Variable Libraries:
- Environment-specific variables for CI/CD automation
- Automatic value substitution during workspace promotion
- Eliminates separate parameter files per environment
Connector Enhancements:
- ServiceNow V2 (V1 End of Support)
- Enhanced PostgreSQL and Snowflake connectors
- Native OneLake connectivity for zero-copy integration
Node.js 20.x Requirement for CI/CD
CRITICAL: As of 2025, npm package @microsoft/azure-data-factory-utilities requires Node.js 20.x
Breaking Change:
- Older Node.js versions (14.x, 16.x, 18.x) may cause package incompatibility errors
- Update CI/CD pipelines to use Node.js 20.x or compatible versions
GitHub Actions:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20.x'
Azure DevOps:
- task: UseNode@1
inputs:
version: '20.x'
Official Documentation Sources
Primary Microsoft Learn Resources
Main Documentation Hub:
- URL: https://learn.microsoft.com/en-us/azure/data-factory/
- Last Updated: February 2025
- Coverage: Complete ADF documentation including tutorials, concepts, how-to guides, and reference materials
- Key Topics: Pipelines, datasets, triggers, linked services, data flows, integration runtimes, monitoring
Introduction to Azure Data Factory:
Context7 Library Documentation
Library ID: /websites/learn_microsoft_en-us_azure_data-factory
- Trust Score: 7.5
- Code Snippets: 10,839
- Topics: CI/CD, ARM templates, pipeline patterns, data flows, monitoring, troubleshooting
How to Access:
Use Context7 MCP tool to fetch latest documentation:
mcp__context7__get-library-docs:
- context7CompatibleLibraryID: /websites/learn_microsoft_en-us_azure_data-factory
- topic: "CI/CD continuous integration deployment pipelines ARM templates"
- tokens: 8000
CI/CD Deployment
Detailed CI/CD coverage — ARM template generation, the PrePostDeploymentScript.ps1 pattern (stop/start triggers around deploys, cleanup of removed resources), and complete GitHub Actions + Azure DevOps pipeline YAMLs — lives in references/cicd-deployment.md. Load that reference when wiring continuous deployment for an ADF instance or troubleshooting a deploy pipeline.
Troubleshooting Resources
Official Troubleshooting Guide
URL: https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide
Last Updated: January 2025
Common Issues Covered:
- Template parameter validation errors
- Integration Runtime type cannot be changed
- ARM template size exceeds 4MB limit
- Git connection problems
- Authentication failures
- Deployment errors
Diagnostic Logs
Enable Diagnostic Settings:
Azure Portal → Data Factory → Diagnostic settings → Add diagnostic setting
Send to: Log Analytics workspace
Logs to Enable:
- PipelineRuns
- TriggerRuns
- ActivityRuns
- SandboxPipelineRuns
- SandboxActivityRuns
Kusto Queries for Troubleshooting:
// Failed pipeline runs in last 24 hours
ADFPipelineRun
| where Status == "Failed"
| where TimeGenerated > ago(24h)
| project TimeGenerated, PipelineName, RunId, Status, ErrorMessage, Parameters
| order by TimeGenerated desc
// Failed CI/CD deployments
ADFActivityRun
| where ActivityType == "ExecutePipeline"
| where Status == "Failed"
| where TimeGenerated > ago(7d)
| project TimeGenerated, PipelineName, ActivityName, ErrorCode, ErrorMessage
| order by TimeGenerated desc
// Performance analysis
ADFActivityRun
| where TimeGenerated > ago(7d)
| extend DurationMinutes = datetime_diff('minute', End, Start)
| summarize AvgDuration = avg(DurationMinutes) by ActivityType, ActivityName
| where AvgDuration > 10
| order by AvgDuration desc
Common Error Patterns
Error: "Template parameters are not valid"
- Cause: Deleted triggers still referenced in parameters
- Solution: Regenerate ARM template or use PrePostDeploymentScript cleanup
Error: "Updating property type is not supported"
- Cause: Trying to change Integration Runtime type
- Solution: Delete and recreate IR (not in-place update)
Error: "Operation timed out"
- Cause: Network connectivity, large data volume, insufficient compute
- Solution: Increase timeout, optimize query, increase DIUs
Error: "Authentication failed"
- Cause: Service principal expired, missing permissions, wrong credentials
- Solution: Verify credentials, check role assignments, renew if expired
Best Practices
Repository Structure
Recommended Folder Layout:
repository-root/
├── adf-resources/ # ADF JSON files (if using npm approach)
│ ├── dataset/
│ ├── pipeline/
│ ├── trigger/
│ ├── linkedService/
│ └── integrationRuntime/
├── .github/
│ └── workflows/ # GitHub Actions workflows
│ ├── adf-build.yml
│ └── adf-deploy.yml
├── azure-pipelines/ # Azure DevOps pipelines
│ ├── build.yml
│ └── release.yml
├── parameters/ # Environment-specific parameters
│ ├── ARMTemplateParametersForFactory.dev.json
│ ├── ARMTemplateParametersForFactory.test.json
│ └── ARMTemplateParametersForFactory.prod.json
├── package.json # npm configuration
└── README.md
Git Configuration
Only Configure Git on Development ADF:
- Development: Git-integrated for source control
- Test: CI/CD deployment only (no Git)
- Production: CI/CD deployment only (no Git)
Rationale: Prevents accidental manual changes in higher environments
Multi-Environment Strategy
Environment Flow:
Dev (Git) → Build → Test → Approval → Production
↓
ARM Templates
Parameter Management:
- Separate parameter file per environment
- Store secrets in Azure Key Vault
- Reference Key Vault in parameter files
- Never commit secrets to source control
Monitoring and Alerting
Set up alerts for:
- Build pipeline failures
- Deployment failures
- Pipeline run failures
- Performance degradation
- Cost anomalies
Recommended Tools:
- Azure Monitor (Metrics and Alerts)
- Log Analytics (Kusto queries)
- Application Insights (for custom logging)
- Azure Advisor (optimization recommendations)
Additional Resources
GitHub Repositories
Official Azure Data Factory Samples:
Community Examples:
- Search GitHub for "azure-data-factory-cicd" for real-world examples
- Many organizations publish their CI/CD patterns as reference
Community Support
Microsoft Q&A:
Stack Overflow:
- Tag:
azure-data-factory
- Large knowledge base of resolved issues
Azure Status:
When to Fetch Latest Information
Situations requiring current documentation:
- npm package version updates
- New ADF features or activities
- Changes to ARM template schema
- Updates to PrePostDeploymentScript
- New GitHub Actions or Azure DevOps tasks
- Breaking changes or deprecations
How to Fetch:
- Use WebFetch for Microsoft Learn articles
- Check npm for latest package version
- Use Context7 for comprehensive topic coverage
- Review Azure Data Factory GitHub for script updates
This knowledge base should be your starting point for all Azure Data Factory questions. Always verify critical information with the latest official documentation when making production decisions.
Progressive Disclosure References
For detailed JSON schemas and complete reference materials, see:
- Activity Types:
references/activity-types.md - Complete JSON schemas for all activity types (Copy, ForEach, IfCondition, Switch, Until, Lookup, ExecutePipeline, WebActivity, DatabricksJob, SetVariable, AppendVariable, Wait, Fail, GetMetadata)
- Expression Functions:
references/expression-functions.md - Complete reference for all ADF expression functions (string, collection, logical, conversion, math, date/time, pipeline/activity references)
- Linked Services:
references/linked-services.md - Complete JSON configurations for all connector types (Blob Storage, ADLS Gen2, Azure SQL, Synapse, Fabric Lakehouse/Warehouse, Databricks, Key Vault, REST, SFTP, Snowflake, PostgreSQL)
- Triggers:
references/triggers.md - Complete JSON schemas for schedule, tumbling window, and event triggers
- Datasets:
references/datasets.md - Complete JSON schemas for all dataset types with parameterization patterns
For machine learning and analytics patterns, see the dedicated skill:
- ML & Analytics:
adf-master:adf-ml-analytics - Azure ML pipelines, batch endpoints, Azure AI Services, Databricks ML/MLflow, SQL-to-Storage archival, feature engineering with Data Flows
1---
2name: adf-master
3description: |
4 Azure Data Factory (ADF) CI/CD, deployment, and pipeline development.
5 PROACTIVELY activate for: (1) ADF CI/CD setup (npm validation, ARM template export), (2) ADF ARM template deployment, (3) ADF npm build validation in CI, (4) PrePostDeploymentScript for trigger and resource cleanup, (5) ADF GitHub Actions workflows, (6) ADF Azure DevOps pipelines, (7) ADF Git integration (collaboration vs publish branch), (8) parameterizing linked services and datasets across environments, (9) ADF triggers (schedule, tumbling window, event), (10) deployment slots and blue-green for ADF.
6 Provides: complete CI/CD YAML for GitHub Actions and Azure DevOps, PrePostDeploymentScript reference, parameterization patterns, and trigger management recipes.
7metadata:
8 source: https://skillmd.com/skills/josiahsiegel/adf-master
9---
10
11# Azure Data Factory Master Knowledge Base
12
13## Deprecated Features
14
15### Apache Airflow Workflow Orchestration Manager - DEPRECATED
16
17**Status:** Deprecated since early 2025. Available only for existing customers.
18**Retirement Date:** Not yet announced, but no new deployments permitted.
19**Impact:** New customers cannot provision Apache Airflow in Azure Data Factory.
20
21**Deprecation Details:**
22- Apache Airflow Workflow Orchestration Manager is deprecated with no retirement date set
23- Only existing deployments can continue using this feature
24- No new Airflow integrations can be created in ADF
25
26**Migration Path:**
27- **Recommended:** Migrate to Fabric Data Factory with native Airflow support
28- **Alternative:** Use standalone Apache Airflow deployments (Azure Container Instances, AKS, or VM-based)
29- **Alternative:** Migrate orchestration logic to native ADF pipelines with control flow activities
30
31**Why Deprecated:**
32- Microsoft focus shifted to Fabric Data Factory as the unified data integration platform
33- Fabric provides modern orchestration capabilities superseding Airflow integration
34- Limited adoption and maintenance burden for standalone Airflow feature in ADF
35
36**Action Required:**
37- If using Airflow in ADF: Migrate to Fabric Data Factory, standalone Airflow, or native ADF patterns
38- For new projects: Do NOT use Airflow in ADF
39- Monitor Microsoft announcements for official retirement timeline
40
41**Reference:**
42- Microsoft Roadmap: https://www.directionsonmicrosoft.com/roadmaps/ref/azure-data-factory-roadmap/
43
44## Feature Updates (2025-2026)
45
46### Microsoft Fabric Integration (GA)
47
48**ADF Mounting in Fabric:**
49- Bring existing ADF pipelines into Fabric workspaces without rebuilding
50- Generally Available since June 2025
51- Seamless integration enables hybrid ADF + Fabric workflows
52
53**Cross-Workspace Pipeline Orchestration:**
54- New **Invoke Pipeline** activity supports cross-platform calls
55- Invoke pipelines across Fabric, Azure Data Factory, and Synapse
56- Managed VNet support for secure cross-workspace communication
57
58**Variable Libraries:**
59- Environment-specific variables for CI/CD automation
60- Automatic value substitution during workspace promotion
61- Eliminates separate parameter files per environment
62
63**Connector Enhancements:**
64- ServiceNow V2 (V1 End of Support)
65- Enhanced PostgreSQL and Snowflake connectors
66- Native OneLake connectivity for zero-copy integration
67
68### Node.js 20.x Requirement for CI/CD
69
70**CRITICAL:** As of 2025, npm package `@microsoft/azure-data-factory-utilities` requires Node.js 20.x
71
72**Breaking Change:**
73- Older Node.js versions (14.x, 16.x, 18.x) may cause package incompatibility errors
74- Update CI/CD pipelines to use Node.js 20.x or compatible versions
75
76**GitHub Actions:**
77```yaml
78- name: Setup Node.js
79 uses: actions/setup-node@v4
80 with:
81 node-version: '20.x'
82```
83
84**Azure DevOps:**
85```yaml
86- task: UseNode@1
87 inputs:
88 version: '20.x'
89```
90
91## Official Documentation Sources
92
93### Primary Microsoft Learn Resources
94
95**Main Documentation Hub:**
96- URL: https://learn.microsoft.com/en-us/azure/data-factory/
97- Last Updated: February 2025
98- Coverage: Complete ADF documentation including tutorials, concepts, how-to guides, and reference materials
99- Key Topics: Pipelines, datasets, triggers, linked services, data flows, integration runtimes, monitoring
100
101**Introduction to Azure Data Factory:**
102- URL: https://learn.microsoft.com/en-us/azure/data-factory/introduction
103- Summary: Managed cloud service for complex hybrid ETL, ELT, and data integration projects
104- Key Features: 90+ built-in connectors, serverless architecture, code-free UI, single-pane monitoring
105
106### Context7 Library Documentation
107
108**Library ID:** `/websites/learn_microsoft_en-us_azure_data-factory`
109- Trust Score: 7.5
110- Code Snippets: 10,839
111- Topics: CI/CD, ARM templates, pipeline patterns, data flows, monitoring, troubleshooting
112
113**How to Access:**
114```text
115Use Context7 MCP tool to fetch latest documentation:
116mcp__context7__get-library-docs:
117 - context7CompatibleLibraryID: /websites/learn_microsoft_en-us_azure_data-factory
118 - topic: "CI/CD continuous integration deployment pipelines ARM templates"
119 - tokens: 8000
120```
121
122## CI/CD Deployment
123
124Detailed CI/CD coverage — ARM template generation, the `PrePostDeploymentScript.ps1` pattern (stop/start triggers around deploys, cleanup of removed resources), and complete GitHub Actions + Azure DevOps pipeline YAMLs — lives in `references/cicd-deployment.md`. Load that reference when wiring continuous deployment for an ADF instance or troubleshooting a deploy pipeline.
125
126## Troubleshooting Resources
127
128### Official Troubleshooting Guide
129
130**URL:** https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide
131**Last Updated:** January 2025
132
133**Common Issues Covered:**
1341. Template parameter validation errors
1352. Integration Runtime type cannot be changed
1363. ARM template size exceeds 4MB limit
1374. Git connection problems
1385. Authentication failures
1396. Deployment errors
140
141### Diagnostic Logs
142
143**Enable Diagnostic Settings:**
144```text
145Azure Portal → Data Factory → Diagnostic settings → Add diagnostic setting
146Send to: Log Analytics workspace
147
148Logs to Enable:
149- PipelineRuns
150- TriggerRuns
151- ActivityRuns
152- SandboxPipelineRuns
153- SandboxActivityRuns
154```
155
156**Kusto Queries for Troubleshooting:**
157
158```kusto
159// Failed pipeline runs in last 24 hours
160ADFPipelineRun
161| where Status == "Failed"
162| where TimeGenerated > ago(24h)
163| project TimeGenerated, PipelineName, RunId, Status, ErrorMessage, Parameters
164| order by TimeGenerated desc
165
166// Failed CI/CD deployments
167ADFActivityRun
168| where ActivityType == "ExecutePipeline"
169| where Status == "Failed"
170| where TimeGenerated > ago(7d)
171| project TimeGenerated, PipelineName, ActivityName, ErrorCode, ErrorMessage
172| order by TimeGenerated desc
173
174// Performance analysis
175ADFActivityRun
176| where TimeGenerated > ago(7d)
177| extend DurationMinutes = datetime_diff('minute', End, Start)
178| summarize AvgDuration = avg(DurationMinutes) by ActivityType, ActivityName
179| where AvgDuration > 10
180| order by AvgDuration desc
181```
182
183### Common Error Patterns
184
185**Error: "Template parameters are not valid"**
186- Cause: Deleted triggers still referenced in parameters
187- Solution: Regenerate ARM template or use PrePostDeploymentScript cleanup
188
189**Error: "Updating property type is not supported"**
190- Cause: Trying to change Integration Runtime type
191- Solution: Delete and recreate IR (not in-place update)
192
193**Error: "Operation timed out"**
194- Cause: Network connectivity, large data volume, insufficient compute
195- Solution: Increase timeout, optimize query, increase DIUs
196
197**Error: "Authentication failed"**
198- Cause: Service principal expired, missing permissions, wrong credentials
199- Solution: Verify credentials, check role assignments, renew if expired
200
201## Best Practices
202
203### Repository Structure
204
205**Recommended Folder Layout:**
206```text
207repository-root/
208├── adf-resources/ # ADF JSON files (if using npm approach)
209│ ├── dataset/
210│ ├── pipeline/
211│ ├── trigger/
212│ ├── linkedService/
213│ └── integrationRuntime/
214├── .github/
215│ └── workflows/ # GitHub Actions workflows
216│ ├── adf-build.yml
217│ └── adf-deploy.yml
218├── azure-pipelines/ # Azure DevOps pipelines
219│ ├── build.yml
220│ └── release.yml
221├── parameters/ # Environment-specific parameters
222│ ├── ARMTemplateParametersForFactory.dev.json
223│ ├── ARMTemplateParametersForFactory.test.json
224│ └── ARMTemplateParametersForFactory.prod.json
225├── package.json # npm configuration
226└── README.md
227```
228
229### Git Configuration
230
231**Only Configure Git on Development ADF:**
232- Development: Git-integrated for source control
233- Test: CI/CD deployment only (no Git)
234- Production: CI/CD deployment only (no Git)
235
236**Rationale:** Prevents accidental manual changes in higher environments
237
238### Multi-Environment Strategy
239
240```text
241Environment Flow:
242Dev (Git) → Build → Test → Approval → Production
243 ↓
244 ARM Templates
245```
246
247**Parameter Management:**
248- Separate parameter file per environment
249- Store secrets in Azure Key Vault
250- Reference Key Vault in parameter files
251- Never commit secrets to source control
252
253### Monitoring and Alerting
254
255**Set up alerts for:**
256- Build pipeline failures
257- Deployment failures
258- Pipeline run failures
259- Performance degradation
260- Cost anomalies
261
262**Recommended Tools:**
263- Azure Monitor (Metrics and Alerts)
264- Log Analytics (Kusto queries)
265- Application Insights (for custom logging)
266- Azure Advisor (optimization recommendations)
267
268## Additional Resources
269
270### GitHub Repositories
271
272**Official Azure Data Factory Samples:**
273- URL: https://github.com/Azure/Azure-DataFactory
274- Path: SamplesV2/ContinuousIntegrationAndDelivery/
275- Contents: PrePostDeploymentScript.Ver2.ps1, example pipelines, documentation
276
277**Community Examples:**
278- Search GitHub for "azure-data-factory-cicd" for real-world examples
279- Many organizations publish their CI/CD patterns as reference
280
281### Community Support
282
283**Microsoft Q&A:**
284- URL: https://learn.microsoft.com/en-us/answers/tags/130/azure-data-factory
285- Active community, Microsoft employees respond
286
287**Stack Overflow:**
288- Tag: `azure-data-factory`
289- Large knowledge base of resolved issues
290
291**Azure Status:**
292- URL: https://status.azure.com
293- Check for service outages and incidents
294
295## When to Fetch Latest Information
296
297**Situations requiring current documentation:**
2981. npm package version updates
2992. New ADF features or activities
3003. Changes to ARM template schema
3014. Updates to PrePostDeploymentScript
3025. New GitHub Actions or Azure DevOps tasks
3036. Breaking changes or deprecations
304
305**How to Fetch:**
306- Use WebFetch for Microsoft Learn articles
307- Check npm for latest package version
308- Use Context7 for comprehensive topic coverage
309- Review Azure Data Factory GitHub for script updates
310
311This knowledge base should be your starting point for all Azure Data Factory questions. Always verify critical information with the latest official documentation when making production decisions.
312
313## Progressive Disclosure References
314
315For detailed JSON schemas and complete reference materials, see:
316
317- **Activity Types**: `references/activity-types.md` - Complete JSON schemas for all activity types (Copy, ForEach, IfCondition, Switch, Until, Lookup, ExecutePipeline, WebActivity, DatabricksJob, SetVariable, AppendVariable, Wait, Fail, GetMetadata)
318- **Expression Functions**: `references/expression-functions.md` - Complete reference for all ADF expression functions (string, collection, logical, conversion, math, date/time, pipeline/activity references)
319- **Linked Services**: `references/linked-services.md` - Complete JSON configurations for all connector types (Blob Storage, ADLS Gen2, Azure SQL, Synapse, Fabric Lakehouse/Warehouse, Databricks, Key Vault, REST, SFTP, Snowflake, PostgreSQL)
320- **Triggers**: `references/triggers.md` - Complete JSON schemas for schedule, tumbling window, and event triggers
321- **Datasets**: `references/datasets.md` - Complete JSON schemas for all dataset types with parameterization patterns
322
323For machine learning and analytics patterns, see the dedicated skill:
324- **ML & Analytics**: `adf-master:adf-ml-analytics` - Azure ML pipelines, batch endpoints, Azure AI Services, Databricks ML/MLflow, SQL-to-Storage archival, feature engineering with Data Flows