# Azure Data Factory

> Azure Data Factory Skill

- Skill: `comeonoliver/azure-data-factory` (Agent Skill)
- Install (CLI): `npx skillmds@latest add comeonoliver/azure-data-factory`
- Raw SKILL.md: https://api.skillmd.com/api/skills/comeonoliver/azure-data-factory/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: ComeOnOliver (https://skillmd.com/u/comeonoliver)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/comeonoliver/azure-data-factory

---

# Azure Data Factory Skill

This skill provides expert guidance for Azure Data Factory. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

## How to Use This Skill

> **IMPORTANT for Agent**: Use the **Category Index** below to locate relevant sections. For categories with line ranges (e.g., `L35-L120`), use `read_file` with the specified lines. For categories with file links (e.g., `[security.md](security.md)`), use `read_file` on the linked reference file

> **IMPORTANT for Agent**: If `metadata.generated_at` is more than 3 months old, suggest the user pull the latest version from the repository. If `mcp_microsoftdocs` tools are not available, suggest the user install it: [Installation Guide](https://github.com/MicrosoftDocs/mcp/blob/main/README.md)

This skill requires **network access** to fetch documentation content:
- **Preferred**: Use `mcp_microsoftdocs:microsoft_docs_fetch` with query string `from=learn-agent-skill`. Returns Markdown.
- **Fallback**: Use `fetch_webpage` with query string `from=learn-agent-skill&accept=text/markdown`. Returns Markdown.

## Category Index

| Category | Lines | Description |
|----------|-------|-------------|
| Troubleshooting | L37-L90 | Diagnosing and fixing ADF issues: connector and format errors, pipeline/orchestration and data flow failures, CI/CD and Studio problems, SHIR/SSIS IR troubleshooting, and performance/debug guidance. |
| Best Practices | L91-L116 | Performance, reliability, and migration best practices for ADF: tuning data flows/IRs, optimizing copy/source/sink, handling schema drift/errors, DataOps/BCDR, SAP CDC, and common migration patterns. |
| Decision Making | L117-L137 | Guidance on ADF cost planning, pricing models, runtimes and compute choices, connector selection and upgrades, and assessing/migrating pipelines, SSIS, and data warehouses to ADF/Fabric |
| Architecture & Design Patterns | L138-L148 | Patterns and architectures for ADF pipelines: efficient mapping data flows, metadata‑driven copy at scale, SSIS IR with SQL/MI/VNet, Cosmos DB migration, and SAP CDC design. |
| Limits & Quotas | L149-L154 | Info on ADF connector lifecycle stages and timelines, plus how reservation discounts work for Mapping Data Flows and how they affect cost and capacity planning |
| Security | L155-L182 | Securing Data Factory with identity, encryption, Key Vault, network isolation (Private Link, managed VNets, private endpoints), firewall rules, policies, and secure runtimes/SQL access. |
| Configuration | L183-L318 | Configuring ADF and integration runtimes (Azure, self-hosted, SSIS, Airflow), datasets, activities, data flows, triggers, formats, CDC, monitoring, logging, networking, and parameters. |
| Integrations & Coding Patterns | L319-L502 | Using ADF with external systems: connector how-tos, copy/transform patterns, SSIS/Databricks/Synapse/ML integration, REST/webhooks, SDK usage, and pipeline templates for common ETL scenarios. |
| Deployment | L503-L519 | CI/CD and deployment for ADF and Workflow Orchestration Manager: ARM/linked templates, Azure DevOps pipelines, hotfix flows, pre/post scripts, and integration runtime install/containerization |

### Troubleshooting
| Topic | URL |
|-------|-----|
| Resolve Azure Data Factory change data capture issues | https://learn.microsoft.com/en-us/azure/data-factory/change-data-capture-troubleshoot |
| Troubleshoot CI/CD, Azure DevOps, and GitHub integration for ADF | https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide |
| Resolve common Azure Data Factory connector upgrade issues | https://learn.microsoft.com/en-us/azure/data-factory/connector-deprecation-frequently-asked-questions |
| Fix Azure Data Factory Amazon S3 connector errors | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-amazon-simple-storage-service |
| Resolve Azure Blob Storage connector issues in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-blob-storage |
| Troubleshoot Azure Cosmos DB connectors in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-cosmos-db |
| Fix Azure Data Explorer connector problems in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-explorer |
| Resolve Azure Data Lake Storage connector errors in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-lake |
| Fix Azure Files connector issues in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-files |
| Resolve Azure Table Storage connector errors in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-table-storage |
| Troubleshoot DB2 connector problems in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-db2 |
| Fix delimited text format connector issues in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-delimited-text |
| Resolve Dynamics 365 and Dataverse connector errors in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-dynamics-dataverse |
| Troubleshoot file system connector failures in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-file-system |
| Fix FTP, SFTP, and HTTP connector issues in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-ftp-sftp-http |
| Troubleshoot Google Ads connector problems in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-google-ads |
| Troubleshoot Azure Data Factory connector failures | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-guide |
| Resolve Hive connector errors in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-hive |
| Troubleshoot Microsoft Fabric Lakehouse connector in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-microsoft-fabric-lakehouse |
| Fix Microsoft Fabric Warehouse connector issues in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-microsoft-fabric-warehouse |
| Troubleshoot MongoDB connector failures in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-mongodb |
| Resolve Oracle connector issues in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-oracle |
| Fix ORC format connector problems in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-orc |
| Troubleshoot Parquet format connector issues in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-parquet |
| Troubleshoot Azure Database for PostgreSQL connector in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-postgresql |
| Resolve REST connector errors in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-rest |
| Troubleshoot Salesforce and Service Cloud connectors in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-salesforce |
| Fix SAP Table, BW Open Hub, and ODP connector issues in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-sap |
| Troubleshoot ServiceNow connector problems in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-servicenow |
| Resolve SharePoint Online list connector errors in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-sharepoint-online-list |
| Troubleshoot Snowflake connector issues in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-snowflake |
| Troubleshoot SQL-based connectors in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-synapse-sql |
| Fix Teradata connector failures in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-teradata |
| Troubleshoot XML format connector problems in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-xml |
| Troubleshoot Copy activity performance issues | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance-troubleshooting |
| Troubleshoot external control activities in ADF pipelines | https://learn.microsoft.com/en-us/azure/data-factory/data-factory-troubleshoot-guide |
| Diagnose and fix Azure Data Factory Studio issues | https://learn.microsoft.com/en-us/azure/data-factory/data-factory-ux-troubleshoot-guide |
| Troubleshoot mapping data flow connector and format issues | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-connector-format |
| Resolve mapping data flow issues in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-guide |
| Iteratively debug Azure Data Factory pipelines | https://learn.microsoft.com/en-us/azure/data-factory/iterative-development-debugging |
| Work around known Azure Data Factory issues | https://learn.microsoft.com/en-us/azure/data-factory/known-issues-troubleshoot-guide |
| Troubleshoot Azure Data Factory pipeline orchestration and triggers | https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide |
| Send SHIR logs to debug SAP CDC connector issues | https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-debug-shir-logs |
| Resolve security and access control issues in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/security-and-access-control-troubleshoot-guide |
| Use the self-hosted integration runtime diagnostic tool | https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-diagnostic-tool |
| Diagnose and fix self-hosted IR issues in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-troubleshoot-guide |
| Use SSIS IR connectivity diagnostics to fix failures | https://learn.microsoft.com/en-us/azure/data-factory/ssis-integration-runtime-diagnose-connectivity-faq |
| Resolve SSIS Integration Runtime management problems | https://learn.microsoft.com/en-us/azure/data-factory/ssis-integration-runtime-management-troubleshoot |
| Troubleshoot SSIS package execution on SSIS IR | https://learn.microsoft.com/en-us/azure/data-factory/ssis-integration-runtime-ssis-activity-faq |
| Interpret pipeline failure status and error messages in ADF | https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-failure-error-handling |

### Best Practices
| Topic | URL |
|-------|-----|
| Apply DataOps practices to Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/apply-dataops |
| Use automatic connector upgrades in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/automatic-connector-upgrade |
| Use column patterns in mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-column-pattern |
| Reuse logic with flowlets in mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-flowlet |
| Tune mapping data flow performance in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance |
| Optimize sink performance in mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sinks |
| Optimize source performance in mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sources |
| Improve transformation performance in mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-transformations |
| Handle schema drift in mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-schema-drift |
| Tune integration runtime performance for data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance |
| Use nested activities effectively in Data Factory pipelines | https://learn.microsoft.com/en-us/azure/data-factory/concepts-nested-activities |
| Tune Azure-SSIS integration runtime for high performance | https://learn.microsoft.com/en-us/azure/data-factory/configure-azure-ssis-integration-runtime-performance |
| Optimize Copy activity performance and scalability | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance |
| Migrate on-premises HDFS data to Azure Storage | https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-hdfs-azure-storage |
| Migrate Netezza data to Azure with Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-netezza-azure-sqldw |
| Migrate data from Amazon S3 to Azure Storage with ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-s3-azure-storage |
| Use data flow snippets to dedupe rows and find nulls | https://learn.microsoft.com/en-us/azure/data-factory/how-to-data-flow-dedupe-nulls-snippets |
| Handle SQL truncation and error rows in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/how-to-data-flow-error-rows |
| Implement BCDR strategies for Data Factory pipelines | https://learn.microsoft.com/en-us/azure/data-factory/pipelines-disaster-recovery |
| Apply advanced patterns and best practices for SAP CDC | https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-advanced-topics |
| Operate and manage SAP CDC pipelines in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-management |
| Apply best practices for writing data lake files with ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-write-to-lake |

### Decision Making
| Topic | URL |
|-------|-----|
| Estimate costs with Workflow Orchestration Manager pricing | https://learn.microsoft.com/en-us/azure/data-factory/airflow-pricing |
| Interpret pricing examples for ADF integration runtimes | https://learn.microsoft.com/en-us/azure/data-factory/better-understand-different-integration-runtime-charges |
| Choose the right integration runtime architecture | https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configuration |
| Choose compute environments for Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/compute-linked-services |
| Decide when to use Workflow Orchestration Manager | https://learn.microsoft.com/en-us/azure/data-factory/concepts-workflow-orchestration-manager |
| Use connector upgrade advisor for ADF and Synapse | https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-advisor |
| Plan and execute Azure Data Factory connector upgrades | https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-guidance |
| Buy and use ADF data flow reserved capacity | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-reserved-capacity-overview |
| Plan data lake and warehouse migration with ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-overview |
| Answer common Azure Data Factory usage questions | https://learn.microsoft.com/en-us/azure/data-factory/frequently-asked-questions |
| Assess Azure Data Factory pipelines for Fabric migration | https://learn.microsoft.com/en-us/azure/data-factory/how-to-assess-your-azure-data-factory-to-fabric-data-factory-migration |
| Choose and provision Enterprise Edition for Azure-SSIS integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition |
| Assess and upgrade Azure Data Factory pipelines to Fabric | https://learn.microsoft.com/en-us/azure/data-factory/how-to-upgrade-your-azure-data-factory-pipelines-to-fabric-data-factory |
| Choose the right Azure Data Factory SAP connector | https://learn.microsoft.com/en-us/azure/data-factory/industry-sap-connectors |
| Plan and manage Azure Data Factory costs | https://learn.microsoft.com/en-us/azure/data-factory/plan-manage-costs |
| Plan migration of on-premises SSIS workloads to ADF | https://learn.microsoft.com/en-us/azure/data-factory/scenario-ssis-migration-overview |
| Apply migration assessment rules for SSIS to ADF | https://learn.microsoft.com/en-us/azure/data-factory/scenario-ssis-migration-rules |

### Architecture & Design Patterns
| Topic | URL |
|-------|-----|
| Design efficient pipelines using mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-pipelines |
| Configure Azure-SSIS IR with SQL failover groups | https://learn.microsoft.com/en-us/azure/data-factory/configure-bcdr-azure-ssis-integration-runtime |
| Build metadata-driven large-scale copy pipelines | https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven |
| Migrate Azure SQL schemas to Azure Cosmos DB with ADF | https://learn.microsoft.com/en-us/azure/data-factory/how-to-sqldb-to-cosmosdb |
| Use Azure SQL Managed Instance with Azure-SSIS IR | https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-sql-managed-instance-with-ir |
| Decide when and how to join Azure-SSIS integration runtime to a virtual network | https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network |
| Understand architecture of SAP CDC in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-introduction-architecture |

### Limits & Quotas
| Topic | URL |
|-------|-----|
| Understand connector release stages and timelines in ADF | https://learn.microsoft.com/en-us/azure/data-factory/connector-release-stages-and-timelines |
| Understand reservation discounts for ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-understand-reservation-charges |

### Security
| Topic | URL |
|-------|-----|
| Allow Azure Integration Runtime IP ranges in firewalls | https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses |
| Configure roles and permissions for Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/concepts-roles-permissions |
| Configure outbound FQDN allow lists with Azure Policy for ADF | https://learn.microsoft.com/en-us/azure/data-factory/configure-outbound-allow-list-azure-policy |
| Use Azure credentials and secrets in ADF | https://learn.microsoft.com/en-us/azure/data-factory/credentials |
| Plan secure data access strategies for ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-access-strategies |
| Configure Azure Private Link for Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/data-factory-private-link |
| Configure managed identities for Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/data-factory-service-identity |
| Security infrastructure and data movement in ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-movement-security-considerations |
| Enable Microsoft Entra authentication for Azure-SSIS integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/enable-aad-authentication-azure-ssis-ir |
| Use Azure Key Vault as secrets backend for Airflow | https://learn.microsoft.com/en-us/azure/data-factory/enable-azure-key-vault |
| Enable customer-managed keys (BYOK) for ADF encryption | https://learn.microsoft.com/en-us/azure/data-factory/enable-customer-managed-key |
| Encrypt on-premises credentials for self-hosted IR | https://learn.microsoft.com/en-us/azure/data-factory/encrypt-credentials-self-hosted-integration-runtime |
| Access secured Purview from Data Factory via private endpoints | https://learn.microsoft.com/en-us/azure/data-factory/how-to-access-secured-purview-account |
| Use Azure Key Vault secrets in ADF pipeline activities | https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-azure-key-vault-secrets-pipeline-activities |
| Use managed virtual networks and private endpoints in ADF | https://learn.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint |
| Apply Azure Policy built-ins for Data Factory governance | https://learn.microsoft.com/en-us/azure/data-factory/policy-reference |
| Secure Azure Data Factory with network and identity controls | https://learn.microsoft.com/en-us/azure/data-factory/secure-your-azure-data-factory |
| Detect and mask PII data with ADF template | https://learn.microsoft.com/en-us/azure/data-factory/solution-template-pii-detection-and-masking |
| Configure Windows authentication for SSIS packages in Azure | https://learn.microsoft.com/en-us/azure/data-factory/ssis-azure-connect-with-windows-auth |
| Store ADF credentials securely in Azure Key Vault | https://learn.microsoft.com/en-us/azure/data-factory/store-credentials-in-key-vault |
| Use private endpoints to securely copy data with ADF | https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal-private |
| Secure self-hosted integration runtime with TLS from intranet | https://learn.microsoft.com/en-us/azure/data-factory/tutorial-enable-remote-access-intranet-tls-ssl-certificate |
| Secure on-prem SQL access via Data Factory managed VNet | https://learn.microsoft.com/en-us/azure/data-factory/tutorial-managed-virtual-network-on-premise-sql-server |
| Configure private endpoint access to SQL Managed Instance from Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/tutorial-managed-virtual-network-sql-managed-instance |

### Configuration
| Topic | URL |
|-------|-----|
| Reference supported Apache Airflow configurations in ADF | https://learn.microsoft.com/en-us/azure/data-factory/airflow-configurations |
| Retrieve IP address of Workflow Orchestration Manager cluster | https://learn.microsoft.com/en-us/azure/data-factory/airflow-get-ip-airflow-cluster |
| Install private Python packages in Airflow runtime | https://learn.microsoft.com/en-us/azure/data-factory/airflow-install-private-package |
| Define and use global parameters in ADF | https://learn.microsoft.com/en-us/azure/data-factory/author-global-parameters |
| Use the Azure Data Factory management hub | https://learn.microsoft.com/en-us/azure/data-factory/author-management-hub |
| Configure express virtual network injection for Azure-SSIS integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-express-virtual-network-injection |
| Manage SSIS packages with Azure-SSIS package store | https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-package-store |
| Configure standard virtual network injection for Azure-SSIS integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-standard-virtual-network-injection |
| Configure virtual network for Azure-SSIS integration runtime injection | https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-virtual-network-configuration |
| Reference built-in and preinstalled components on Azure-SSIS integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/built-in-preinstalled-components-ssis-integration-runtime |
| Configure change data capture resources in ADF | https://learn.microsoft.com/en-us/azure/data-factory/concepts-change-data-capture-resource |
| Define and configure datasets in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services |
| Configure integration runtimes in Data Factory and Synapse | https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime |
| Configure linked services in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/concepts-linked-services |
| Configure Append Variable activity in pipelines | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-append-variable-activity |
| Configure Data Flow activity execution | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-execute-data-flow-activity |
| Configure Execute Pipeline activity chaining | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-execute-pipeline-activity |
| Use Azure Data Factory expression language and functions | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions |
| Use Fail activity to raise custom errors | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-fail-activity |
| Configure Filter activity for pipeline arrays | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-filter-activity |
| Configure ForEach looping in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity |
| Use Get Metadata activity on datasets | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity |
| Configure If Condition branching in pipelines | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-if-condition-activity |
| Configure Lookup activity for dynamic datasets | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity |
| Use Power Query activity for data wrangling | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-power-query-activity |
| Use Set Variable activity and return values | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-set-variable-activity |
| Configure Switch activity for case routing | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-switch-activity |
| Use system variables in Azure Data Factory expressions | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-system-variables |
| Configure Until looping with timeouts | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-until-activity |
| Configure Validation activity for datasets | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-validation-activity |
| Configure Wait activity for pipeline delays | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-wait-activity |
| Enable data consistency verification in Copy activity | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-data-consistency |
| Configure fault tolerance for Copy activity | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-fault-tolerance |
| Configure session logging for Copy activity | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-log |
| Monitor Copy activity runs and metrics | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-monitoring |
| Configure Copy activity in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview |
| Use Copy activity performance optimization features | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance-features |
| Preserve metadata and ACLs in Copy activity | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-preserve-metadata |
| Configure schema and data type mapping in Copy activity | https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping |
| Use Copy Data tool for guided ingestion | https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool |
| Configure an Apache Airflow environment in ADF | https://learn.microsoft.com/en-us/azure/data-factory/create-airflow-environment |
| Create and configure Azure integration runtime for Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/create-azure-integration-runtime |
| Provision Azure-SSIS integration runtime in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/create-azure-ssis-integration-runtime |
| Create Azure-SSIS integration runtime via Azure portal | https://learn.microsoft.com/en-us/azure/data-factory/create-azure-ssis-integration-runtime-portal |
| Create Azure-SSIS integration runtime with Azure PowerShell | https://learn.microsoft.com/en-us/azure/data-factory/create-azure-ssis-integration-runtime-powershell |
| Create and configure self-hosted integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime |
| Create shared self-hosted integration runtime with PowerShell | https://learn.microsoft.com/en-us/azure/data-factory/create-shared-self-hosted-integration-runtime-powershell |
| Configure cross-tenant Azure DevOps connections for ADF | https://learn.microsoft.com/en-us/azure/data-factory/cross-tenant-connections-to-azure-devops |
| Configure Aggregate transformation in data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-aggregate |
| Configure Alter row transformation for upserts | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-alter-row |
| Set data quality assertions in data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-assert |
| Configure Cast transformation for type changes | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-cast |
| Configure Conditional split transformation rules | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-conditional-split |
| Configure Derived column expressions in data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-derived-column |
| Configure Exists transformation for row matching | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-exists |
| Configure Filter transformation for row selection | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-filter |
| Configure Flatten transformation for hierarchical data | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-flatten |
| Use Flowlet transformation for reusable logic | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-flowlet |
| Configure Join transformation for combining streams | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-join |
| Configure Lookup transformation for reference data | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-lookup |
| Configure multiple branches in mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-new-branch |
| Configure Parse transformation for embedded documents | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-parse |
| Configure Pivot transformation for row-to-column | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-pivot |
| Configure Rank transformation in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-rank |
| Author and manage mapping data flow script in ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-script |
| Configure Select transformation in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-select |
| Configure sink transformation options in ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink |
| Configure Sort transformation in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sort |
| Configure source transformation in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-source |
| Use Stringify transformation for complex types in ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-stringify |
| Configure Surrogate Key transformation in ADF | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-surrogate-key |
| Configure Union transformation in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-union |
| Configure Unpivot transformation in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-unpivot |
| Configure Window transformation in ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-window |
| Deactivate activities in Azure Data Factory pipelines | https://learn.microsoft.com/en-us/azure/data-factory/deactivate-activity |
| Configure Delete activity for file cleanup | https://learn.microsoft.com/en-us/azure/data-factory/delete-activity |
| Configure diagnostics logs and metrics for Airflow IR | https://learn.microsoft.com/en-us/azure/data-factory/diagnostic-logs-and-metrics-for-workflow-orchestration-manager |
| Configure Avro format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-avro |
| Configure Binary format handling in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-binary |
| Configure Common Data Model format in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-common-data-model |
| Configure delimited text datasets in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-delimited-text |
| Configure Delta Lake format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-delta |
| Configure Excel file format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-excel |
| Configure Iceberg table format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-iceberg |
| Configure JSON file format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-json |
| Configure ORC format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-orc |
| Configure Parquet format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-parquet |
| Configure XML format in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/format-xml |
| Configure Bring Your Own Driver connectors in ADF | https://learn.microsoft.com/en-us/azure/data-factory/guidance-for-byod-approach |
| Configure CDC resource from ADLS Gen2 to Azure SQL | https://learn.microsoft.com/en-us/azure/data-factory/how-to-change-data-capture-resource |
| Configure CDC with schema evolution to Delta sinks | https://learn.microsoft.com/en-us/azure/data-factory/how-to-change-data-capture-resource-with-schema-evolution |
| Customize Azure-SSIS integration runtime setup with additional components | https://learn.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup |
| Configure self-hosted integration runtime for Log Analytics | https://learn.microsoft.com/en-us/azure/data-factory/how-to-configure-shir-for-log-analytics-collection |
| Configure custom Event Grid triggers for Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-custom-event-trigger |
| Create storage event-based triggers in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger |
| Configure schedule triggers in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-schedule-trigger |
| Configure tumbling window triggers in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger |
| Develop and install licensed components for Azure-SSIS integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/how-to-develop-azure-ssis-ir-licensed-components |
| Use parameters and expressions in ADF pipelines | https://learn.microsoft.com/en-us/azure/data-factory/how-to-expression-language-functions |
| Process fixed-width text files with ADF mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/how-to-fixed-width |
| Manage Azure Data Factory studio settings and preferences | https://learn.microsoft.com/en-us/azure/data-factory/how-to-manage-settings |
| Manage preview features in Azure Data Factory studio | https://learn.microsoft.com/en-us/azure/data-factory/how-to-manage-studio-preview-exp |
| Schedule start and stop of Azure-SSIS integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/how-to-schedule-azure-ssis-integration-runtime |
| Use trigger metadata parameters in ADF pipelines | https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-trigger-parameterization |
| Join Azure-SSIS integration runtime to a virtual network via PowerShell | https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network-powershell |
| Join Azure-SSIS integration runtime to a virtual network via portal | https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network-ui |
| Configure Kubernetes secret for private container images | https://learn.microsoft.com/en-us/azure/data-factory/kubernetes-secret-pull-image-from-private-container-registry |
| Reconfigure Azure-SSIS integration runtime settings in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/manage-azure-ssis-integration-runtime |
| Configure diagnostic settings and Log Analytics for ADF | https://learn.microsoft.com/en-us/azure/data-factory/monitor-configure-diagnostics |
| Monitor Azure Data Factory with Azure Monitor | https://learn.microsoft.com/en-us/azure/data-factory/monitor-data-factory |
| Use monitoring reference data for Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/monitor-data-factory-reference |
| Configure monitoring for Azure Data Factory integration runtimes | https://learn.microsoft.com/en-us/azure/data-factory/monitor-integration-runtime |
| Monitor integration runtimes in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/monitor-integration-runtime |
| Set up ADF diagnostic logs via Azure Monitor REST API | https://learn.microsoft.com/en-us/azure/data-factory/monitor-logs-rest |
| Monitor managed virtual network integration runtimes in ADF | https://learn.microsoft.com/en-us/azure/data-factory/monitor-managed-virtual-network-integration-runtime |
| Monitor Azure VMs hosting self-hosted integration runtime | https://learn.microsoft.com/en-us/azure/data-factory/monitor-shir-in-azure |
| Monitor SSIS operations in ADF using Azure Monitor | https://learn.microsoft.com/en-us/azure/data-factory/monitor-ssis |
| Visually monitor pipelines and activities in ADF | https://learn.microsoft.com/en-us/azure/data-factory/monitor-visually |
| Apply naming rules for Azure Data Factory artifacts | https://learn.microsoft.com/en-us/azure/data-factory/naming-rules |
| Parameterize linked services in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services |
| Parameterize mapping data flows in Azure Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/parameters-data-flow |
| Change user passwords in Workflow Orchestration Manager | https://learn.microsoft.com/en-us/azure/data-factory/password-change-airflow |
| Create SAP CDC linked services and datasets | https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-prepare-linked-service-source-dataset |
| Configure prerequisites for the SAP CDC connector | https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-prerequisites-configuration |
| Prepare self-hosted IR for SAP CDC workloads | https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-shir-preparation |
| Configure autoupdate and version management for self-hosted IR | https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-auto-update |
| Configure self-hosted integration runtime as proxy for Azure-SSIS IR | https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-run

…(truncated)