azure-data-factory

name	azure-data-factory
description	Expert knowledge for Azure Data Factory development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when building ADF pipelines with connectors, mapping data flows, SSIS IR, SAP CDC, or CI/CD deployments, and other Azure Data Factory related development tasks. Not for Azure Data Explorer (use azure-data-explorer), Azure Synapse Analytics (use azure-synapse-analytics), Azure Stream Analytics (use azure-stream-analytics), Azure Databricks (use azure-databricks).
compatibility	Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata	{"generated_at":"2026-08-09","generator":"docs2skills/1.0.0"}

Azure Data Factory Skill

This skill provides expert guidance for Azure Data Factory. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

• Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
• Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Lines	Description
Troubleshooting	L37-L91	Diagnosing and fixing ADF pipeline, connector, runtime, performance, CDC, security, and Studio issues, plus interpreting errors and using tools/logs to resolve failures.
Best Practices	L92-L113	Performance, DataOps, and reliability best practices for ADF: tuning data flows/copy, optimizing sources/sinks/IR, handling schema drift/errors, and migrating data (HDFS, Netezza, S3) to Azure.
Decision Making	L114-L132	Guidance on cost planning, connector upgrades, integration runtime and compute choices, and migration/modernization of ADF, SSIS, Synapse, and pipelines to Fabric.
Architecture & Design Patterns	L133-L141	Patterns and architectures for ADF: efficient mapping data flows, metadata‑driven copy, SSIS IR with SQL MI/failover/VNet, Cosmos DB migration, and SAP CDC design.
Limits & Quotas	L142-L149	Configuring Until/Wait activity limits and timeouts, understanding data flow reservation discounts, and reviewing ADF service quotas and capacity constraints.
Security	L150-L181	Securing Data Factory with identity, encryption, and network controls: managed identities, Key Vault, Private Link, VNets, firewalls, trusted services, and secure runtimes/on-prem access.
Configuration	L182-L295	Configuring ADF pipelines, triggers, runtimes, connectors, formats, and data flows, including parameters, monitoring, fault tolerance, SAP CDC, SSIS IR, and studio/global settings.
Integrations & Coding Patterns	L296-L463	Integrating ADF with dozens of data sources (SAP, SaaS apps, DBs, files, Fabric), using connectors, mapping data flow functions, SSIS in ADF, templates, and activities to build end‑to‑end pipelines.
Deployment	L464-L477	CI/CD and deployment for ADF: ARM/linked templates, Azure DevOps pipelines, hotfix flows, pre/post scripts, SSIS job migration, and integration runtime install/migration.

Troubleshooting

Topic	URL
Troubleshoot Azure Data Factory change data capture	https://learn.microsoft.com/en-us/azure/data-factory/change-data-capture-troubleshoot
Troubleshoot CI/CD, Azure DevOps, and GitHub integration for ADF	https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide
Resolve common Azure Data Factory connector upgrade issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-deprecation-frequently-asked-questions
Diagnose and fix Azure Data Factory Amazon S3 connector errors	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-amazon-simple-storage-service
Fix Azure Blob Storage connector issues in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-blob-storage
Diagnose Azure Cosmos DB connector issues in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-cosmos-db
Resolve Azure Data Explorer connector problems in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-explorer
Troubleshoot Azure Data Lake Storage connectors in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-lake
Resolve Azure Data Factory Azure Files connector issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-files
Troubleshoot Azure Data Factory Azure Table Storage connector	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-table-storage

Best Practices

Topic	URL
Use column patterns in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-column-pattern
Reuse logic with flowlets in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-flowlet
Tune mapping data flow performance in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance
Optimize sink performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sinks
Optimize source performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sources
Improve transformation performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-transformations
Handle schema drift in Data Factory mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-schema-drift
Tune integration runtime performance for data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance
Tune Azure-SSIS integration runtime for high performance	https://learn.microsoft.com/en-us/azure/data-factory/configure-azure-ssis-integration-runtime-performance
Optimize Azure Data Factory copy activity performance	https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance

Decision Making

Topic	URL
Use automatic connector upgrades in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/automatic-connector-upgrade
Interpret pricing examples for ADF integration runtimes	https://learn.microsoft.com/en-us/azure/data-factory/better-understand-different-integration-runtime-charges
Choose the right integration runtime architecture	https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configuration
Choose compute environments for Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/compute-linked-services
Use connector upgrade advisor for ADF and Synapse	https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-advisor
Decide how and when to upgrade ADF connectors	https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-guidance
Buy and use ADF data flow reserved capacity for cost savings	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-reserved-capacity-overview
Plan data lake and EDW migration to Azure with ADF	https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-overview
Assess Azure Data Factory pipelines for Fabric migration	https://learn.microsoft.com/en-us/azure/data-factory/how-to-assess-your-azure-data-factory-to-fabric-data-factory-migration
Choose and provision Enterprise Edition Azure-SSIS IR	https://learn.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition

Architecture & Design Patterns

Topic	URL
Design efficient pipelines using mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-pipelines
Configure Azure-SSIS IR with SQL failover groups	https://learn.microsoft.com/en-us/azure/data-factory/configure-bcdr-azure-ssis-integration-runtime
Build metadata-driven large-scale copy pipelines	https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven
Migrate Azure SQL schemas to Azure Cosmos DB with ADF	https://learn.microsoft.com/en-us/azure/data-factory/how-to-sqldb-to-cosmosdb
Decide when and how to join Azure-SSIS integration runtime to a virtual network	https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network

Limits & Quotas

Topic	URL
Configure Until activity loops and timeouts in ADF	https://learn.microsoft.com/en-us/azure/data-factory/control-flow-until-activity
Configure Wait activity durations in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/control-flow-wait-activity
Understand reservation discounts for ADF data flows	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-understand-reservation-charges
Review Azure Data Factory limits and quotas from FAQ	https://learn.microsoft.com/en-us/azure/data-factory/frequently-asked-questions

Security

Topic	URL
Allow Azure Integration Runtime IP ranges through firewalls	https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses
Configure VNet injection for Azure-SSIS IR securely	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-virtual-network-configuration
Configure roles and permissions for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-roles-permissions
Configure outbound FQDN allow lists with Azure Policy for ADF	https://learn.microsoft.com/en-us/azure/data-factory/configure-outbound-allow-list-azure-policy
Use Azure credentials and managed identities in ADF	https://learn.microsoft.com/en-us/azure/data-factory/credentials
Plan secure data access strategies for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-access-strategies
Configure Azure Private Link for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-private-link
Configure managed identities for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-service-identity
Security infrastructure and data movement in ADF	https://learn.microsoft.com/en-us/azure/data-factory/data-movement-security-considerations
Enable Microsoft Entra authentication for Azure-SSIS integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/enable-aad-authentication-azure-ssis-ir

Configuration

Topic	URL
Define and use global parameters in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/author-global-parameters
Configure connections and global settings in ADF management hub	https://learn.microsoft.com/en-us/azure/data-factory/author-management-hub
Configure express virtual network injection for Azure-SSIS IR	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-express-virtual-network-injection
Manage SSIS packages with Azure-SSIS package store	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-package-store
Configure standard virtual network injection for Azure-SSIS integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-standard-virtual-network-injection
Use built-in components on Azure-SSIS Integration Runtime	https://learn.microsoft.com/en-us/azure/data-factory/built-in-preinstalled-components-ssis-integration-runtime
Configure integration runtimes for Azure Data Factory and Synapse	https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime
Define and configure linked services in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-linked-services
Configure Azure Data Factory Oracle connector settings	https://learn.microsoft.com/en-us/azure/data-factory/connector-oracle
Configure custom ARM template parameters for ADF CI/CD	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters

Integrations & Coding Patterns

Topic	URL
Configure SAP CDC connector to load data into Fabric OneLake	https://learn.microsoft.com/en-us/azure/data-factory/change-data-capture-from-sap-to-onelake-with-azure-data-factory
Use expression builder for ADF mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-expression-builder
Connect Azure Data Factory to Microsoft Purview	https://learn.microsoft.com/en-us/azure/data-factory/connect-data-factory-to-azure-purview
Configure ADF connector for Amazon Marketplace Web Service	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-marketplace-web-service
Integrate Amazon RDS for Oracle with Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-rds-for-oracle
Copy data from Amazon RDS for SQL Server with ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-rds-for-sql-server
Integrate Azure Data Factory with Amazon Redshift	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-redshift
Configure ADF Amazon S3-compatible storage connector	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-s3-compatible-storage
Copy and transform data in Amazon S3 with ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-simple-storage-service
Transform AppFigures data with ADF Data Flows	https://learn.microsoft.com/en-us/azure/data-factory/connector-appfigures

Deployment

Topic	URL
Set up CI/CD for Azure Data Factory pipelines	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery
Automate ADF CI/CD with Azure Pipelines	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-automate-azure-pipelines
Use hotfix production environments for ADF CI/CD	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-hotfix-environment
Automate CI/CD publishing for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements
Use linked ARM templates in Azure Data Factory deployments	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-linked-templates
Manually promote ADF ARM templates across environments	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-manual-promotion
Run pre- and post-deployment scripts for ADF CI/CD	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-sample-script
Deploy linked ARM templates via Azure DevOps for ADF	https://learn.microsoft.com/en-us/azure/data-factory/deploy-linked-arm-templates-with-vsts
Migrate SSIS SQL Agent jobs to Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/how-to-migrate-ssis-job-ssms
Automate self-hosted integration runtime installation with PowerShell	https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-automation-scripts