azure-data-factory

name	azure-data-factory
description	Expert knowledge for Azure Data Factory development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when designing ADF pipelines, Mapping Data Flows, SHIR/SSIS IR, secure VNet/Private Link setups, or CI/CD deployments, and other Azure Data Factory related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure Databricks (use azure-databricks), Azure Stream Analytics (use azure-stream-analytics), Azure Data Explorer (use azure-data-explorer).
compatibility	Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata	{"generated_at":"2026-04-12","generator":"docs2skills/1.0.0"}

Azure Data Factory Skill

This skill provides expert guidance for Azure Data Factory. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

• Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
• Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Lines	Description
Troubleshooting	L37-L90	Diagnosing and fixing ADF issues: connector/format errors, copy & data flow performance, pipelines/triggers, SHIR/SSIS IR problems, security/access, and interpreting failure logs.
Best Practices	L91-L113	Performance, DataOps, and reliability best practices for ADF: tuning data flows/copy, optimizing sources/sinks/IR, handling schema drift/errors, and migrating data (HDFS, Netezza, S3) to Azure.
Decision Making	L114-L134	Guidance on ADF architecture, runtimes, connector upgrades, cost optimization, and migration/modernization of pipelines, SSIS, Synapse, and data warehouses to Azure/Fabric
Architecture & Design Patterns	L135-L145	Patterns and architectures for ADF: efficient mapping data flows, metadata‑driven copy, SSIS IR with SQL MI/failover/VNet, Cosmos DB migration, and SAP CDC design.
Limits & Quotas	L146-L150	Info on ADF connector lifecycle stages and timelines, plus how reservation discounts work for Mapping Data Flows and how they affect cost and capacity planning
Security	L151-L181	Securing Data Factory with identity, encryption, Key Vault, and Azure Policy, plus network controls like VNets, Private Link, firewalls, private endpoints, and secure runtimes (Azure-SSIS, self-hosted).
Configuration	L182-L295	Configuring ADF artifacts and runtime: pipelines, activities, data flows, triggers, copy behavior, formats, IRs (Azure/self-hosted/SSIS), monitoring, logging, parameters, and DevOps setup.
Integrations & Coding Patterns	L296-L482	Connector how-tos, patterns, and activities for copying/transforming data between ADF and many services (databases, SaaS apps, files, Fabric, Databricks, SSIS, ML, Synapse) using data flows and SDKs.
Deployment	L483-L496	CI/CD and deployment for ADF: ARM/linked templates, Azure DevOps pipelines, hotfix flows, pre/post scripts, IR automation, SSIS job migration, and networked IR migration.

Troubleshooting

Topic	URL
Troubleshoot Azure Data Factory change data capture	https://learn.microsoft.com/en-us/azure/data-factory/change-data-capture-troubleshoot
Troubleshoot CI/CD, Azure DevOps, and GitHub integration for ADF	https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide
Resolve common Azure Data Factory connector upgrade issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-deprecation-frequently-asked-questions
Resolve Azure Data Factory Amazon S3 connector errors	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-amazon-simple-storage-service
Fix Azure Blob Storage connector issues in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-blob-storage
Troubleshoot Azure Cosmos DB connectors in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-cosmos-db
Resolve Azure Data Explorer connector problems in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-explorer
Troubleshoot Azure Data Lake Storage connectors in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-lake
Resolve Azure Files connector errors in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-files
Troubleshoot Azure Table Storage connector in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-table-storage

Best Practices

Topic	URL
Apply DataOps practices to Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/apply-dataops
Use column patterns in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-column-pattern
Reuse logic with flowlets in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-flowlet
Tune mapping data flow performance in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance
Optimize sink performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sinks
Optimize source performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sources
Improve transformation performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-transformations
Handle schema drift in Data Factory mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-schema-drift
Tune integration runtime performance for data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance
Tune Azure-SSIS integration runtime for high performance	https://learn.microsoft.com/en-us/azure/data-factory/configure-azure-ssis-integration-runtime-performance
Optimize Azure Data Factory copy activity performance

Decision Making

Topic	URL
Use automatic connector upgrades in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/automatic-connector-upgrade
Interpret pricing examples for ADF integration runtimes	https://learn.microsoft.com/en-us/azure/data-factory/better-understand-different-integration-runtime-charges
Choose the right integration runtime architecture	https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configuration
Choose compute environments for Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/compute-linked-services
Choose Azure Data Factory connector release stages	https://learn.microsoft.com/en-us/azure/data-factory/connector-release-stages-and-timelines
Use connector upgrade advisor for ADF and Synapse	https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-advisor
Decide how and when to upgrade ADF connectors	https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-guidance
Buy and use ADF data flow reserved capacity for cost savings	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-reserved-capacity-overview
Plan data lake and EDW migration to Azure with ADF	https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-overview
Answer common Azure Data Factory usage questions	https://learn.microsoft.com/en-us/azure/data-factory/frequently-asked-questions
Assess Azure Data Factory pipelines for Fabric migration

Architecture & Design Patterns

Topic	URL
Design efficient pipelines using mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-pipelines
Configure Azure-SSIS IR with SQL failover groups	https://learn.microsoft.com/en-us/azure/data-factory/configure-bcdr-azure-ssis-integration-runtime
Build metadata-driven large-scale copy pipelines	https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven
Migrate Azure SQL schemas to Azure Cosmos DB with ADF	https://learn.microsoft.com/en-us/azure/data-factory/how-to-sqldb-to-cosmosdb
Use Azure SQL Managed Instance with Azure-SSIS IR	https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-sql-managed-instance-with-ir
Decide when and how to join Azure-SSIS integration runtime to a virtual network	https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network
Understand architecture of SAP CDC in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-introduction-architecture

Limits & Quotas

Topic	URL
Understand reservation discounts for ADF data flows	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-understand-reservation-charges

Security

Topic	URL
Allow Azure Integration Runtime IP ranges through firewalls	https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses
Configure VNet injection for Azure-SSIS IR securely	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-virtual-network-configuration
Configure roles and permissions for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-roles-permissions
Configure outbound FQDN allow lists with Azure Policy for ADF	https://learn.microsoft.com/en-us/azure/data-factory/configure-outbound-allow-list-azure-policy
Use Azure credentials and managed identities in ADF	https://learn.microsoft.com/en-us/azure/data-factory/credentials
Plan secure data access strategies for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-access-strategies
Configure Azure Private Link for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-private-link
Configure managed identities for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-service-identity
Security infrastructure and data movement in ADF	https://learn.microsoft.com/en-us/azure/data-factory/data-movement-security-considerations
Enable Microsoft Entra authentication for Azure-SSIS integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/enable-aad-authentication-azure-ssis-ir

Configuration

Topic	URL
Define and use global parameters in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/author-global-parameters
Configure connections and global settings in ADF management hub	https://learn.microsoft.com/en-us/azure/data-factory/author-management-hub
Configure express virtual network injection for Azure-SSIS integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-express-virtual-network-injection
Manage SSIS packages with Azure-SSIS package store	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-package-store
Configure standard virtual network injection for Azure-SSIS integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-standard-virtual-network-injection
Reference built-in and preinstalled components on Azure-SSIS integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/built-in-preinstalled-components-ssis-integration-runtime
Define and configure datasets in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services
Configure integration runtimes in Data Factory and Synapse	https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime
Define and configure linked services in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-linked-services
Override default ARM template parameters for ADF CI/CD	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters

Integrations & Coding Patterns

Topic	URL
Configure SAP CDC connector to load data into Fabric OneLake	https://learn.microsoft.com/en-us/azure/data-factory/change-data-capture-from-sap-to-onelake-with-azure-data-factory
Use expression builder for ADF mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-expression-builder
Connect Azure Data Factory to Microsoft Purview	https://learn.microsoft.com/en-us/azure/data-factory/connect-data-factory-to-azure-purview
Configure ADF connector for Amazon Marketplace Web Service	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-marketplace-web-service
Integrate Amazon RDS for Oracle with Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-rds-for-oracle
Copy data from Amazon RDS for SQL Server with ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-rds-for-sql-server
Copy data from Amazon Redshift using ADF/Synapse	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-redshift
Copy data from S3-compatible storage using ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-s3-compatible-storage
Copy and transform data in Amazon S3 with ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-simple-storage-service
Transform AppFigures data with ADF Data Flows	https://learn.microsoft.com/en-us/azure/data-factory/connector-appfigures

Deployment

Topic	URL
Implement CI/CD for Azure Data Factory pipelines	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery
Automate ADF CI/CD with Azure Pipelines	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-automate-azure-pipelines
Use hotfix production environments for ADF CI/CD	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-hotfix-environment
Automate publishing for Azure Data Factory CI/CD	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements
Deploy Azure Data Factory with linked ARM templates	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-linked-templates
Manually promote ADF ARM templates across environments	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-manual-promotion
Run pre- and post-deployment scripts for ADF CI/CD	https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-sample-script
Deploy linked ARM templates via Azure DevOps for ADF	https://learn.microsoft.com/en-us/azure/data-factory/deploy-linked-arm-templates-with-vsts
Migrate SSIS SQL Agent jobs to Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/how-to-migrate-ssis-job-ssms
Automate self-hosted integration runtime installation with PowerShell	https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-automation-scripts