azure-hdinsight

name	azure-hdinsight
description	Expert knowledge for Azure HDInsight development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when working with HDInsight Spark/Hive/Kafka/HBase clusters, Ambari, VNet networking, or Azure SQL/Cosmos DB integration, and other Azure HDInsight related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure Databricks (use azure-databricks), Azure Stream Analytics (use azure-stream-analytics).
compatibility	Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata	{"generated_at":"2026-04-05","generator":"docs2skills/1.0.0"}

Azure HDInsight Skill

This skill provides expert guidance for Azure HDInsight. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

• Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
• Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Lines	Description
Troubleshooting	L37-L132	Diagnosing and fixing HDInsight cluster issues: creation/auth, networking, storage, Ambari/HDFS/Hive/HBase/Kafka/Spark/YARN problems, performance, disk/CPU, and known error codes/workarounds.
Best Practices	L133-L174	Best practices for designing, securing, monitoring, scaling, and tuning HDInsight clusters and workloads (Hadoop, Spark, Hive, HBase, Kafka), including storage, migration, and performance optimization.
Decision Making	L175-L199	Planning and migration guidance for HDInsight: sizing and performance, choosing storage/VMs/tools, upgrading versions/components, and moving Hadoop, HBase, Kafka, and configs to newer clusters.
Architecture & Design Patterns	L200-L214	HDInsight cluster architecture, security/VNet design, HA/DR and business continuity patterns, migration from on-prem Hadoop, shared storage, streaming (Spark/YARN), and Oozie-based pipelines.
Limits & Quotas	L215-L222	Guidance on HDInsight capacity limits: log size/retention, supported cluster node sizes, external metastore constraints, and requesting/managing CPU core quota increases.
Security	L223-L266	Securing HDInsight clusters: identity and access (Entra, LDAP, Ranger, RBAC), network isolation (NSG, Private Link), TLS/encryption, Kafka/Hive/Spark security, and security best practices.
Configuration	L267-L323	Configuring and tuning HDInsight clusters: networking/VPN, Ambari/Hive/Spark/HBase settings, autoscale, monitoring/logging, SSH/Jupyter/VS Code access, and script-based customizations.
Integrations & Coding Patterns	L324-L391	Patterns and code samples for integrating HDInsight (Hive, Spark, Kafka, HBase, MapReduce, Sqoop) with tools, SDKs, REST/CLI, and external services like SQL, Cosmos DB, Power BI, IoT, and Synapse
Deployment	L392-L405	Creating, configuring, migrating, and automating HDInsight clusters (Hadoop, HBase, Kafka) using portal, CLI, PowerShell, ARM/REST, Data Factory, Marketplace, AMA, and runbooks

Troubleshooting

Topic	URL
Address reliability issues on older HDInsight images	https://learn.microsoft.com/en-us/azure/hdinsight/cluster-reliability-issues
Fix component version validation errors in HDInsight ARM templates	https://learn.microsoft.com/en-us/azure/hdinsight/component-version-validation-error-arm-templates
Troubleshoot Azure HDInsight cluster creation errors	https://learn.microsoft.com/en-us/azure/hdinsight/create-cluster-error-dictionary
Troubleshoot authentication issues for secure HDInsight clusters	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/domain-joined-authentication-issues
Run diagnostic script when HDInsight cluster creation fails with DomainNotFound	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/sample-script
Fix DomainNotFound errors during HDInsight cluster creation	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/troubleshoot-domainnotfound
Fix Apache Ambari directory alerts in HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-ambari-troubleshoot-directory-alerts
Troubleshoot Ambari UI down hosts and services in HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-ambari-troubleshoot-down-hosts-services
Fix Apache Ambari UI 502 errors on HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-ambari-troubleshoot-fivezerotwo-error
Resolve Apache Ambari heartbeat issues in HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-ambari-troubleshoot-heartbeat-issues

Best Practices

Topic	URL
Use Azure Monitor logs for HDInsight availability	https://learn.microsoft.com/en-us/azure/hdinsight/cluster-availability-monitor-logs
Apply cluster management best practices in HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/cluster-management-best-practices
Apply general best practices for HDInsight Enterprise Security	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/general-guidelines
Plan and execute data migration to Azure HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-on-premises-migration-best-practices-data-migration
Apply infrastructure best practices for Azure HDInsight clusters	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-on-premises-migration-best-practices-infrastructure
Implement storage best practices for HDInsight migrations	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-on-premises-migration-best-practices-storage
Optimize HDInsight HBase with Accelerated Writes	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-accelerated-writes
Apply HBase performance advisor recommendations on HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-advisor
Tune Apache Phoenix performance on HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-phoenix-performance
Tune Apache HBase performance on HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/troubleshoot-hbase-performance-issues

Decision Making

Topic	URL
Plan ETL at scale with Azure HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-etl-at-scale
Assess benefits of migrating on-premises Hadoop to Azure HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-on-premises-migration-motivation
Choose HDInsight tools for custom MapReduce jobs	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-run-custom-programs
Choose backup and replication options for HBase	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-backup-replication
Migrate Apache HBase clusters to HDInsight 5.1	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-migrate-hdinsight-5-1
Migrate HBase to HDInsight 5.1 with a new storage account	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-migrate-hdinsight-5-1-new-storage-account
Migrate Apache HBase clusters to a newer HDInsight version	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-migrate-new-version
Migrate HBase to new HDInsight version and storage account	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-migrate-new-version-new-storage-account
Plan HDInsight cluster capacity and performance	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-capacity-planning
Plan migrations for retiring Azure HDInsight components	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-component-retirements-and-action-required

Architecture & Design Patterns

Topic	URL
Use Apache Ambari for HDInsight cluster management	https://learn.microsoft.com/en-us/azure/hdinsight/apache-ambari-usage
Understand HDInsight architecture with Enterprise Security Package	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/apache-domain-joined-architecture
Design architecture for migrating on-premises Hadoop to HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-on-premises-migration-best-practices-architecture
Choose HDInsight business continuity architectures	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-business-continuity-architecture
Study HDInsight high availability and DR case design	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-high-availability-case-study
Understand HDInsight high availability architecture components	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-high-availability-components
Share one Data Lake Storage account across multiple HDInsight clusters	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-multiple-clusters-data-lake-store
Operationalize HDInsight data pipelines with Oozie	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-operationalize-data-pipeline
Design scalable streaming architectures with HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-streaming-at-scale-overview
Azure HDInsight virtual network architecture and resources	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-virtual-network-architecture

Limits & Quotas

Topic	URL
Plan HDInsight log sizes and retention policies	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-log-management
Use supported node configurations for HDInsight clusters	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-supported-node-configuration
Use external metastores and understand HDInsight default metastore limits	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-use-external-metadata-stores
Request and manage HDInsight CPU core quota increases	https://learn.microsoft.com/en-us/azure/hdinsight/quota-increase-request

Security

Topic	URL
Configure managed identity access to Blob storage for HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/configure-azure-blob-storage
Configure double disk encryption for HDInsight data at rest	https://learn.microsoft.com/en-us/azure/hdinsight/disk-encryption
Configure HDInsight clusters with Entra Domain Services integration	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/apache-domain-joined-configure-using-azure-adds
Create and configure HDInsight Enterprise Security Package clusters	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/apache-domain-joined-create-configure-enterprise-security-cluster
Manage users, roles, and security for HDInsight ESP clusters	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/apache-domain-joined-manage
Configure Apache Ranger policies for HBase with ESP	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/apache-domain-joined-run-hbase
Configure Apache Ranger Hive policies in HDInsight ESP	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/apache-domain-joined-run-hive
Set Apache Ranger policies for Kafka with ESP	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/apache-domain-joined-run-kafka
Implement encryption in transit for Azure HDInsight nodes	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/encryption-in-transit
Plan enterprise security options for Azure HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/domain-joined/hdinsight-security-overview

Configuration

Topic	URL
Configure Ambari Web UI auto-logout timeout in HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/ambari-web-ui-auto-logout
Connect HDInsight clusters to on-premises networks with VPN and DNS	https://learn.microsoft.com/en-us/azure/hdinsight/connect-on-premises-network
Configure HBase cluster replication in Azure VNets	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-replication
Use HBCK2 to repair HBase on HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/how-to-use-hbck2-tool
Check HDInsight 4.0 open-source component versions	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-40-component-versioning
Check HDInsight 5.x open-source component versions	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-5x-component-versioning
Manage HDInsight clusters using Azure CLI commands	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-administer-use-command-line
Automate HDInsight cluster management with PowerShell	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-administer-use-powershell
Configure and use empty edge nodes in HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-apps-use-edge-node
Configure HDInsight Autoscale policies and limits	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-autoscale-clusters
Tune HDInsight cluster settings using Ambari	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-changing-configs-via-ambari

Integrations & Coding Patterns

Topic	URL
Configure Ambari email alerts with SendGrid in HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/apache-ambari-email
Stream from Kafka to Azure Cosmos DB with Spark	https://learn.microsoft.com/en-us/azure/hdinsight/apache-kafka-spark-structured-streaming-cosmosdb
Execute common HDInsight tasks with Azure CLI samples	https://learn.microsoft.com/en-us/azure/hdinsight/azure-cli-samples
Connect Excel to HDInsight Hadoop via Power Query	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-connect-excel-power-query
Query HDInsight Hive from Java using JDBC	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-connect-hive-jdbc-driver
Visualize HDInsight Hive data in Power BI via ODBC	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-connect-hive-power-bi
Integrate C# UDFs with Hive and Pig on HDInsight	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-hive-pig-udf-dotnet-csharp
Call WebHCat REST API for Hive with Curl	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-use-hive-curl
Submit Hive jobs using HDInsight .NET SDK	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-use-hive-dotnet-sdk
Run HDInsight Hive queries with Azure PowerShell	https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-use-hive-powershell
Use Visual Studio Data Lake tools for Hive on HDInsight

Deployment

Topic	URL
Migrate HDInsight monitoring to Azure Monitor Agent (AMA)	https://learn.microsoft.com/en-us/azure/hdinsight/azure-monitor-agent
Deploy HBase clusters in Azure Virtual Networks	https://learn.microsoft.com/en-us/azure/hdinsight/hbase/apache-hbase-provision-vnet
Publish Azure HDInsight applications to Azure Marketplace	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-apps-publish-applications
Operationalize on-demand HDInsight Hadoop clusters with Data Factory	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-adf
Deploy HDInsight clusters using ARM templates	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-arm-templates
Provision HDInsight 4.0 clusters using Azure CLI	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-azure-cli
Create Linux HDInsight clusters using PowerShell scripts	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-azure-powershell
Create HDInsight clusters via Azure REST and ARM templates	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-curl-rest
Create Linux-based HDInsight clusters via Azure portal	https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-portal
Migrate HDInsight Kafka clusters using MirrorMaker 2	https://learn.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-mirror-maker-2