azure-databricks

name	azure-databricks
description	Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Lakehouse/Lakebase, Lakeflow pipelines, Vector Search/RAG, or model serving, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory).
compatibility	Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata	{"generated_at":"2026-04-19","generator":"docs2skills/1.0.0"}

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

• Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
• Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Location	Description
Troubleshooting	L37-L138	Diagnosing and fixing Databricks errors, job and compute failures, connector/ingestion issues, SQL error codes, and performance/debugging problems across Spark, AI, Lakeflow, and tooling.
Best Practices	L139-L313	End-to-end Databricks best practices for performance, cost, governance, streaming, ML/LLM/RAG, BI, Lakeflow, Vector Search, and operational reliability across Azure Databricks workloads.
Decision Making	L314-L401	Guides for choosing Azure Databricks architectures, compute, runtimes, ML/LLM options, and detailed migration paths (Unity Catalog, Delta, SQL, Connect, MLflow, serverless, Lakebase, and networking).
Architecture & Design Patterns	L402-L444	Architectural blueprints and patterns for Databricks: lakehouse, networking, storage, HA/DR, governance, performance, ML/MLOps, RAG/agents, Lakebase, streaming, and external data access.
Limits & Quotas	limits-quotas.md	Limits, quotas, and constraints for Databricks compute, AI/BI, connectors, Lakeflow, Lakebase, model serving, tokens, data types, and Unity Catalog resources, plus related workarounds.
Security	security.md	Identity, access control, encryption, networking, compliance, and secure integrations for Azure Databricks, Unity Catalog, Lakeflow, Lakebase, apps, and external data sources.
Configuration	configuration.md	Configuring and administering Azure Databricks: accounts, workspaces, security, networking, compute, storage, jobs, ML/serving, Lakehouse/Unity Catalog, Lakeflow, apps, and system-table–based monitoring.
Integrations & Coding Patterns	integrations.md	Patterns and APIs for integrating Databricks with external data systems, tools, and AI/ML frameworks, plus detailed PySpark/SQL function references and Lakehouse Federation/streaming examples.
Deployment	deployment.md	Deploying and operating Databricks apps, agents, models, jobs, and infrastructure using CI/CD, IaC, bundles, serving, Terraform, Git, and region/release planning.

Troubleshooting

Topic	URL
Monitor Genie space activity with audit log queries	https://learn.microsoft.com/en-us/azure/databricks/ai-bi/admin/audit
Troubleshoot Azure Databricks compute startup issues	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Resolve Databricks classic compute termination error codes	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UI	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot Apache Kafka usage on Databricks	https://learn.microsoft.com/en-us/azure/databricks/connect/streaming/kafka/faq
Audit and monitor Delta Sharing activity with Databricks logs	https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/audit-logs
Troubleshoot common Delta Sharing access errors	https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
Troubleshoot common Databricks CLI errors and issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Use Databricks app details for monitoring and troubleshooting	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/view-app-details
Troubleshoot Databricks Connect for Python issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Troubleshoot Databricks Connect for Scala problems

Best Practices

Topic	URL
Use Databricks default compute policy families effectively	https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply identity best practices and migrate to federation	https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Apply best practices for Azure Databricks serverless workspaces	https://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Migrate Databricks library installs from init scripts	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply best practices for Databricks compute policies	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databricks	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hints	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelines	https://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Model business data with Unity Catalog metric views	https://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/basic-modeling
Apply Azure Databricks platform administration best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration

Decision Making

Topic	URL
Manage and change Azure Databricks subscription plans	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/account
Plan migration from Databricks Standard to Premium tier	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/standard-tier
Decide when and how to use serverless workspaces	https://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces
Decide and migrate from dbx to Databricks bundles	https://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/dbx/dbx-migrate
Migrate optimized LLM endpoints to provisioned throughput	https://learn.microsoft.com/en-us/azure/databricks/archive/machine-learning/migrate-provisioned-throughput
Decide when to use Databricks Light runtime	https://learn.microsoft.com/en-us/azure/databricks/archive/runtime/light
Plan migration of Databricks workloads to Spark 3.x	https://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/
Select and manage the default Unity Catalog catalog	https://learn.microsoft.com/en-us/azure/databricks/catalogs/default
Choose appropriate Azure Databricks compute types	https://learn.microsoft.com/en-us/azure/databricks/compute/choose-compute
Decide when and how to use GPU Databricks compute	https://learn.microsoft.com/en-us/azure/databricks/compute/gpu
Decide when and how to use Azure Databricks pools	https://learn.microsoft.com/en-us/azure/databricks/compute/pool-index

Architecture & Design Patterns

Topic	URL
Plan disaster recovery architecture for Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/admin/disaster-recovery
Design and use materialization for Databricks metric views	https://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/materialization
Implement fan-in and fan-out patterns in Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/data-engineering/fan-in-fan-out
Choose patterns for external access to Databricks data	https://learn.microsoft.com/en-us/azure/databricks/external-access/
Build an IDP pipeline with Databricks AI Functions	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/idp-pipeline-tutorial
Build multi-agent orchestrator apps on Databricks	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-apps
Create Genie-based multi-agent systems on Databricks	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-genie
Build non-conversational Databricks AI agents with MLflow	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/non-conversational-agents
Implement AI agent memory on Databricks Model Serving	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/stateful-agents-model-serving
Apply agent system design patterns on Databricks	https://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/agent-system-design-patterns