azure-databricks

name	azure-databricks
description	Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Lakehouse/Lakebase, Lakeflow pipelines, Model Serving, or AI Gateway/agents, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory).
compatibility	Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata	{"generated_at":"2026-08-09","generator":"docs2skills/1.0.0"}

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

• Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
• Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Location	Description
Troubleshooting	L37-L165	Diagnosing and fixing Databricks errors, job/compute/Spark performance issues, connector and ingestion failures, SQL/runtime error codes, and debugging ML, Lakeflow, and model serving workloads.
Best Practices	L166-L365	Best-practice guidance for Databricks architecture, governance, performance, cost, streaming, Lakehouse/RAG/AI apps, Lakeflow pipelines, ML/LLM serving, and SQL/Spark optimization.
Decision Making	L366-L470	Guides for choosing Azure Databricks tiers, compute, networking, AI/ML features, and migration paths (Runtime, Unity Catalog, Lakeflow, Model Serving, connectors, and cost optimization).
Architecture & Design Patterns	architecture-patterns.md	Architectural blueprints and design patterns for Databricks: DR/HA, networking, storage, medallion, streaming, Lakeflow, AI Gateway, agents, MLOps, security, scaling, and data modeling.
Limits & Quotas	limits-quotas.md	Limits, quotas, and constraints for Databricks compute, AI/model serving, Lakeflow pipelines, connectors/ingestion, Unity Catalog, dashboards, notebooks, and real-time/streaming features.
Security	security.md	Securing Azure Databricks and Lakebase: identity and access control (RBAC/ABAC/Unity Catalog), encryption and networking, secrets and OAuth, compliance controls, and secure integrations/connectors.
Configuration	configuration.md	Configuring Azure Databricks and Unity Catalog: account/workspace settings, compute, networking, storage, security/governance, AI/ML/GenAI features, Lakeflow, Lakebase, connectors, CLI, and SQL behavior.
Integrations & Coding Patterns	integrations.md	Patterns and APIs for integrating Azure Databricks with agents, AI/BI, external data systems, IDEs/CLIs, Lakehouse Federation, streaming, and SQL/PySpark functions and connectors.
Deployment	deployment.md	Deploying and migrating Azure Databricks workspaces, apps, agents, pipelines, dashboards, and Lakebase/Lakeflow projects, plus CI/CD, IaC, Unity Catalog upgrades, and regional release/availability details

Troubleshooting

Topic	URL
Reference Databricks diagnostic audit log events	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/audit-logs
Troubleshoot Databricks MLflow Agent Evaluation issues	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-evaluation/troubleshooting
Debug custom code agents on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/debug-agent
Troubleshoot Azure Databricks compute startup issues	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Diagnose Azure Databricks classic compute termination errors	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UI	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot file events for Unity Catalog external locations	https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/file-events-faq
Troubleshoot common Databricks CLI issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Diagnose and fix Databricks Connect Python issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Diagnose and fix Databricks Connect Scala issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting

Best Practices

Topic	URL
Tag Databricks resources for cost attribution	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags
Use default Databricks policy families to enforce compute best practices	https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply identity federation best practices in Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Apply best practices for serverless Databricks workspaces	https://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Build and refine Knowledge Assistant chatbots on Databricks	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-bricks/knowledge-assistant
Design effective evaluation sets for Databricks agents	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-evaluation/evaluation-set
Load test Databricks Apps agents for QPS limits	https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/load-test-agent-app
Measure RAG performance with retrieval and response metrics	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-assess-performance
Define RAG application quality with evaluation sets	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-define-quality
Set up Databricks infrastructure to measure RAG quality	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-enable-measurement

Decision Making

Topic	URL
Manage and change Azure Databricks subscription tier	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/account
Plan migration from Standard to Premium Databricks tier	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/standard-tier
Decide when to enable Mission Critical add-on for Databricks	https://learn.microsoft.com/en-us/azure/databricks/admin/mission-critical
Decide when to use serverless Databricks workspaces	https://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces
Decide and migrate agents from Model Serving to Databricks Apps	https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/migrate-agent-to-apps
Choose approaches to connect Databricks agents to external services	https://learn.microsoft.com/en-us/azure/databricks/agents/mcp-tools/connect-external
Plan AI governance using Unity AI Gateway	https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/ai-governance
Manage Unity AI Gateway budgets and AI spend controls	https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/budgets
Analyze Unity AI Gateway cost with billing-based observability	https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/cost-observability
Optimize Databricks AI Search costs and usage	https://learn.microsoft.com/en-us/azure/databricks/ai-search/cost-management
Decide and migrate from dbx to Databricks bundles	https://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/dbx/dbx-migrate