databricks-deploy-integration

name	databricks-deploy-integration
description	Deploy Databricks jobs and pipelines with Declarative Automation Bundles. Use when deploying jobs to different environments, managing deployments, or setting up deployment automation. Trigger with phrases like "databricks deploy", "asset bundles", "databricks deployment", "deploy to production", "bundle deploy".
allowed-tools	Read, Write, Edit, Bash(databricks:*)
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","databricks","deployment"]
compatibility	Designed for Claude Code, also compatible with Codex and OpenClaw

[!WARNING] DEPRECATED — to be removed in databricks-pack@2.0.0.

This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-bundle-medic. See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Deploy Integration

Overview

Deploy Databricks jobs, DLT pipelines, and ML models using Declarative Automation Bundles (DABs, formerly Asset Bundles). Bundles provide infrastructure-as-code with databricks.yml defining resources, targets (dev/staging/prod), variables, and permissions. The CLI handles validation, deployment, and lifecycle management.

Prerequisites

• Databricks CLI v0.200+ (databricks --version)
• Workspace access with service principal for automated deploys
• databricks.yml bundle configuration at project root

Instructions

Step 1: Initialize a Bundle

# Create from a template
databricks bundle init

# Available templates:
# - default-python: Python notebook project
# - default-sql: SQL project
# - mlops-stacks: Full MLOps template with feature engineering

Step 2: Configure databricks.yml

# databricks.yml — single source of truth for project deployment
bundle:
  name: sales-etl-pipeline

workspace:
  host: ${DATABRICKS_HOST}

variables:
  catalog:
    description: Unity Catalog name
    default: dev_catalog
  alert_email:
    description: Alert notification email
    default: [e-mail]
  warehouse_size:
    default: "2X-Small"

include:
  - resources/*.yml

targets:
  dev:
    default: true
    mode: development
    # dev mode auto-prefixes resources with [username] and enables debug
    workspace:
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev
    variables:
      catalog: dev_catalog

  staging:
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/staging
    variables:
      catalog: staging_catalog
      alert_email: [e-mail]

  prod:
    mode: production
    # production mode prevents accidental destruction
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/prod
    variables:
      catalog: prod_catalog
      alert_email: [e-mail]
      warehouse_size: "Medium"

Step 3: Define Resources

# resources/jobs.yml
resources:
  jobs:
    daily_etl:
      name: "daily-etl-${bundle.target}"
      max_concurrent_runs: 1
      timeout_seconds: 14400

      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "UTC"

      email_notifications:
        on_failure: ["${var.alert_email}"]

      tasks:
        - task_key: extract
          notebook_task:
            notebook_path: ./src/extract.py
            base_parameters:
              catalog: "${var.catalog}"
          job_cluster_key: etl

        - task_key: transform
          depends_on: [{task_key: extract}]
          notebook_task:
            notebook_path: ./src/transform.py
          job_cluster_key: etl

        - task_key: load
          depends_on: [{task_key: transform}]
          notebook_task:
            notebook_path: ./src/load.py
          job_cluster_key: etl

      job_clusters:
        - job_cluster_key: etl
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            node_type_id: "i3.xlarge"
            autoscale:
              min_workers: 1
              max_workers: 4
            aws_attributes:
              availability: SPOT_WITH_FALLBACK
              first_on_demand: 1

# resources/pipelines.yml (DLT)
resources:
  pipelines:
    dlt_pipeline:
      name: "dlt-pipeline-${bundle.target}"
      target: "${var.catalog}.silver"
      catalog: "${var.catalog}"
      libraries:
        - notebook:
            path: ./src/dlt_pipeline.py
      continuous: false
      development: ${bundle.target == "dev"}

Step 4: Deploy Lifecycle Commands

# Validate — checks YAML syntax, variable resolution, permissions
databricks bundle validate -t staging

# Deploy — creates/updates jobs, uploads notebooks, syncs config
databricks bundle deploy -t staging

# Summary — show what's deployed
databricks bundle summary -t staging

# Run — trigger a specific job/pipeline
databricks bundle run daily_etl -t staging

# Run and wait for completion
databricks bundle run daily_etl -t staging --restart-all-workflows

# Sync — live-reload files during development
databricks bundle sync -t dev --watch

# Destroy — remove all deployed resources (dev only!)
databricks bundle destroy -t dev --auto-approve

Step 5: Promote Staging to Production

# 1. Validate staging is clean
databricks bundle validate -t staging

# 2. Deploy and test on staging
databricks bundle deploy -t staging
RUN=$(databricks bundle run daily_etl -t staging --output json | jq -r '.run_id')
databricks runs get --run-id $RUN | jq '.state.result_state'

# 3. After staging passes, deploy to production
databricks bundle validate -t prod
databricks bundle deploy -t prod

# 4. Verify production deployment
databricks bundle summary -t prod
databricks jobs list --output json | \
  jq '.[] | select(.settings.name | contains("daily-etl-prod"))'

Step 6: Permissions in Bundles

# resources/jobs.yml — add permissions block
resources:
  jobs:
    daily_etl:
      name: "daily-etl-${bundle.target}"
      permissions:
        - group_name: data-engineers
          level: CAN_MANAGE
        - group_name: data-analysts
          level: CAN_VIEW
        - service_principal_name: cicd-service-principal
          level: CAN_MANAGE_RUN

Output

• databricks.yml with multi-target deployment (dev/staging/prod)
• Job and pipeline resources defined as code
• Environment-specific variables (catalog, alerts, sizing)
• Promotion workflow from staging to production
• Permissions managed declaratively in bundle config

Error Handling

Issue	Cause	Solution
bundle validate fails	Invalid YAML or unresolved variable	Check variable definitions and target config
PERMISSION_DENIED on deploy	Service principal lacks workspace access	Add SP to workspace in Account Console
RESOURCE_CONFLICT	Resource name collision across targets	Bundle auto-prefixes in development mode
Cluster quota exceeded	Too many active clusters	Use instance pools or terminate idle clusters
Cannot destroy production	mode: production prevents accidental destroy	This is intentional — remove mode or use --force

Examples

Override Variables per Target

# Override a variable at deploy time
databricks bundle deploy -t prod --var="warehouse_size=Large"

Clean Slate Redeploy (Dev Only)

databricks bundle destroy -t dev --auto-approve
databricks bundle deploy -t dev

Resources

• Declarative Automation Bundles
• Bundle Configuration Reference
• Bundle Resources
• Deployment Modes

Next Steps

For multi-environment setup, see databricks-multi-env-setup.