databricks-ci-integration

name	databricks-ci-integration
description	Configure Databricks CI/CD integration with GitHub Actions and Asset Bundles. Use when setting up automated testing, configuring CI pipelines, or integrating Databricks deployments into your build process. Trigger with phrases like "databricks CI", "databricks GitHub Actions", "databricks automated tests", "CI databricks", "databricks pipeline".
allowed-tools	Read, Write, Edit, Bash(gh:*), Bash(databricks:*)
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>
compatible-with	claude-code, codex, openclaw
tags	["saas","databricks","deployment","testing","ci-cd"]

Databricks CI Integration

Overview

Automate Databricks deployments with Declarative Automation Bundles (DABs) and GitHub Actions. Covers bundle validation, unit testing PySpark transforms locally, deploying to staging on PR, production on merge, and integration testing against live workspaces. Uses databricks/setup-cli action and OAuth M2M for secure CI auth.

Prerequisites

• Databricks workspace with service principal (OAuth M2M)
• Asset Bundle (databricks.yml) configured
• GitHub repo with Actions enabled
• GitHub environment secrets: DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET

Instructions

Step 1: GitHub Actions — Validate and Test on PR

# .github/workflows/databricks-ci.yml
name: Databricks CI

on:
  pull_request:
    paths: ['src/**', 'resources/**', 'databricks.yml', 'tests/**']

jobs:
  validate-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          pip install pytest pyspark delta-spark databricks-sdk
          pip install -e .  # If using pyproject.toml

      - name: Run unit