databricks-ci-integration

name	databricks-ci-integration
description	Configure Databricks CI/CD integration with GitHub Actions and Asset Bundles. Use when setting up automated testing, configuring CI pipelines, or integrating Databricks deployments into your build process. Trigger with phrases like "databricks CI", "databricks GitHub Actions", "databricks automated tests", "CI databricks", "databricks pipeline".
allowed-tools	Read, Write, Edit, Bash(gh:*), Bash(databricks:*)
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>

Databricks CI Integration

Overview

Set up CI/CD pipelines for Databricks using GitHub Actions and Asset Bundles.

Prerequisites

• GitHub repository with Actions enabled
• Databricks workspace with service principal
• Asset Bundles project structure

Instructions

Step 1: Configure Service Principal

# Create service principal in Databricks
databricks service-principals create --json '{
  "display_name": "GitHub Actions CI",
  "active": true
}'

# Note the application_id returned

# Create OAuth secret
databricks service-principal-secrets create \
  --service-principal-id <application_id>

# Grant permissions to service principal
databricks permissions update workspace --json '{
  "access_control_list": [{
    "service_principal_name": "<application_id>",
    "permission_level": "CAN_MANAGE"
  }]
}'

Step 2: Configure GitHub Secrets

# Set GitHub secrets
gh secret set DATABRICKS_HOST --body "https://adb-1234567890.1.azuredatabricks.net"
gh secret set DATABRICKS_CLIENT_ID --body "your-client-id"
gh secret set DATABRICKS_CLIENT_SECRET --body "your-client-secret"

# For staging/prod environments
gh secret set DATABRICKS_HOST_STAGING --body "https://staging.azuredatabricks.net"
gh secret set DATABRICKS_HOST_PROD --body "https://prod.azuredatabricks.net"

Step 3: Create GitHub Actions Workflow

# .github/workflows/databricks-ci.yml
name: Databricks CI/CD

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
  DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
  DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install dependencies
        run: |
          pip install databricks-cli databricks-sdk pytest

      - name: Validate Asset 
           

          
            

  
     
       
     
     
    
         
    
        

         
         
        
           

          
           

          
         
          
         
  
     [, ]
       
     
    
       
         
    
         
    
        

         
         
        
           

          
           

          
         
         
         

Step 4: PR Validation Workflow

# .github/workflows/pr-validation.yml
name: PR Validation

on:
  pull_request:
    branches: [main, develop]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install ruff mypy pytest pytest-cov databricks-sdk

      - name: Lint with ruff
        run: ruff check src/

      - name: Type check with mypy
        run: mypy src/ --ignore-missing-imports

      - name: Run tests  
            

         
         
        
           

  
     
    
         
         
         
    
        

          
           

            
         
           
         

Step 5: Nightly Test Workflow

# .github/workflows/nightly-tests.yml
name: Nightly Tests

on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily
  workflow_dispatch:

jobs:
  integration-tests:
    runs-on: ubuntu-latest
    env:
      DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST_STAGING }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
      DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: pip install databricks-cli

      - name: Run full integration test suite
        run: |
          databricks bundle deploy -t staging
          RUN_ID=$(databricks bundle run -t staging full-integration-tests | jq -r '.run_id')
          databricks runs get --run-id $RUN_ID --wait

      - name:   
         
         
          
         
         
        
           
           

          
         
         
        
           
           
        
             

Output

• Automated test pipeline
• PR checks configured
• Staging deployment on merge to develop
• Production deployment on merge to main

Error Handling

Issue	Cause	Solution
Auth failed	Invalid credentials	Regenerate service principal secret
Bundle validation failed	Invalid YAML	Run databricks bundle validate locally
Deployment timeout	Slow cluster startup	Use warm pools or increase timeout
Tests failed	Code regression	Fix code and re-run

Examples

Matrix Testing (Multiple DBR Versions)

jobs:
  test-matrix:
    strategy:
      matrix:
        dbr_version: ['13.3', '14.3', '15.1']
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Test on DBR ${{ matrix.dbr_version }}
        run: |
          databricks bundle deploy -t test-${{ matrix.dbr_version }}
          databricks bundle run -t test-${{ matrix.dbr_version }} tests

Branch Protection Rules

# Set via GitHub API or UI
required_status_checks:
  - "lint-and-test"
  - "bundle-validation"
required_reviews: 1
dismiss_stale_reviews: true

Resources

• Databricks Asset Bundles
• GitHub Actions
• Service Principal Auth

Next Steps

For deployment patterns, see databricks-deploy-integration.