databricks-multi-env-setup

name	databricks-multi-env-setup
description	Configure Databricks across development, staging, and production environments. Use when setting up multi-environment deployments, configuring per-environment secrets, or implementing environment-specific Databricks configurations. Trigger with phrases like "databricks environments", "databricks staging", "databricks dev prod", "databricks environment setup", "databricks config by env".
allowed-tools	Read, Write, Edit, Bash(databricks:*), Bash(terraform:*)
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>

Databricks Multi-Environment Setup

Overview

Configure Databricks across development, staging, and production environments.

Prerequisites

• Multiple Databricks workspaces (or single workspace with isolation)
• Secret management solution (Databricks Secrets, Azure Key Vault, etc.)
• CI/CD pipeline configured
• Unity Catalog for cross-workspace governance

Environment Strategy

Environment	Purpose	Workspace	Data	Permissions
Development	Local dev, experimentation	Shared dev workspace	Sample/synthetic	Broad access
Staging	Integration testing, UAT	Dedicated staging	Prod clone (masked)	Team access
Production	Live workloads	Dedicated prod	Real data	Restricted

Instructions

Step 1: Asset Bundle Environment Configuration

# databricks.yml
bundle:
  name: data-platform

variables:
  catalog:
    description: Unity Catalog name
    default: dev_catalog
  schema_prefix:
    description: Schema prefix for isolation
    default: dev
  warehouse_size:
    description: SQL Warehouse size
    default: "2X-Small"
  cluster_size:
    description: Default cluster workers
    default: 1

include:
  - resources/*.yml

workspace:
  host: ${DATABRICKS_HOST}

targets:
  # Development - personal workspaces
  dev:
    default: true
    mode: development
    variables:
      catalog: dev_catalog
      schema_prefix: "${workspace.current_user.short_name}"
      warehouse_size: "2X-Small"
       
    
       

  
  
     
    
       
       
       
       
    
       
       
    
       
    
        
         
        
         

  
  
     
    
       
       
       
       
    
       
       
    
       
    
        
         
        
         
        
         

Step 2: Environment-Specific Resources

# resources/jobs.yml
resources:
  jobs:
    etl_pipeline:
      name: "${bundle.name}-etl-${bundle.target}"

      # Environment-specific schedule
      schedule:
        quartz_cron_expression: >-
          ${bundle.target == "prod" ? "0 0 6 * * ?" :
            bundle.target == "staging" ? "0 0 8 * * ?" : null}
        timezone_id: "America/New_York"
        pause_status: ${bundle.target == "dev" ? "PAUSED" : "UNPAUSED"}

      # Environment-specific notifications
      email_notifications:
        on_failure: ${bundle.target == "prod" ?
          ["[e-mail]", "[e-mail]"] :
          ["[e-mail]"]}

      # Environment-specific cluster sizing
      job_clusters:
        - job_cluster_key: etl_cluster
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            node_type_id: >-
              ${bundle.target == "prod" ? "Standard_DS4_v2" : "Standard_DS3_v2"}
            num_workers: ${var.cluster_size}
            autoscale:
              min_workers:      
                    

            
            
                              

Step 3: Unity Catalog Cross-Environment Setup

-- Create environment-specific catalogs
CREATE CATALOG IF NOT EXISTS dev_catalog;
CREATE CATALOG IF NOT EXISTS staging_catalog;
CREATE CATALOG IF NOT EXISTS prod_catalog;

-- Grant cross-environment read access for data lineage
GRANT USAGE ON CATALOG prod_catalog TO `staging-service-principal`;
GRANT SELECT ON CATALOG prod_catalog TO `staging-service-principal`;

-- Set up data sharing between environments
CREATE SHARE IF NOT EXISTS prod_to_staging;
ALTER SHARE prod_to_staging ADD SCHEMA prod_catalog.reference;

-- Create recipient for staging workspace
CREATE RECIPIENT IF NOT EXISTS staging_workspace
  USING IDENTITY ('staging-workspace-identity');

GRANT SELECT ON SHARE prod_to_staging TO RECIPIENT staging_workspace;

Step 4: Secret Management by Environment

# src/config/secrets.py
from databricks.sdk import WorkspaceClient
import os

class EnvironmentSecrets:
    """Environment-aware secret management."""

    def __init__(self, environment: str = None):
        self.environment = environment or os.getenv("ENVIRONMENT", "dev")
        self.w = WorkspaceClient()
        self._secret_scope = f"{self.environment}-secrets"

    def get_secret(self, key: str) -> str:
        """Get secret for current environment."""
        # In notebooks, use dbutils
        # return dbutils.secrets.get(scope=self._secret_scope, key=key)

        # Via API (for testing)
        return self.w.secrets.get_secret(
            scope=self._secret_scope,
            key=key
        ).value

    def get_database_url(self) -> str:
        """Get environment-specific database URL."""
        return self.get_secret("database_url")

    def get_api_key(self, service: str) -> str:
        
         .get_secret()

Step 5: Environment Detection

# src/config/environment.py
import os
from dataclasses import dataclass
from typing import Optional

@dataclass
class EnvironmentConfig:
    """Environment-specific configuration."""
    name: str
    catalog: str
    schema_prefix: str
    is_production: bool
    debug_enabled: bool
    max_cluster_size: int

def detect_environment() -> EnvironmentConfig:
    """Detect current environment from context."""
    # From environment variable (set by Asset Bundles)
    env = os.getenv("ENVIRONMENT", "dev")

    # Or from Databricks tags
    # spark.conf.get("spark.databricks.tags.Environment")

    configs = {
        "dev": EnvironmentConfig(
            name="dev",
            catalog="dev_catalog",
            schema_prefix="dev",
            is_production=False,
            debug_enabled=True,
            max_cluster_size=4,
        ),
        "staging": EnvironmentConfig(
            name="staging",
            catalog="staging_catalog",
            schema_prefix="staging",
            is_production=False,
            debug_enabled=True,
            max_cluster_size=8,
        ),
        "prod": EnvironmentConfig(
            name="prod",
            catalog="prod_catalog",
            schema_prefix=,
            is_production=,
            debug_enabled=,
            max_cluster_size=,
        ),
    }

     configs.get(env, configs[])


env = detect_environment()
full_table_name = 

Step 6: Environment Promotion Pipeline

# scripts/promote_to_prod.py
from databricks.sdk import WorkspaceClient
import subprocess

def promote_to_production(
    version_tag: str,
    dry_run: bool = True,
) -> dict:
    """
    Promote code from staging to production.

    Steps:
    1. Verify staging tests passed
    2. Tag release in git
    3. Deploy to production
    4. Run smoke tests
    5. Enable schedules
    """
    results = {"steps": []}

    # 1. Verify staging tests
    print("Verifying staging tests...")
    staging_result = subprocess.run(
        ["databricks", "bundle", "run", "-t", "staging", "integration-tests"],
        capture_output=True
    )
    if staging_result.returncode != 0:
        raise Exception("Staging tests failed")
    results["steps"].append({"stage": "verify_staging", "status": "passed"})

    # 2. Tag release
    print(f"Tagging release {version_tag}...")
    if not dry_run:
        subprocess.run(["git", "tag", version_tag])
        subprocess.run(["git", "push", "origin", version_tag])
    results["steps"].append({"stage": , :    dry_run  })

    
    ()
      dry_run:
        subprocess.run([, , , , ])
    results[].append({: , :    dry_run  })

    
    ()
      dry_run:
        subprocess.run([, , , , , ])
    results[].append({: , :    dry_run  })

     results

 __name__ == :
     argparse
    parser = argparse.ArgumentParser()
    parser.add_argument(, required=)
    parser.add_argument(, action=)
    args = parser.parse_args()

    result = promote_to_production(args.version, args.dry_run)
    (result)

Output

• Multi-environment Asset Bundle configuration
• Environment-specific job settings
• Cross-environment Unity Catalog setup
• Secret management by environment
• Promotion pipeline

Error Handling

Issue	Cause	Solution
Wrong environment	Missing env var	Check ENVIRONMENT variable
Secret not found	Wrong scope	Verify scope name matches environment
Permission denied	Missing grants	Add Unity Catalog grants
Config mismatch	Target override issue	Check bundle target syntax

Examples

Quick Environment Check

env = detect_environment()
print(f"Running in {env.name} environment")
print(f"Using catalog: {env.catalog}")
print(f"Production mode: {env.is_production}")

Terraform Multi-Workspace

# infrastructure/terraform/main.tf
locals {
  environments = {
    dev = {
      workspace_name = "data-platform-dev"
      sku           = "premium"
    }
    staging = {
      workspace_name = "data-platform-staging"
      sku           = "premium"
    }
    prod = {
      workspace_name = "data-platform-prod"
      sku           = "premium"
    }
  }
}

resource "azurerm_databricks_workspace" "workspace" {
  for_each = local.environments

  name                = each.value.workspace_name
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  sku                 = each.value.sku

  tags = {
    environment = each.key
    managed_by  = "terraform"
  }
}

Resources

• Asset Bundles Targets
• Unity Catalog Isolation
• Workspace Federation

Next Steps

For observability setup, see databricks-observability.