Skip to main content 首页 创作者 comeonoliver skillshub databricks-multi-env-setup
databricks-multi-env-setup Configure Databricks across development, staging, and production environments.
Use when setting up multi-environment deployments, configuring per-environment secrets,
or implementing environment-specific Databricks configurations.
Trigger with phrases like "databricks environments", "databricks staging",
"databricks dev prod", "databricks environment setup", "databricks config by env".
跳到安装 Skills Marketplace 发现并探索由社区构建的 Agent Skills
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ComeOnOliver/skillshub --skill databricks-multi-env-setup命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
下载 Zip 下载中... name databricks-multi-env-setup description Configure Databricks across development, staging, and production environments.
Use when setting up multi-environment deployments, configuring per-environment secrets,
or implementing environment-specific Databricks configurations.
Trigger with phrases like "databricks environments", "databricks staging",
"databricks dev prod", "databricks environment setup", "databricks config by env".
allowed-tools Read, Write, Edit, Bash(databricks:*), Bash(terraform:*) version 1.0.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> compatible-with claude-code, codex, openclaw tags ["saas","databricks","deployment"]
Databricks Multi-Environment Setup
Overview
Configure Databricks across dev, staging, and production with isolated workspaces (or catalog-level isolation), per-environment secrets, Asset Bundle targets, and Terraform for workspace provisioning. Each environment gets its own credentials, Unity Catalog namespace, and compute policies.
Prerequisites
Databricks account with multiple workspaces (or Premium for catalog-level isolation)
Service principals per environment
Secret management (Databricks Secret Scopes, AWS Secrets Manager, or GCP Secret Manager)
CI/CD pipeline (GitHub Actions, Azure DevOps, etc.)
Environment Strategy
Environment Workspace Catalog Auth Compute Development Shared or dedicated dev_catalogPersonal PAT Single-node, 15min auto-stop Staging Dedicated staging_catalogService principal Production-like, spot instances Production Dedicated prod_catalogService principal (OAuth M2M) Instance pools, auto-scale
Instructions
Step 1: CLI Profiles per Environment
[dev]
host = https://adb-dev-workspace.7 .azuredatabricks.net
token = dapi_dev_token
[staging]
host = https://adb-staging-workspace.7 .azuredatabricks.net
client_id = staging-sp-client-id
client_secret = staging-sp-secret
[production]
host = https://adb-prod-workspace.7 .azuredatabricks.net
client_id = prod-sp-client-id
client_secret = prod-sp-secret
databricks workspace list / --profile staging
databricks clusters list --profile production
Step 2: Asset Bundle Targets
bundle:
name: data-platform
variables:
catalog:
description: Unity Catalog for this environment
default: dev_catalog
alert_email:
default: dev@company.com
cluster_size:
default: "2X-Small"
targets:
dev:
default: true
mode: development
workspace:
host: https://adb-dev-workspace.7.azuredatabricks.net
root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev
variables:
catalog: dev_catalog
staging:
workspace:
host: https://adb-staging-workspace.7.azuredatabricks.net
root_path: /Shared/.bundle/${bundle.name}/staging
variables:
catalog: staging_catalog
alert_email: staging-alerts@company.com
prod:
mode: production
workspace:
host: https://adb-prod-workspace.7.azuredatabricks.net
root_path: /Shared/.bundle/${bundle.name}/prod
variables:
catalog: prod_catalog
alert_email: oncall@company.com
cluster_size: "Medium"
Step 3: Per-Environment Secret Scopes
for env in dev staging prod; do
databricks secrets create-scope "${env} -secrets" --profile $env
databricks secrets put-secret "${env} -secrets" db-password --profile $env
databricks secrets put-secret "${env} -secrets" api-key --profile $env
done
import os
env = os.getenv("ENVIRONMENT" , "dev" )
db_password = dbutils.secrets.get(scope=f"{env} -secrets" , key="db-password" )
api_key = dbutils.secrets.get(scope=f"{env} -secrets" , key="api-key" )
Step 4: Environment-Aware Python Config
from dataclasses import dataclass
import os
@dataclass
class DatabricksEnvConfig :
host: str
catalog: str
secret_scope: str
debug: bool
max_retries: int
timeout_seconds: int
CONFIGS = {
"dev" : DatabricksEnvConfig(
host=os.getenv("DATABRICKS_HOST_DEV" , "" ),
catalog="dev_catalog" ,
secret_scope="dev-secrets" ,
debug=True ,
max_retries=3 ,
timeout_seconds=30 ,
),
"staging" : DatabricksEnvConfig(
host=os.getenv("DATABRICKS_HOST_STAGING" , "" ),
catalog="staging_catalog" ,
secret_scope="staging-secrets" ,
debug=False ,
max_retries=3 ,
timeout_seconds=60 ,
),
"prod" : DatabricksEnvConfig(
host=os.getenv("DATABRICKS_HOST_PROD" , "" ),
catalog="prod_catalog" ,
secret_scope="prod-secrets" ,
debug=False ,
max_retries=5 ,
timeout_seconds=120 ,
),
}
def get_config () -> DatabricksEnvConfig:
env = os.getenv("ENVIRONMENT" , "dev" )
config = CONFIGS.get(env)
if not config:
raise ValueError(f"Unknown environment: {env} " )
if not config.host:
raise ValueError(f"DATABRICKS_HOST_{env.upper()} not set" )
return config
Step 5: CI/CD with Environment Secrets
name: Deploy Pipeline
on:
push:
branches: [main ]
jobs:
deploy-staging:
runs-on: ubuntu-latest
environment: staging
steps:
- uses: actions/checkout@v4
- uses: databricks/setup-cli@main
- run: databricks bundle deploy -t staging
env:
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
deploy-production:
needs: deploy-staging
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: databricks/setup-cli@main
- run: databricks bundle deploy -t prod
env:
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST_PROD }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID_PROD }}
DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET_PROD }}
Step 6: Terraform for Workspace Provisioning (Optional) # terraform/main.tf
resource "databricks_workspace" "staging" {
provider = databricks.accounts
workspace_name = "data-platform-staging"
aws_region = "us-east-1"
pricing_tier = "PREMIUM"
deployment_name = "data-platform-staging"
managed_services_customer_managed_key_id = var.cmk_id
}
resource "databricks_catalog" "staging" {
provider = databricks.staging
name = "staging_catalog"
comment = "Staging environment catalog"
}
resource "databricks_schema" "staging_bronze" {
provider = databricks.staging
catalog_name = databricks_catalog.staging.name
name = "bronze"
}
Output
CLI profiles configured per environment (~/.databrickscfg)
Asset Bundle with dev/staging/prod targets and variable overrides
Per-environment secret scopes with isolated credentials
Python config class for environment-aware code
CI/CD pipeline with GitHub environment secrets and approval gates
Error Handling Issue Cause Solution Wrong environment targeted Missing --profile or -t flag Default profile should always be dev Cross-env data leak Shared catalog Use separate catalogs per environment Secret not found Wrong scope name Verify scope exists: databricks secrets list-scopes --profile $env CI auth failure Expired service principal secret Regenerate OAuth secret or use OIDC
Examples
Quick Environment Verification for profile in dev staging production; do
echo "=== $profile ==="
databricks current-user me --profile $profile 2>/dev/null && echo "OK" || echo "FAILED"
done
Startup Validation config = get_config()
print (f"Environment: {os.getenv('ENVIRONMENT' , 'dev' )} " )
print (f"Catalog: {config.catalog} " )
print (f"Debug: {config.debug} " )
Resources
Next Steps For deployment, see databricks-deploy-integration.
