databricks-asset-bundles

name	databricks-asset-bundles
description	Standard patterns for Databricks Asset Bundles configuration files for serverless jobs, DLT pipelines, dashboards, alerts, apps, and workflows. Use when creating, configuring, or deploying DABs for infrastructure-as-code deployments. Covers mandatory serverless environment configuration, hierarchical job architecture (atomic/composite/orchestrator), DLT pipeline patterns, dashboard resources with dataset_catalog/dataset_schema, SQL Alerts v2 API schema, Apps lifecycle, Python notebook parameter passing (dbutils.widgets.get vs argparse), deployment error prevention, and pre-deployment validation.
metadata	{"author":"prashanth subrahmanyam","version":"3.0","domain":"infrastructure","role":"shared","used_by_stages":[1,2,3,4,5,6,7,8,9],"last_verified":"2026-06-02","volatility":"medium","clients":["ide_cli","genie_code"],"deploy_verb":"bundle deploy --target dev","deploy_note":"the canonical deploy spine — IDE local CLI ≡ Genie Code runDatabricksCli; App via apps deploy","bundle_resource":"jobs, pipelines, dashboards, alerts, apps, volumes, schemas, postgres_*; genie_spaces Tier-1 LANDED (native resource; CLI >= 1.3.0, verified on 1.10.0)","coverage":"all_stages","upstream_sources":[{"name":"databricks-agent-skills","repo":"databricks/databricks-agent-skills","paths":"[Truncated]","relationship":"extended","last_synced":"2026-08-30","sync_commit":"ca92a6c"}]}

Databricks Asset Bundles (DABs)

Overview

Databricks Asset Bundles provide infrastructure-as-code for deploying Databricks workflows, jobs, and DLT pipelines. This skill standardizes configuration patterns for serverless-first, production-ready deployments with hierarchical job architecture, proper parameter passing, and comprehensive error prevention.

When to Use This Skill

• Creating or configuring Databricks Asset Bundle YAML files
• Deploying serverless jobs, DLT pipelines, dashboards, alerts, apps, or workflows
• Setting up hierarchical job architectures (atomic/composite/orchestrator)
• Configuring dashboard resources with dataset_catalog/dataset_schema (CLI 0.281.0+)
• Setting up SQL Alerts v2 (schema differs significantly from other resources)
• Configuring Databricks Apps in DABs (env vars in app.yaml, not databricks.yml)
• Troubleshooting deployment errors or configuration issues
• Converting notebooks to use proper parameter passing patterns
• Validating bundle configurations before deployment

The Deploy Contract (canonical — every other skill references this)

The Asset Bundle is the one and only build artifact for the data-product spine. Every artifact — jobs, pipelines, schemas, volumes, Genie Spaces, the App's data resources — comes into existence exactly one way: defined as a resource in databricks.yml, brought to life by deploying the bundle. Both clients (IDE+CLI and Genie Code) author the same bundle and deploy it identically. Other skills link here for deploy mechanics rather than restating them.

Deploy verb (both clients):

databricks bundle deploy --target dev

Client	How databricks bundle deploy --target dev runs
IDE (Cursor)	the local databricks CLI, from the bundle's working directory
Genie Code	the runDatabricksCli tool — never a bare-shell databricks call

• --target dev (or another non-prod target) is mandatory. A targetless bundle deploy is rejected by a content safety guardrail ("could affect staging/production"). [TESTED P5]
• All bundle resources are YAML-defined (databricks.yml + resources/*.yml) — the single representation. Do not use the Python databricks_bundles flavor. (RULE_5)
• dev → prod is the CI promotion lane: bundle deploy --target prod runs in CI only; in-session deploys stay non-prod.

Per-user prefix is an invariant (no regression). In shared workshop catalogs the bundle's catalog/schema variables resolve to a per-user prefix (schema {user_schema_prefix}, Lakebase/app {user_app_name}). Every resource name — and every Genie Space title — carries that prefix so participants stay isolated inside one catalog. The deploy path changes only how an artifact is created (always bundle deploy), never what it is named.

Working in Genie Code (reference → genie-code-environment)

The deploy verb is identical on Genie Code; these are the environmental facts that differ (the full behavioral catalog lives in the genie-code-environment skill — load it on demand, don't restate it):

• CWD is pinned to the current page's bundle root — be on the page of the bundle you are deploying. There is no cd and no --bundle-root flag; you can only validate/deploy the bundle tied to the current page. [TESTED P2]
• Edit the existing on-page databricks.yml. Files newly created via createAsset/the workspace API do not reach the CLI's FUSE mount in the same session, so "create a new bundle, then validate it" fails — edit the bundle already on the page. [TESTED P3]
• bundle validate / bundle summary / --help are pre-approved from any bundle-context page — use them as safe pre-flight; bundle deploy --target dev then runs against the on-page bundle. [TESTED P4/P6]

The App is the one deliberate exception to bundle-deploy: it ships via apps deploy (IDE local CLI; Genie Code SDK w.apps.deploy(<name>, AppDeployment(source_code_path=…, mode=SNAPSHOT)) — see genie-code-environment and the AppKit skills). Note the Genie SDK (WorkspaceClient) is the most capable path for individual API operations but has no bundle deploy equivalent (it is a composite client-side op) — so bundle deploy always runs through runDatabricksCli, never the SDK.

No in-session artifact creation (RULE_10)

The single creation event is deploy. SDK w.*.create(), hand-run SQL DDL, and createAsset are read-only authoring support only (inspect schemas, confirm column names/types, check lineage, sample rows) — never the channel that brings a deliverable into existence. A CREATE … that is the body of a bundle-authored DLT/SQL resource runs during bundle deploy and stays — that is not in-session creation. The sole carve-out is a Genie Space via createAsset (RULE_8 Tier 3), Genie-Code-only and last-resort (see Genie Spaces — three deploy tiers).

Verifying a deploy (client-agnostic)

After bundle deploy + bundle run, verify the produced UC state — but verify it deterministically:

• Never iterate a raw SHOW TABLES for object counts. Staging src_* views and helper objects are not real deliverables and inflate/skew the count. Assert against an explicit allowlist of the fully-qualified objects the bundle was supposed to create (under the prefixed schema).
• Failed-task diagnostics read the task-level run_id (run_details.tasks[i].run_id), not the parent run — a parent get_run_output returns {}. Pull the failing task's own run id, then its output/logs.

(Bucket-B lesson B10, lifted here as cross-client guidance.)

Critical Rules (Quick Reference)

🔴 MANDATORY: Serverless Environment Configuration (Environments V4)

EVERY JOB MUST INCLUDE THIS — NO EXCEPTIONS:

resources:
  jobs:
    <job_name>:
      name: "[${bundle.target} ${var.user_prefix}] <Display Name>"
      
      # ✅ MANDATORY: Serverless environment with V4
      environments:
        - environment_key: "default"
          spec:
            environment_version: "4"  # 🔴 ALWAYS V4 - never omit or use older versions
      
      tasks:
        - task_key: <task_name>
          environment_key: default  # ✅ MANDATORY: Reference environment in EVERY task
          notebook_task:
            notebook_path: ../src/<script>.py

Validation: Before deploying ANY job YAML:

• environments: block exists at job level
• environment_version: "4" is set (NEVER omit, NEVER use older versions)
• Every task has environment_key: default
• NO job_clusters:, existing_cluster_id:, or new_cluster: defined (serverless only)

🔴 MANDATORY: Hierarchical Job Architecture

3-LAYER HIERARCHY - NO EXCEPTIONS:

1. Layer 1: Atomic Jobs - Contain actual notebook_task references (single notebook per job)
2. Layer 2: Composite Jobs - Reference atomic jobs via run_job_task (NO direct notebooks)
3. Layer 3: Master Orchestrators - Reference composite/atomic jobs via run_job_task (NO direct notebooks)

Rule: Each notebook appears in EXACTLY ONE atomic job. Higher-level jobs reference lower-level jobs, never duplicate notebooks.

🔴 MANDATORY: Parameter Passing Pattern

ALWAYS use dbutils.widgets.get() for notebook_task, NEVER argparse:

# ✅ CORRECT: Databricks notebook
def get_parameters():
    catalog = dbutils.widgets.get("catalog")  # ✅ Works in notebook_task
    schema = dbutils.widgets.get("schema")
    return catalog, schema

# ✅ CORRECT: YAML configuration
notebook_task:
  notebook_path: ../src/script.py
  base_parameters:  # ✅ Dictionary format
    catalog: ${var.catalog}
    schema: ${var.schema}

Why: notebook_task passes parameters through widgets, not command-line arguments. Using argparse causes immediate failure.

🔴 MANDATORY: Task Type Pattern

ALWAYS use notebook_task, NEVER python_task:

# ✅ CORRECT
tasks:
  - task_key: my_task
    notebook_task:  # ✅ Use notebook_task
      notebook_path: ../src/script.py
      base_parameters:  # ✅ Dictionary format
        catalog: ${var.catalog}

# ❌ WRONG
tasks:
  - task_key: my_task
    python_task:  # ❌ Invalid task type!
      python_file: ../src/script.py
      parameters:  # ❌ CLI-style doesn't work!
        - "--catalog=value"

🔴 MANDATORY: Notebook Source Format

Python files executed via notebook_task MUST use Databricks notebook source format:

# Databricks notebook source

# COMMAND ----------

catalog = dbutils.widgets.get("catalog")

# COMMAND ----------

spark.sql(f"USE CATALOG {catalog}")

Rules:

• First line: # Databricks notebook source
• Cell separator: # COMMAND ---------- (exactly 10 dashes)
• Markdown cells: prefix each line with # MAGIC %md
• Missing separators → all code runs as a single cell (silent failure)
• NEVER mix Python code with # MAGIC %md in the same cell — the entire cell renders as markdown and Python is silently ignored
• Common failure: NameError: name 'xxx' is not defined — caused by placing import or assignments in a # MAGIC %md cell. Fix: insert # COMMAND ---------- between the markdown cell and the code cell

See Notebook Source Format for complete reference with examples.

Core Patterns

Serverless Job Pattern

resources:
  jobs:
    <job_key>:
      name: "[${bundle.target} ${var.user_prefix}] <Job Display Name>"
      
      # ✅ MANDATORY: Serverless environment
      environments:
        - environment_key: "default"
          spec:
            environment_version: "4"
      
      tasks:
        - task_key: <task_key>
          environment_key: default  # ✅ MANDATORY
          notebook_task:
            notebook_path: ../src/<script>.py
            base_parameters:
              catalog: ${var.catalog}
      
      tags:
        environment: ${bundle.target}
        project: <project_name>
        layer: <bronze|silver|gold>

DLT Pipeline Pattern

resources:
  pipelines:
    <pipeline_key>:
      name: "[${bundle.target} ${var.user_prefix}] <Pipeline Display Name>"
      
      # ✅ MANDATORY: Root path for Lakeflow Pipelines Editor
      root_path: ../src/<layer>_pipeline
      
      # ✅ Direct Publishing Mode (Modern Pattern)
      catalog: ${var.catalog}
      schema: ${var.<layer>_schema}
      
      libraries:
        - notebook:
            path: ../src/<layer>/<notebook>.py
      
      configuration:
        catalog: ${var.catalog}
        bronze_schema: ${var.bronze_schema}
      
      serverless: true
      photon: true
      edition: ADVANCED
      
      tags:
        environment: ${bundle.target}
        layer: <layer>

Job Reference Pattern (Hierarchical Architecture)

# Layer 1: Atomic Job (contains notebook)
resources:
  jobs:
    tvf_deployment_job:
      name: "[${bundle.target} ${var.user_prefix}] TVF Deployment"
      environments:
        - environment_key: default
          spec:
            environment_version: "4"
      tasks:
        - task_key: deploy_tvfs
          environment_key: default
          notebook_task:  # ✅ Actual notebook reference
            notebook_path: ../../src/semantic/tvfs/deploy_tvfs.py
      tags:
        job_level: atomic

# Layer 2: Composite Job (references atomic jobs)
resources:
  jobs:
    semantic_layer_setup_job:
      name: "[${bundle.target} ${var.user_prefix}] Semantic Layer Setup"
      tasks:
        - task_key: deploy_tvfs
          run_job_task:  # ✅ Reference job, NOT notebook
            job_id: ${resources.jobs.tvf_deployment_job.id}
        - task_key: deploy_metric_views
          depends_on:
            - task_key: deploy_tvfs
          run_job_task:
            job_id: ${resources.jobs.metric_view_deployment_job.id}
      tags:
        job_level: composite

Job Hierarchy Overview

Layer 1: Atomic Jobs

• Purpose: Single-purpose jobs with actual notebook references
• Pattern: Use notebook_task with notebook_path
• Tag: job_level: atomic
• Example: tvf_deployment_job, gold_setup_job

Layer 2: Composite Jobs

• Purpose: Domain-level coordination (e.g., semantic layer setup)
• Pattern: Use run_job_task to reference atomic jobs
• Tag: job_level: composite
• Example: semantic_layer_setup_job, monitoring_layer_setup_job

Layer 3: Master Orchestrators

• Purpose: Complete workflow coordination across layers
• Pattern: Use run_job_task to reference composite/atomic jobs
• Tag: job_level: orchestrator
• Example: master_setup_orchestrator, master_refresh_orchestrator

Key Principle: No notebook duplication. Each notebook appears in exactly ONE atomic job.

Genie Spaces — three deploy tiers (RULE_8)

Genie Spaces are the one resource where the canonical "everything is a bundle resource" ideal is not yet fully reachable, so the spine defines three tiers, preferred first, with a GO/NO-GO that selects the active one. In every tier the Space title carries the per-user prefix (decision #7) and table_identifiers are fully-qualified under the prefixed schema. Record the chosen tier in deploy_note (tier_1_native once it lands, else tier_2_provisioning_job, or tier_3_createasset).

Tier 1 — native genie_spaces bundle resource (LANDED — preferred, use this)