databricks-asset-bundles

# Databricks Asset Bundles (DABs) ## Overview Databricks Asset Bundles provide infrastructure-as-code for deploying Databricks workflows, jobs, and DLT pipelines. This skill standardizes configuration patterns for serverless-first, production-ready deployments with hierarchical job architecture, proper parameter passing, and comprehensive error prevention. ## When to Use This Skill - Creating or configuring Databricks Asset Bundle YAML files - Deploying serverless jobs, DLT pipelines, dashboards, alerts, apps, or workflows - Setting up hierarchical job architectures (atomic/composite/orchestrator) - Configuring dashboard resources with `dataset_catalog`/`dataset_schema` (CLI 0.281.0+) - Setting up SQL Alerts v2 (schema differs significantly from other resources) - Configuring Databricks Apps in DABs (env vars in `app.yaml`, not `databricks.yml`) - Troubleshooting deployment errors or configuration issues - Converting notebooks to use proper parameter passing patterns - Validating bundle configurations before deployment ## The Deploy Contract (canonical — every other skill references this) The Asset Bundle is the **one and only build artifact** for the data-product spine. Every artifact — jobs, pipelines, schemas, volumes, Genie Spaces, the App's data resources — comes into existence exactly one way: **defined as a resource in `databricks.yml`, brought to life by deploying the bundle.** Both clients (IDE+CLI and Genie Code) author the *same* bundle and deploy it *identically*. Other skills link here for deploy mechanics rather than restating them. **Deploy verb (both clients):** ```bash databricks bundle deploy --target dev ``` | Client | How `databricks bundle deploy --target dev` runs | |---|---| | **IDE (Cursor)** | the local `databricks` CLI, from the bundle's working directory | | **Genie Code** | the **`runDatabricksCli`** tool — never a bare-shell `databricks` call | - **`--target dev` (or another non-prod target) is mandatory.** A *targetless* `bundle deploy` is rejected by a content safety guardrail ("could affect staging/production"). [TESTED P5] - **All bundle resources are YAML-defined** (`databricks.yml` + `resources/*.yml`) — the single representation. Do **not** use the Python `databricks_bundles` flavor. (RULE_5) - **`dev` → `prod` is the CI promotion lane:** `bundle deploy --target prod` runs in **CI only**; in-session deploys stay non-prod. > **Per-user prefix is an invariant (no regression).** In shared workshop catalogs the bundle's > `catalog`/`schema` variables resolve to a **per-user prefix** (schema `{user_schema_prefix}`, > Lakebase/app `{user_app_name}`). Every resource name — and every **Genie Space title** — carries that > prefix so participants stay isolated inside one catalog. The deploy path changes only *how* an artifact > is created (always `bundle deploy`), **never what it is named**. ### Working in Genie Code (reference → `genie-code-environment`) The deploy verb is identical on Genie Code; these are the environmental facts that differ (the full behavioral catalog lives in the **`genie-code-environment`** skill — load it on demand, don't restate it): - **CWD is pinned to the current page's bundle root** — be **on the page of the bundle you are deploying**. There is no `cd` and no `--bundle-root` flag; you can only validate/deploy the bundle tied to the current page. [TESTED P2] - **Edit the *existing* on-page `databricks.yml`.** Files newly created via `createAsset`/the workspace API do **not** reach the CLI's FUSE mount in the same session, so "create a new bundle, then validate it" fails — edit the bundle already on the page. [TESTED P3] - **`bundle validate` / `bundle summary` / `--help` are pre-approved** from any bundle-context page — use them as safe pre-flight; `bundle deploy --target dev` then runs against the on-page bundle. [TESTED P4/P6] The **App** is the one deliberate exception to bundle-deploy: it ships via `apps deploy` (IDE local CLI; Genie Code SDK `w.apps.deploy(<name>, AppDeployment(source_code_path=…, mode=SNAPSHOT))` — see `genie-code-environment` and the AppKit skills). Note the Genie SDK (`WorkspaceClient`) is the most capable path for individual API operations but has **no `bundle deploy` equivalent** (it is a composite client-side op) — so `bundle deploy` always runs through `runDatabricksCli`, never the SDK. ### No in-session artifact creation (RULE_10) The single creation event is **deploy**. SDK `w.*.create()`, hand-run SQL DDL, and `createAsset` are **read-only authoring support** only (inspect schemas, confirm column names/types, check lineage, sample rows) — never the channel that brings a deliverable into existence. A `CREATE …` that is the *body of a bundle-authored DLT/SQL resource* runs **during** `bundle deploy` and **stays** — that is not in-session creation. The **sole carve-out** is a Genie Space via `createAsset` (RULE_8 **Tier 3**), Genie-Code-only and last-resort (see [Genie Spaces — three deploy tiers](#genie-spaces--three-deploy-tiers-rule_8)). ### Verifying a deploy (client-agnostic) After `bundle deploy` + `bundle run`, verify the produced UC state — but verify it **deterministically**: - **Never iterate a raw `SHOW TABLES` for object counts.** Staging `src_*` views and helper objects are not real deliverables and inflate/skew the count. Assert against an **explicit allowlist** of the fully-qualified objects the bundle was supposed to create (under the **prefixed** schema). - **Failed-task diagnostics read the task-level `run_id`** (`run_details.tasks[i].run_id`), not the parent run — a parent `get_run_output` returns `{}`. Pull the failing task's own run id, then its output/logs. (Bucket-B lesson **B10**, lifted here as cross-client guidance.) ## Critical Rules (Quick Reference) ### 🔴 MANDATORY: Serverless Environment Configuration (Environments V4) **EVERY JOB MUST INCLUDE THIS — NO EXCEPTIONS:** ```yaml resources: jobs: <job_name>: name: "[${bundle.target} ${var.user_prefix}] <Display Name>" # ✅ MANDATORY: Serverless environment with V4 environments: - environment_key: "default" spec: environment_version: "4" # 🔴 ALWAYS V4 - never omit or use older versions tasks: - task_key: <task_name> environment_key: default # ✅ MANDATORY: Reference environment in EVERY task notebook_task: notebook_path: ../src/<script>.py ``` **Validation:** Before deploying ANY job YAML: - [ ] `environments:` block exists at job level - [ ] `environment_version: "4"` is set (NEVER omit, NEVER use older versions) - [ ] Every task has `environment_key: default` - [ ] NO `job_clusters:`, `existing_cluster_id:`, or `new_cluster:` defined (serverless only) ### 🔴 MANDATORY: Hierarchical Job Architecture **3-LAYER HIERARCHY - NO EXCEPTIONS:** 1. **Layer 1: Atomic Jobs** - Contain actual `notebook_task` references (single notebook per job) 2. **Layer 2: Composite Jobs** - Reference atomic jobs via `run_job_task` (NO direct notebooks) 3. **Layer 3: Master Orchestrators** - Reference composite/atomic jobs via `run_job_task` (NO direct notebooks) **Rule:** Each notebook appears in EXACTLY ONE atomic job. Higher-level jobs reference lower-level jobs, never duplicate notebooks. ### 🔴 MANDATORY: Parameter Passing Pattern **ALWAYS use `dbutils.widgets.get()` for `notebook_task`, NEVER `argparse`:** ```python # ✅ CORRECT: Databricks notebook def get_parameters(): catalog = dbutils.widgets.get("catalog") # ✅ Works in notebook_task schema = dbutils.widgets.get("schema") return catalog, schema ``` ```yaml # ✅ CORRECT: YAML configuration notebook_task: notebook_path: ../src/script.py base_parameters: # ✅ Dictionary format catalog: ${var.catalog} schema: ${var.schema} ``` **Why:** `notebook_task` passes parameters through widgets, not command-line arguments. Using `argparse` causes immediate failure. ### 🔴 MANDATORY: Task Type Pattern **ALWAYS use `notebook_task`, NEVER `python_task`:** ```yaml # ✅ CORRECT tasks: - task_key: my_task notebook_task: # ✅ Use notebook_task notebook_path: ../src/script.py base_parameters: # ✅ Dictionary format catalog: ${var.catalog} # ❌ WRONG tasks: - task_key: my_task python_task: # ❌ Invalid task type! python_file: ../src/script.py parameters: # ❌ CLI-style doesn't work! - "--catalog=value" ``` ### 🔴 MANDATORY: Notebook Source Format **Python files executed via `notebook_task` MUST use Databricks notebook source format:** ```python # Databricks notebook source # COMMAND ---------- catalog = dbutils.widgets.get("catalog") # COMMAND ---------- spark.sql(f"USE CATALOG {catalog}") ``` **Rules:** - First line: `# Databricks notebook source` - Cell separator: `# COMMAND ----------` (exactly 10 dashes) - Markdown cells: prefix each line with `# MAGIC %md` - Missing separators → all code runs as a single cell (silent failure) - **NEVER mix Python code with `# MAGIC %md` in the same cell** — the entire cell renders as markdown and Python is silently ignored - Common failure: `NameError: name 'xxx' is not defined` — caused by placing `import` or assignments in a `# MAGIC %md` cell. Fix: insert `# COMMAND ----------` between the markdown cell and the code cell See [Notebook Source Format](references/notebook-source-format.md) for complete reference with examples. ## Core Patterns ### Serverless Job Pattern ```yaml resources: jobs: <job_key>: name: "[${bundle.target} ${var.user_prefix}] <Job Display Name>" # ✅ MANDATORY: Serverless environment environments: - environment_key: "default" spec: environment_version: "4" tasks: - task_key: <task_key> environment_key: default # ✅ MANDATORY notebook_task: notebook_path: ../src/<script>.py base_parameters: catalog: ${var.catalog} tags: environment: ${bundle.target} project: <project_name> layer: <bronze|silver|gold> ``` ### DLT Pipeline Pattern ```yaml resources: pipelines: <pipeline_key>: name: "[${bundle.target} ${var.user_prefix}] <Pipeline Display Name>" # ✅ MANDATORY: Root path for Lakeflow Pipelines Editor root_path: ../src/<layer>_pipeline # ✅ Direct Publishing Mode (Modern Pattern) catalog: ${var.catalog} schema: ${var.<layer>_schema} libraries: - notebook: path: ../src/<layer>/<notebook>.py configuration: catalog: ${var.catalog} bronze_schema: ${var.bronze_schema} serverless: true photon: true edition: ADVANCED tags: environment: ${bundle.target} layer: <layer> ``` ### Job Reference Pattern (Hierarchical Architecture) ```yaml # Layer 1: Atomic Job (contains notebook) resources: jobs: tvf_deployment_job: name: "[${bundle.target} ${var.user_prefix}] TVF Deployment" environments: - environment_key: default spec: environment_version: "4" tasks: - task_key: deploy_tvfs environment_key: default notebook_task: # ✅ Actual notebook reference notebook_path: ../../src/semantic/tvfs/deploy_tvfs.py tags: job_level: atomic # Layer 2: Composite Job (references atomic jobs) resources: jobs: semantic_layer_setup_job: name: "[${bundle.target} ${var.user_prefix}] Semantic Layer Setup" tasks: - task_key: deploy_tvfs run_job_task: # ✅ Reference job, NOT notebook job_id: ${resources.jobs.tvf_deployment_job.id} - task_key: deploy_metric_views depends_on: - task_key: deploy_tvfs run_job_task: job_id: ${resources.jobs.metric_view_deployment_job.id} tags: job_level: composite ``` ## Job Hierarchy Overview ### Layer 1: Atomic Jobs - **Purpose:** Single-purpose jobs with actual notebook references - **Pattern:** Use `notebook_task` with `notebook_path` - **Tag:** `job_level: atomic` - **Example:** `tvf_deployment_job`, `gold_setup_job` ### Layer 2: Composite Jobs - **Purpose:** Domain-level coordination (e.g., semantic layer setup) - **Pattern:** Use `run_job_task` to reference atomic jobs - **Tag:** `job_level: composite` - **Example:** `semantic_layer_setup_job`, `monitoring_layer_setup_job` ### Layer 3: Master Orchestrators - **Purpose:** Complete workflow coordination across layers - **Pattern:** Use `run_job_task` to reference composite/atomic jobs - **Tag:** `job_level: orchestrator` - **Example:** `master_setup_orchestrator`, `master_refresh_orchestrator` **Key Principle:** No notebook duplication. Each notebook appears in exactly ONE atomic job. ## Genie Spaces — three deploy tiers (RULE_8) Genie Spaces are the one resource where the canonical "everything is a bundle resource" ideal is not yet fully reachable, so the spine defines **three tiers, preferred first, with a GO/NO-GO that selects the active one.** In **every** tier the Space **title carries the per-user prefix** (decision #7) and `table_identifiers` are **fully-qualified under the prefixed schema**. Record the chosen tier in `deploy_note` (`tier_1_native` once it lands, else `tier_2_provisioning_job`, or `tier_3_createasset`). ### Tier 1 — native `genie_spaces` bundle resource (LANDED — preferred, use this)