databricks-deployment-verification

# Deployment Verification — THE THREE PATHS RULE > **STOP. Before declaring any workspace "deployed" or "verified", you MUST run all three of:** > > 1. **Classic cluster** with UC (`data_security_mode = "SINGLE_USER"`) > 2. **Serverless SQL warehouse** (PRO) > 3. **Serverless notebook job** (ephemeral job compute) > > Each must read AND write a UC table end-to-end (CREATE → INSERT → SELECT → DROP). > > One success on serverless does NOT count as verified. Classic must work too — that's where most real-world skill bugs live (cluster policies, init scripts, custom AMIs, SCC/PrivateLink port story, JVM warmup interactions with private storage, `data_security_mode` enforcement). If you only test serverless you're testing the easy path. ## Approval gate for the verification resources Verification is **mandatory** (never skip it), but the verification itself creates real remote resources — a classic cluster, a PRO SQL warehouse, a notebook, and a job. Those are mutations, so before creating them present ONE plan and get explicit approval: 1. **Target** -- the workspace and the `--profile` (or host) you will verify against. 2. **Change set** -- the exact verify-* resources you will create (cluster, warehouse, notebook, job) and the UC table you will CREATE/INSERT/SELECT/DROP. Batched: one approval for the whole verification run, not one prompt per resource. 3. **Wait for explicit approval**, then create **only** those resources. 4. **Retry / recovery** uses the same gate — never create extra resources without re-approval. 5. **Cleanup** (below) destroys only the verify-* resources this run created, and reports what it removed. "Mandatory" means you must not declare a deployment done without running verification — it does not mean skipping this approval. It is the default for interactive use; auto-approve / headless mode is the customer's choice and responsibility (see SECURITY.md). ## Why this is its own skill Deployment verification is the most-skipped step in every platform stress test. Agents reach for the cheapest path (serverless SQL) because it's the fastest and the cluster cold-start warning in `workspace-config` discourages classic. Result: skill gaps that only surface on classic clusters never get caught until a real customer hits them in prod. **This skill exists to make verification impossible to forget and impossible to half-do.** ## The Three Paths ### Path 1 — Classic cluster (UC-enabled) The most informative test. Surfaces: - Cluster policy correctness (`data_security_mode`, `single_user_name`, allowed instance types) - IAM role / managed identity propagation to the cluster - UC metastore + external location + storage credential resolution from worker nodes - Init scripts and custom AMIs (if used by persona) - SCC relay + private networking (workers reach control plane on TCP 443) - Workspace-level secret scopes if used in Spark conf - DBFS root encryption (CMK on managed disks for HIGH personas) **Minimum cluster spec:** ```hcl data "databricks_node_type" "verify" { # DO NOT use the bare "smallest" preset — on AWS it picks m5d.large (2 vCPU / 8GB), # which fails Spark driver startup with DriverStartupTimeout 300s on a fresh # workspace (JVM warmup needs more cores). Pin a real floor: min_cores = 8 min_memory_gb = 32 local_disk = true category = "General Purpose" } resource "databricks_cluster" "verify_classic" { cluster_name = "verify-classic-${var.workspace_name}" spark_version = data.databricks_spark_version.lts.id # latest LTS node_type_id = data.databricks_node_type.verify.id num_workers = 1 # NOT autoscaled — keep deterministic autotermination_minutes = 0 # CRITICAL: 0 = no auto-terminate during JVM warmup # If you set 30, on a fresh workspace the cluster can # reach RUNNING but stay in "Starting Spark" past the # 30-min window → terminates before any job attaches → # infinite restart loop. Set 0 for verify clusters and # destroy them explicitly at end (cleanup section). data_security_mode = "SINGLE_USER" # required for UC table reads single_user_name = var.verifier_principal # for an SP, this is the SP's # application_id (UUID) — NOT the # numeric SP id. For a user, the # email. For a group, the group name. # AWS only — required when local_disk = false OR when the chosen instance type # doesn't have NVMe instance store (most non-d-series). Without this, cluster # creation fails with "Cluster requires at least one EBS volume". aws_attributes { ebs_volume_count = 1 ebs_volume_size = 100 ebs_volume_type = "GENERAL_PURPOSE_SSD" } # Tag so it can be identified + cleaned up custom_tags = { purpose = "deployment-verification" persona = var.persona_name } } ``` **Why the explicit `min_cores`/`min_memory_gb` floor:** the bare `data "databricks_node_type" "smallest"` selector picks the cheapest UC-eligible node, which on AWS is m5d.large (2 vCPU / 8 GB). On a fresh workspace, the Spark driver JVM warmup + UC metadata cache + storage credential resolution doesn't finish in the API's 300s `DriverStartupTimeout` window with only 2 cores — even m5d.xlarge (4 vCPU / 16 GB) has been observed to time out. m5.2xlarge (8 vCPU / 32 GB) is the lowest size that reliably starts cold. Pinning `min_cores = 8` + `min_memory_gb = 32` lets the data source pick a regional equivalent on Azure/GCP without hardcoding instance types. Use the same floor on Azure (selects `Standard_D8s_v5` or similar) and GCP (`n2-standard-8`). **Expected timing:** 5–10 min cold-start on a fresh workspace at the recommended size, 10–15 min on the smaller-than-recommended sizes that often time out anyway. The skill warning about classic cold-start is REAL but it's not an excuse to skip the test — it's a reason to start the cluster early in your deploy and verify near the end. **SQL test (run via `databricks api post /api/2.0/sql/statements` against the cluster, OR notebook task on the cluster):** ```sql -- Replace catalog/schema with the catalog you just deployed CREATE TABLE <catalog>.<schema>.verify_classic (id INT, msg STRING); INSERT INTO <catalog>.<schema>.verify_classic VALUES (1, 'classic verified'); SELECT * FROM <catalog>.<schema>.verify_classic; DROP TABLE <catalog>.<schema>.verify_classic; ``` **Pass criteria:** all four statements succeed. SELECT must return exactly the row inserted. If any fails: do NOT mark verified. Diagnose, fix, retry. **Hard-failure modes to watch for:** | Symptom | Likely root cause | |---|---| | Cluster stuck in "Pending" >20 min | IAM role propagation, instance profile attached but not yet readable | | Cluster started but SQL fails with "credential not found" | UC IAM policy missing `s3:ListBucketMultipartUploads` family (AWS) or DES managed identity not granted Get/UnwrapKey on Key Vault (Azure CMK) | | `data_security_mode=SINGLE_USER` rejected | Cluster policy is too restrictive; relax `data_security_mode` constraint | | Cluster started but timed out fetching control-plane | SCC over PrivateLink misconfigured (AWS) or NSG/firewall rules wrong (Azure) — see private-networking skill | | AWS only: workers can't reach S3 | Missing `s3:GetBucketTagging` or `s3:GetBucketAcl` on the UC IAM policy (often missing from older skill examples) | | Azure only: spark fails reading UC managed table | DES (Disk Encryption Set) managed identity missing Key Vault access policy | ### Path 2 — Serverless SQL warehouse (PRO) Fastest verification. Tests UC + serverless + storage credential resolution from the serverless plane (different from classic — different network egress, different identity). **Minimum warehouse spec:** ```hcl resource "databricks_sql_endpoint" "verify_serverless" { name = "verify-serverless" cluster_size = "2X-Small" # smallest serverless size enable_serverless_compute = true warehouse_type = "PRO" # PRO required for UC-aware features auto_stop_mins = 10 min_num_clusters = 1 # REQUIRED — SDK warehouses.create() max_num_clusters = 1 # rejects with "0 is not a valid value # for max_num_clusters" if either is # omitted. Both must be >= 1. tags { custom_tags { key = "purpose" value = "deployment-verification" } } } ``` The `databricks_sql_endpoint` resource defaults `min_num_clusters` and `max_num_clusters` to 0 if not set, but the underlying `warehouses.create()` SDK call rejects 0. Always set both explicitly to 1 (or higher if you actually want autoscaling). Symptom of omitting: `INVALID_PARAMETER_VALUE: 0 is not a valid value for max_num_clusters`. **Pass criteria:** same four-statement CRUD as Path 1 succeeds. **CRITICAL:** the Statement Execution API (`/api/2.0/sql/statements`) does NOT accept multi-statement bodies — it returns `PARSE_SYNTAX_ERROR: Syntax error at or near 'CREATE': extra input 'CREATE'`. You MUST split into 4 separate calls: ```bash WS_PROFILE=<workspace-profile> WAREHOUSE_ID=<id> TABLE="<catalog>.<schema>.verify_serverless" for STMT in \ "CREATE TABLE $TABLE (id INT, msg STRING)" \ "INSERT INTO $TABLE VALUES (2, 'serverless verified')" \ "SELECT * FROM $TABLE" \ "DROP TABLE $TABLE" ; do databricks api post /api/2.0/sql/statements \ --profile "$WS_PROFILE" \ --json "{\"warehouse_id\": \"$WAREHOUSE_ID\", \"statement\": $(jq -Rs . <<< "$STMT"), \"wait_timeout\": \"30s\"}" \ || { echo "FAIL on: $STMT"; exit 1; } done ``` The third call (SELECT) must return one row containing `[2, "serverless verified"]`. **Hard-failure modes:** | Symptom | Likely root cause | |---|---| | Warehouse fails to start | Serverless not enabled on the workspace OR region doesn't support serverless | | SQL fails with storage error | Storage account firewall (Azure) blocking serverless egress — needs trusted-services exception OR private endpoint to managed storage | | Warehouse PROVISIONING but never STARTING | Account-level entitlement for serverless missing | ### Path 3 — Serverless notebook job (ephemeral job compute) Tests the **third** compute plane: serverless job compute. This is structurally different from both Path 1 (classic) and Path 2 (serverless SQL) because: - Different identity context (job runs as the runner / SP, not as cluster owner) - Different egress (different network path than serverless SQL warehouse) - Tests notebook → UC table flow (different code path than SQL warehouse → UC table) **Minimum job spec — MUST be a Python notebook with `dbutils.notebook.exit(...)`.** SQL notebooks don't surface SELECT results through the run-output API, so the verifier can't programmatically confirm the row was returned. ```hcl resource "databricks_notebook" "verify_notebook" { path = "/Shared/verify_notebook" language = "PYTHON" content_base64 = base64encode(<<-EOT # Databricks notebook source catalog, schema = "<catalog>", "<schema>" table = f"{catalog}.{schema}.verify_notebook" spark.sql(f"CREATE TABLE {table} (id INT, msg STRING)") spark.sql(f"INSERT INTO {table} VALUES (3, 'notebook verified')") rows = [r.asDict() for r in spark.sql(f"SELECT * FROM {table}").collect()] spark.sql(f"DROP TABLE {table}") dbutils.notebook.exit(str(rows)) # surfaces in run-output as notebook_output.result EOT ) } resource "databricks_job" "verify_notebook" { name = "verify-notebook" task { task_key = "verify" notebook_task { notebook_path = databricks_notebook.verify_notebook.path } # No cluster spec — uses serverless job compute by default } } ``` Then trigger and wait. **CRITICAL:** when you submit a job via `runs/submit` or trigger one via `run-now`, the API returns the OUTER run id, but `runs/get-output` errors with `"Retrieving the output of runs with multiple tasks is not supported"` if you query the outer id directly — even when there's only one task. You must extract the task-level run id first: ```bash WS_PROFILE=<workspace-profile> JOB_ID=<job-id> # 1. Trigger RUN_ID=$(databricks jobs run-now --job-id "$JOB_ID" --profile "$WS_PROFILE" | jq -r '.run_id') # 2. Wait for SUCCESS (poll runs/get every 10s until life_cycle_state == TERMINATED) while :; do STATUS=$(databricks api get /api/2.1/jobs/runs/get --profile "$WS_PROFILE" --json "{\"run_id\": $RUN_ID}") LCS=$(echo "$STATUS" | jq -r '.state.life_cycle_state') RES=$(echo "$STATUS" | jq -r '.state.result_state') [ "$LCS" = "TERMINATED" ] && break sleep 10 done [ "$RES" = "SUCCESS" ] || { echo "FAIL: $RES"; exit 1; } # 3. Extract task-level run id (NOT the outer RUN_ID) TASK_RUN_ID=$(echo "$STATUS" | jq -r '.tasks[0].run_id') # 4. Get notebook output via the TASK run id OUT=$(databricks api get /api/2.1/jobs/runs/get-output --profile "$WS_PROFILE" \ --json "{\"run_id\": $TASK_RUN_ID}" | jq -r '.notebook_output.result') # OUT should equal the str(rows) from dbutils.notebook.exit, e.g. "[{'id': 3, 'msg': 'notebook verified'}]" echo "$OUT" | grep -q "notebook verified" || { echo "FAIL: output didn't match"; exit 1; } ``` **Pass criteria:** outer run state = TERMINATED/SUCCESS AND `notebook_output.result` from the task-level run id contains `notebook verified`. **Hard-failure modes:** | Symptom | Likely root cause | |---|---| | Job stuck in `PENDING_QUEUE` | Serverless compute capacity issue or workspace-level serverless not enabled | | Job FAILED with permission error on UC | The job runs as the workspace creator (you) — but the catalog/schema may have been granted only to a group. Either run as SP that's in the group OR grant the runner directly. | | Notebook fails on `CREATE TABLE` | Same UC IAM/identity issue as Path 1 — surface here means it's a write-path issue not a compute-path issue | ## The Verification Workflow (mandatory) Run these phases at the END of every workspace + UC deployment, in this order: ``` 1. Verify all three paths PASS → workspace is verified 2. If any path FAILS → diagnose, fix, RE-RUN ALL THREE 3. Save verification artifacts → see "What to save" below 4. Update transcript with PASS/FAIL → per path ``` **What to save (in the deployment dir):** - `verification.tf` — the three resources defined above (cluster, warehouse, notebook+job) - `verification.log` — output of each SQL statement, exit codes, durations - `verification.json` — structured pass/fail per path: ```json { "classic": {"status": "PASS|FAIL", "duration_s": 720, "cluster_id": "0503-...", "error": null}, "serverless": {"status": "PASS|FAIL", "duration_s": 8, "warehouse_id": "...", "error": null}, "notebook": {"status": "PASS|FAIL", "duration_s": 45, "job_id": "...", "run_id": "...", "error": null} } ``` ## When to skip (NARROW exceptions) You may skip a path ONLY if: - **Skip classic** — only when one of: - Customer **explicitly** requires no classic compute (regulated workloads, serverless-only mandate) - The workspace was created with `compute_mode = "SERVERLESS"` (GCP only) — these workspaces have no worker environments and classic clusters cannot run, period. `databricks clusters create` returns `Current organization X does not have any associated worker environments`. This is structural, not a skip-for-convenience. - **GCP-specific:** PSC-backend-only workspace + UC — classic GKE pods cannot reach the UC API through PSC-backend-only routing (`403 Unauthorized network access to workspace`). If the customer truly needs classic + UC + PSC, they need PSC frontend too. Log this as `SKIPPED-WITH-REASON` and recommend adding frontend PSC or switching to serverless workflows. Log the reason in `verification.json` with `"status": "SKIPPED-WITH-REASON"`. Do NOT skip classic just because cold-start is slow. - **Skip serverless** — only when the customer explicitly forbids serverless (regulated FSI / banking with no serverless approval) AND the workspace genuinely does not have serverless enabled. Log the reason.