Fix the deploy-time foot-guns of Databricks Asset Bundles (DAB) and the
infrastructure operations around them: the bundle-bind gap for UC catalogs and
external locations, the "unexpected EOF reading terraform.tfstate" redeploy failure,
the schema-GRANT-ordering bug that fails the first deploy, customer-managed-key (CMK)
rotation that requires draining the whole workspace, and the PrivateLink cost leak
where S3/STS/Kinesis still traverse the NAT. Includes a PreToolUse hook that backs up
and validates the bundle's terraform state before every deploy, and a PostToolUse
hook that recognises the transient GRANT-ordering failure and recommends a single
retry. Use when a databricks bundle deploy fails, before rotating a CMK, when a UC
resource cannot be bound to a bundle, or when auditing PrivateLink networking cost.
Trigger with "bundle deploy failed", "unexpected EOF terraform.tfstate", "bundle bind
external location", "User does not have CREATE TABLE on Schema", "rotate CMK",
"privatelink still using NAT".
Fix the deploy-time foot-guns of Databricks Asset Bundles (DAB) and the
infrastructure operations around them: the bundle-bind gap for UC catalogs and
external locations, the "unexpected EOF reading terraform.tfstate" redeploy failure,
the schema-GRANT-ordering bug that fails the first deploy, customer-managed-key (CMK)
rotation that requires draining the whole workspace, and the PrivateLink cost leak
where S3/STS/Kinesis still traverse the NAT. Includes a PreToolUse hook that backs up
and validates the bundle's terraform state before every deploy, and a PostToolUse
hook that recognises the transient GRANT-ordering failure and recommends a single
retry. Use when a databricks bundle deploy fails, before rotating a CMK, when a UC
resource cannot be bound to a bundle, or when auditing PrivateLink networking cost.
Trigger with "bundle deploy failed", "unexpected EOF terraform.tfstate", "bundle bind
external location", "User does not have CREATE TABLE on Schema", "rotate CMK",
"privatelink still using NAT".
The deploy + infrastructure spine of the pack. Databricks Asset Bundles (DAB) is
immature tooling — the replacement for the deprecated dbx, with a moving bug list
across CLI versions — and the infrastructure operations around a deploy (encryption
keys, networking, workspace config) bite production platform teams hardest. This skill
turns the five worst deploy-time foot-guns into deterministic, reversible operations.
Overview
Five pains, all from the pack's deploy-ops research:
Asset Bundles (DAB). D4 — databricks bundle bind does NOT yet support UC catalogs
or external locations (databricks/cli#4842), so existing UC resources cannot come under
bundle management without hand-editing terraform.tfstate (hostile) or
destroy-and-recreate (impossible with dependent tables). D5 — "unexpected EOF reading
terraform.tfstate" on every redeploy after the first (databricks/cli#4986), which bricks
the bundle until you switch to DATABRICKS_BUNDLE_ENGINE=direct. D6 — the schema
GRANT-ordering bug (databricks/cli#4573): the first deploy to a fresh workspace fails
with "User does not have CREATE TABLE on Schema", but the second succeeds because the
first applied the grants before dying.
Deploy-time infrastructure. D8 — rotating a workspace's customer-managed key (CMK)
requires terminating every cluster, instance pool, and SQL warehouse first: a hard
maintenance window. D9 — the PrivateLink trap: enabling PrivateLink for the control
plane is mistaken for "all traffic is private," but the data plane's S3 / STS / Kinesis
calls still traverse the NAT, billing NAT-processing + cross-AZ transfer until each gets
its own VPC endpoint.
The two hooks (this is the pack's only two-hook skill). A PreToolUse hook
(hooks/bundle-deploy-guard.py) runs before every databricks bundle deploy: it
validates the bundle's local terraform state parses as JSON, caches a timestamped
known-good backup as a recovery escape hatch, and warns loudly if the state is corrupt
or has shrunk (the D5 signature). A PostToolUse hook (hooks/bundle-grant-retry.py)
watches a deploy's output and — ONLY on the exact D6 "does not have … on Schema"
signature — adds context recommending one retry, with the reason. Both are advisory:
the pre-hook never blocks a deploy, and the post-hook never masks a real error — it
surfaces the diagnosis and stops recommending retries if the same failure persists.
Deterministic work lives in scripts/; deep knowledge in references/. The
import-uc-resource-to-bundle.py D4 workaround is self-deprecating — it exists only
until #4842 closes and says so. Control-plane state comes from the
server (, )
or the CLI; advisory-mode fallback accepts pasted input.
databricks-workspace-mcp
external_locations_list
storage_credentials_list
Prerequisites
Databricks CLI authenticated, plus terraform and jq on PATH — for the bundle,
import, and state operations.
databricks-workspace-mcp registered (optional) — for external_locations_list
and storage_credentials_list. Absent → the CLI (databricks external-locations list) or advisory mode on pasted input.
Account-level OAuth M2M (service principal) for CMK rotation — the D8 operations
are account-scoped; a workspace PAT is insufficient. AWS/Azure/GCP CLI for the
cloud-side key + VPC-endpoint work.
The two hooks are plugin-level — installed with the pack. The pre-hook is silent
unless a state looks corrupt; the post-hook is silent unless the exact D6 error fires.
Instructions
Pick the flow by symptom. Always name the exact error string / issue id —
unexpected EOF reading terraform.tfstate (#4986), does not have CREATE TABLE on Schema (#4573), the bundle bind UC gap (#4842) — an operator greps for those.
Step 1: A bundle deploy that fails on state (D5)
"unexpected EOF reading terraform.tfstate" after the first deploy is #4986. The
bundle-deploy-guard PreToolUse hook already cached a known-good backup; restore it,
or switch engines:
Step 2: First deploy fails "does not have CREATE TABLE on Schema" (D6)
This is the transient GRANT-ordering bug (#4573). The bundle-grant-retry PostToolUse
hook flags it; re-run the SAME deploy exactly once — the failed first pass already
applied the grants. If the identical error persists after one retry, it is NOT this
transient — treat it as a real missing privilege and stop retrying.
Step 3: A UC resource that will not bind (D4)
databricks bundle bind cannot take a UC catalog or external location yet (#4842).
Generate a review-first Terraform import plan (never hand-edit state):
CMK rotation needs the whole workspace drained. Inventory the running compute (via
external_locations_list / the CLI), then plan the drain — dry-run by default:
PrivateLink covers the control plane only. Audit the data-plane VPC for the S3/STS/Kinesis
endpoints and emit remediation Terraform for any that are missing:
Example 1: "My second bundle deploy fails with unexpected EOF reading terraform.tfstate."
D5 (#4986). The bundle-deploy-guard hook already cached a known-good state before the
deploy — restore it, then redeploy with DATABRICKS_BUNDLE_ENGINE=direct (the direct
engine never reads the state file, so the EOF class cannot fire).
Example 2: "First deploy to a fresh workspace: User does not have CREATE TABLE on Schema."
D6 (#4573). The bundle-grant-retry hook recognises the exact signature and recommends
one retry — the failed first deploy applied the schema grants, so the second succeeds.
Example 3: "I need to bring an existing external location under my bundle."
D4 (#4842) — bundle bind can't. import-uc-resource-to-bundle.py --type external_location emits the resources: YAML plus a Terraform import block to adopt
it without recreating it or hand-editing state. The script self-deprecates when #4842 closes.
Example 4: "We turned on PrivateLink but the NAT bill went up."
D9. audit-vpc-endpoints.py --present s3 finds STS and Kinesis have no Interface
endpoint, so credential-vending and log traffic still cross the NAT — it emits the
remediation Terraform for both.
