hc-dab-deployment

name

hc-dab-deployment

description

Deploy services to dev/test/prod via Databricks Asset Bundles. Use when the user asks to "deploy", "promote", "deploy to dev/test/prod", "run bundle deploy", "validate the bundle", "add a resource to the bundle", "set a DAB variable", or anything about DAB targets, profiles, or app deployment. Codifies the per-service resource-include layout, the target → profile mapping, the app → Lakebase + app → app resource wiring, and the verbs to use at each step.

hc-dab-deployment

DABs (Declarative Automation Bundles, formerly Databricks Asset Bundles) deploy every service + every job atomically against the target workspace. Lakebase projects are intentionally OUTSIDE the bundle (see below) — they must be provisioned first via scripts/lakebase-project-up.sh, then databricks bundle deploy -t <target> wires the apps to those projects.

Deploy order (always)

1. Provision Lakebase projects for the target — once per (service, env) pair, idempotent:

   for svc in patient provider appointment lab prescription billing; do
     ./scripts/lakebase-project-up.sh "$svc" <env>
   done
   

   Skipping this step makes bundle deploy fail with Postgres branch projects/<svc>-<env>/branches/production does not exist — the bundle's app resources reference these projects by path string, so the projects must exist first.
2. Validate the bundle: databricks bundle validate -t <target>.
3. Deploy + start the apps: scripts/deploy-and-run-bundle.sh <target>. This wraps bundle deploy (which materializes the seven app resources but leaves them in UNAVAILABLE) with a bundle run per app key, which submits an app deployment from the synced source and starts the app. See "bundle deploy vs. bundle run" below.

For per-PR previews, replace step 1 with scripts/lakebase-branch-up.sh per service (creates the feat-<slug> branch on the existing project) and pass the matching --var overrides into the deploy-and-run script in step 3 — see "Per-PR preview deployment" below.

bundle deploy vs. bundle run

databricks bundle deploy is two things stuck together:

1. Sync the source tree into the workspace (/Workspace/.../source/... under the bundle's root_path).
2. Materialize each declared resource — for an app resource that means registering the app with the Apps platform, attaching Lakebase + cross-app permissions, and plumbing env vars.

What it does NOT do is submit an app deployment — the platform unit that pulls the synced source path and actually starts running the app. So a fresh bundle deploy against a never-deployed workspace leaves every app in UNAVAILABLE.

databricks bundle run -t <target> <app_key> is the verb that flips them on. For an app resource it (a) submits a new app deployment from source_code_path, (b) starts/updates the running app, (c) blocks until ACTIVE (or fails). Run it once per app key after bundle deploy and the apps are live.

scripts/deploy-and-run-bundle.sh does both in deploy-order (services first, BFF last):

# Trunk-dev: deploy bundle, then start all 7 apps
scripts/deploy-and-run-bundle.sh dev

# Already deployed; just (re)launch every app
scripts/deploy-and-run-bundle.sh dev --skip-deploy --restart

# Iterate on one or two services
scripts/deploy-and-run-bundle.sh dev --only=patient,lab

# PR-preview shape (matches ci-local.sh pr-validate's overrides)
SLUG=$(./scripts/sanitize-branch-slug.sh "$(git branch --show-current)")
scripts/deploy-and-run-bundle.sh dev \
  --var "app_name_suffix=-feat-$SLUG" \
  --var "lakebase_branch=feat-$SLUG"

Pass-through flags: --no-wait, --restart (forwarded to bundle run); --var KEY=VALUE (forwarded to both bundle deploy and bundle run); --skip-deploy (skip step 1 if the bundle is already up-to-date); --only=<svc,svc,...> (restrict to a comma-separated subset of app keys). The script does NOT run Lakebase project provisioning, migrations, or smoke tests — for the full pipeline use ci-local.sh deploy <env>.

When to use

• "Deploy all services to dev"
• "Promote to test" / "deploy to prod"
• "Add a new app/job/resource to the bundle"
• "Validate the bundle before pushing"
• "Set a DAB variable for catalog / Lakebase branch"
• "Wire app A so it can call app B"
• The user is editing databricks.yml or resources/*.yml

When NOT to use

• The user is creating a Lakebase branch (per-feature) — that's hc-lakebase-branching. The bundle's lakebase_branch variable selects which branch each app's postgres resource binds to; creating the branch itself is out of band.
• The user is editing GitHub Actions workflows — that's hc-gitflow-cicd.

Layout

databricks.yml                  ← root bundle, includes resources/*.yml
resources/
  patient.yml                   ← postgres_projects.<svc>_db + apps.<svc>_app (with resources block)
  provider.yml
  appointment.yml
  lab.yml
  prescription.yml
  billing.yml
  hc-portal.yml                 ← apps.hc_portal_app (with cross-app resources block)
  shared.yml                    ← shared variables, common labels

One file per service, plus hc-portal.yml for the frontend, plus shared.yml for cross-cutting bits.

databricks.yml (the root)

bundle:
  name: microbricks

include:
  - resources/*.yml

variables:
  catalog:
    description: "Unity Catalog the apps read from"
    default: "main"

  # Suffix appended to every app name. Empty for canonical dev/test/prod
  # deploys; `-feat-<slug>` for per-PR previews driven by pr-validate.yml.
  app_name_suffix:
    description: "Optional suffix on every app name. Empty for stable deploys; '-feat-<slug>' for previews."
    default: ""

  lakebase_branch:
    description: "Lakebase branch every <svc>_db postgres resource binds to. 'production' for test/prod; 'feat-<slug>' for per-feature-branch dev deploys."
    default: "production"

targets:
  dev:
    default: true
    mode: development
    workspace:
      profile: hc-dev
      # `mode: development` requires `${workspace.current_user.userName}` in
      # root_path. The suffix further isolates trunk-dev (no suffix) from
      # per-PR previews so concurrent deploys don't stomp each other's state.
      root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}${var.app_name_suffix}
    variables:
      catalog: "hc_dev"

  test:
    mode: production
    workspace:
      profile: hc-test
      root_path: /Workspace/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: "hc_test"

  prod:
    mode: production
    workspace:
      profile: hc-prod
      root_path: /Workspace/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: "hc_prod"

mode: development automatically prefixes resource names with the deploying user's email so multiple devs can deploy to the same hc-dev workspace without colliding. mode: production skips that prefix and requires workspace.root_path to fix the deploy location.

production mode also rejects bundle destroy without --force, which is the right safety floor.

resources/<svc>.yml shape (the canonical template)

Every service follows this shape. The only differences across the six service files are entity names — auth.py, db.py, and the bundle resource shapes are byte-identical otherwise.

Important: Lakebase projects (postgres_projects) are NOT in the bundle. They live outside via scripts/lakebase-project-{up,down}.sh. Reason: putting the project in the bundle meant bundle deploy/bundle destroy tracked the SHARED project in its terraform state, and a per-PR preview deploy/destroy would attempt to delete the project that trunk-dev shares (see dev-bundle-destroy-disaster memory note). App resources reference projects by path string, which doesn't require bundle ownership.

# NOTE: postgres_projects is NOT in this file by design — see above.
resources:
  apps:
    <svc>_app:
      name: <svc>-${bundle.target}${var.app_name_suffix}
      description: "<one-sentence ownership statement>"
      source_code_path: ../services/<svc>

      # OBO scope the user's forwarded token must carry. The service uses
      # that token to mint Lakebase credentials per request via
      # `WorkspaceClient(token=...).postgres.generate_database_credential(...)`,
      # which requires the `postgres` scope. `iam.*` scopes are platform
      # defaults and are NOT user-grantable.
      user_api_scopes:
        - "postgres"

      # Declaring the postgres app resource:
      #   1. Auto-injects PGHOST / PGUSER / PGDATABASE / PGPORT / PGSSLMODE /
      #      PGAPPNAME at runtime (Apps platform contract — see
      #      docs.databricks.com/aws/en/dev-tools/databricks-apps/lakebase).
      #   2. Creates a Postgres role for this app's service principal with
      #      CONNECT + CREATE on `databricks_postgres` so Alembic migrations
      #      can run.
      # `valueFrom: <svc>_db` in the app's app.yaml resolves to the full
      # endpoint path, used by db.py to mint per-connection OAuth tokens.
      resources:
        - name: <svc>_db
          description: "<svc> service Lakebase Autoscale project."
          postgres:
            branch: projects/<svc>-${bundle.target}/branches/${var.lakebase_branch}
            database: projects/<svc>-${bundle.target}/branches/${var.lakebase_branch}/databases/databricks_postgres
            permission: CAN_CONNECT_AND_CREATE

The bundle deploys the project + production branch + primary endpoint for the service's Lakebase database. Feature branches are out of band — hc-lakebase-branching creates them with the databricks postgres CLI directly, and --var "lakebase_branch=feat-<slug>" rebinds every app's postgres resource to that branch for the deploy.

Why database: .../databricks_postgres

Every Lakebase Autoscale project ships with one default database named databricks_postgres and that's where our schema lives. The app resource needs the full path to that database, including the branch.

If you ever want to host multiple logical databases inside one project, you'd databricks postgres create-database them imperatively (no DAB resource type for that yet) and reference each one in a separate app resource entry.

resources/hc-portal.yml shape

The BFF doesn't talk to a DB itself — only the six backend services do — but it needs (a) OBO scopes so it can forward the user's token and (b) a CAN_USE ACL on each backend app + the URL of each.

resources:
  apps:
    hc_portal_app:
      name: hc-portal-${bundle.target}
      description: "Healthcare clinician portal — frontend + BFF."
      source_code_path: ../frontend/hc-portal

      # No `config.env` block by default — the BFF derives every downstream
      # URL at runtime from the platform-injected `DATABRICKS_APP_URL`
      # (see `frontend/hc-portal/src/hc_portal/clients/_base.py`). The
      # convention is `<svc>-<target>-<wsid>.<region>.databricksapps.com`
      # for every app in this workspace+target.
      #
      # Why not inject URLs from DAB: `${resources.apps.<key>.url}` does
      # NOT resolve at bundle-deploy time (`bundle validate -o json` shows
      # `url=null`), and `bundle deploy` does NOT push `config.env` to the
      # running App spec without an extra `apps update` round-trip.
      #
      # OVERRIDE: when a downstream lives somewhere non-canonical (PR
      # preview, local dev, alternate region), set `<SVC>_SVC_URL` here:
      #
      #   config:
      #     env:
      #       - name: PATIENT_SVC_URL
      #         value: "https://patient-pr-1234-<wsid>.<region>.databricksapps.com"
      #
      # The BFF's `_resolve_base_url` reads `<SVC>_SVC_URL` first, falls
      # back to runtime derivation. Same override can also live in
      # `frontend/hc-portal/app.yml` `env:`.

      # OBO scopes the BFF needs (the six backend services declare only
      # `postgres` for their own Lakebase access). `iam.*` scopes are platform
      # defaults and are NOT user-grantable.
      #   - sql       → portal can query Lakehouse warehouses (saved configs,
      #                 dashboards-style reads, cross-UC joins).
      #   - postgres  → forwarded token can be used by the BFF *and* by
      #                 downstream services to mint Lakebase credentials.
      user_api_scopes:
        - "sql"
        - "postgres"

      # CAN_USE on each backend app: (a) records the dependency so deploys
      # order correctly (services first, portal last); (b) grants the BFF's
      # service principal the platform ACL Apps uses to gate cross-app
      # traffic. The actual URL goes through `config.env` above — the app
      # resource block here is for ACL/wiring, not URL exposure.
      resources:
        - name: patient_app
          description: "Patient service backend."
          app:
            name: ${resources.apps.patient_app.name}
            permission: CAN_USE
        - name: provider_app
          description: "Provider service backend."
          app:
            name: ${resources.apps.provider_app.name}
            permission: CAN_USE
        - name: appointment_app
          app:
            name: ${resources.apps.appointment_app.name}
            permission: CAN_USE
        - name: lab_app
          app:
            name: ${resources.apps.lab_app.name}
            permission: CAN_USE
        - name: prescription_app
          app:
            name: ${resources.apps.prescription_app.name}
            permission: CAN_USE
        - name: billing_app
          app:
            name: ${resources.apps.billing_app.name}
            permission: CAN_USE

How the per-app app.yaml looks

app.yaml is the runtime config — read by the Apps platform when the app starts. It has NO bundle interpolation. Anything that needs DAB-time substitution (${resources.apps.X.url} and friends) must go in the bundle's app.config.env block instead, where it overrides app.yaml.

For the six services, app.yaml is minimal because the Lakebase resource auto-injects PG*:

# services/<svc>/app.yaml
command:
  - "python"
  - "-m"
  - "uvicorn"
  - "<svc>.app:app"
  - "--host"
  - "0.0.0.0"
  - "--port"
  - "8000"

env:
  - name: SERVICE_NAME
    value: "<svc>"

  # PG* env vars (PGHOST/PGUSER/PGDATABASE/PGPORT/PGSSLMODE/PGAPPNAME) are
  # auto-injected by the Apps platform once the `<svc>_db` postgres resource
  # is declared in resources/<svc>.yml.
  #
  # ENDPOINT_NAME is the Lakebase endpoint path used by db.py to mint
  # per-connection OAuth credentials. `valueFrom: <svc>_db` resolves the
  # postgres resource to its endpoint path automatically.
  - name: ENDPOINT_NAME
    valueFrom: "<svc>_db"

For the BFF, app.yml is even simpler — all cross-app URLs are resolved at runtime from DATABRICKS_APP_URL:

# frontend/hc-portal/app.yml
command:
  - "uvicorn"
  - "hc_portal.backend.app:app"
  - "--workers"
  - "2"

env:
  - name: SERVICE_NAME
    value: "hc-portal"

# Per-service URLs are resolved at runtime by `clients/_base.py`:
#   1. Explicit override `<SVC>_SVC_URL` (e.g. `PATIENT_SVC_URL`) if set.
#   2. Otherwise, derived from `DATABRICKS_APP_URL` + the canonical
#      `<svc>-<target>-<workspace_suffix>` host pattern.
# Add `<SVC>_SVC_URL` entries above when the canonical pattern doesn't
# fit (local dev, PR preview, cross-region fan-out).

Why valueFrom looks the way it does

Per the Apps platform, valueFrom: <key> resolves the resource named <key> to a single string. The mapping is: