dabs-migrator

# DABs Migrator > [!WARNING] > **Paused for staging and prod use (2026-10-26).** Projects generated by this version are not safe to deploy to staging or prod, or to run in CI/CD, as generated. Migrated assets are not bound, stateful resources have no destroy protection, `dev` can point at production data, and the CI/CD templates have authentication and gating gaps. See the [open dabs-migrator issues](https://github.com/databricks-solutions/technical-services-solutions/issues?q=is%3Aissue+is%3Aopen+dabs-migrator) (#106, #107, #108 and follow-ups). > > **Agent instructions until a fixed version is released:** > 1. Before generating anything, show the user this notice and the issues link. > 2. Generate the `dev` target only. Do not generate `staging` or `prod` targets or any CI/CD pipeline files, even if asked. Explain why and point the user to the issues link. > 3. Run `databricks bundle deploy -t dev` only after the user confirms the `dev` target uses non-production catalogs and schemas, paused schedules, and no production `run_as` identities or pipeline/job IDs. Never run `bundle deployment bind` or `bundle destroy`. Generates a Databricks Asset Bundle project from a list of Databricks resource names. Outputs a ready-to-deploy repo following the conventions in the canonical DABs example projects ([sts-dabs-demo](https://github.com/databricks-solutions/databricks-dab-examples/tree/main/sts-dabs-demo), [flights-simple](https://github.com/databricks-solutions/databricks-dab-examples/tree/main/flights/flights-simple), [flights-advanced](https://github.com/databricks-solutions/databricks-dab-examples/tree/main/flights/flights-advanced)). ## When to invoke this skill The user wants to bootstrap a DABs project. They will typically: - name one or more existing Databricks assets (e.g. `@my_job_1`, `@my_pipeline_1`) - optionally name a CI/CD tool (GitHub Actions, Azure DevOps, GitLab CI, Bitbucket Pipelines, Jenkins, CircleCI) - optionally name target environments (dev/staging/prod) If any of these are missing, ask once before generating; default to GitHub Actions and `dev`/`staging`/`prod` if the user is happy with defaults. ## Inputs | Input | Required | Default | Notes | |---|---|---|---| | Project name | yes | — | becomes `bundle.name` and the root folder | | List of resources | yes | — | each item is `<resource_type>:<resource_name>` e.g. `job:my_job_1`. Supported types listed in `resources/` — see the [official supported-resources table](https://docs.databricks.com/aws/en/dev-tools/bundles/resources#supported-resources) | | CI/CD tool | no | `github-actions` | one of: `github-actions`, `azure-devops`, `gitlab-ci`, `bitbucket`, `jenkins`, `circleci`. See `cicd/` | | Targets | no | `dev`, `staging`, `prod` | bundle deployment environments | | Workspace host(s) | no | `${var.workspace_host}` placeholder | staging and prod only, can be filled in later. `dev` has no host (it uses the developer's auth profile) | | Source target | yes | none | which target the source workspace represents (`dev`, `staging` or `prod`). Values read from the live assets fill only that target | | Dev catalog | yes | none | catalog the `dev` target writes to. Must differ from the prod catalog; the dev schema is `dev_${workspace.current_user.short_name}` | ## Output: project layout ``` <project_name>/ ├── .github/workflows/ # or .azure-pipelines/, .gitlab-ci.yml, etc. │ ├── deploy_to_staging.yml │ ├── deploy_to_prod.yml │ └── pr_validate.yml ├── resources/ │ ├── jobs/<job_name>.yml # one file per job resource │ ├── pipelines/<pipeline_name>.yml # one file per pipeline resource │ └── <other_type>/<name>.yml # see resources/ reference ├── src/ │ ├── <job_name>/ # stubs matching the resource key │ │ └── notebook.py │ └── <pipeline_name>/ │ ├── bronze.py │ ├── silver.py │ └── gold.py ├── tests/ │ └── test_unity_catalog.py # single pytest file for Unity Catalog validation ├── databricks.yml # bundle entrypoint ├── requirements.txt # local dev deps ├── .gitignore └── README.md ``` ## Workflow 1. **Parse inputs.** Extract project name, the `<type>:<name>` resource list, and CI/CD tool. Confirm any missing required fields before proceeding. Always ask which target the source workspace represents and which catalog `dev` should use. Stop if the dev catalog and schema would equal the production ones. 2. **Detect mode — fresh start vs. incremental.** Check whether `databricks.yml` already exists in the target project folder. - **Fresh start** (file absent): proceed through all steps below. - **Incremental** (file present): the project already exists. Skip steps 3, 5, and 6. Go directly to step 4 for the new resources only. Never overwrite existing files — if a resource file for the named asset already exists, report a conflict and stop for that asset. 3. **Create root folder** named after the project and **generate `databricks.yml`** from `templates/databricks.yml.tmpl` — fills in `bundle.name`, `include:` globs, and `targets` (dev/staging/prod). Set `catalog`, `schema` and every other environment value under `targets.<target>.variables` for each target: the source values go only into the source target, and the other targets get the values the user gave (or a placeholder the user must replace). Never give `catalog` or `schema` a top-level default. *(Fresh start only.)* 4. **For each resource** in the input list, open `resources/<type>.md` and use its `## Complete schema reference` as the authoritative field catalogue. Build `resources/<type_plural>/<name>.yml` by mapping the asset's **actual existing attributes** onto the schema — include only the fields the asset uses, using the correct field names and types from the schema. Do not copy the complete schema verbatim and do not invent placeholder values for fields the asset does not have. If the resource type owns source code (jobs, pipelines, apps, dashboards), also populate `src/<name>/`: - **If migrating an existing resource** (the default case — the user named a real workspace asset): pull the source code used by the resource (original notebook(s) / script(s) / SQL files) and copy their content **verbatim** into `src/<name>/`. Preserve filenames, structure, comments, and logic exactly. Do **not** add headers like `# Originally sourced from: <path>` or replace any block with `# TODO: Replace with actual ingestion logic`. See the corresponding hard rule below. - **If starting from scratch** (only when the user explicitly says so): create minimal stub files and populate only the required fields (marked `REQUIRED` in the schema reference). 5. **Generate CI/CD files** from the user's chosen tool's reference under `cicd/<tool>.md`. Always emit at least: PR validation pipeline, staging deploy pipeline, prod deploy pipeline. All pipelines must follow the **CI/CD action contract** below. *(Fresh start only.)* 6. **Write supporting files**: `requirements.txt` (databricks-cli, pytest, ruff baseline), `.gitignore` (Python + DABs `.databricks/`), `tests/test_unity_catalog.py` (copied from `templates/test_unity_catalog.py`), and a minimal `README.md` documenting how to deploy. *(Fresh start only.)* 7. **Report** what was generated: tree of created/modified files. In incremental mode, explicitly list which files were added and confirm that no existing files were touched. ## Reading assets out of the workspace Step 4 depends on clean live reads. Most "the read came back empty, retry" loops are not the asset's fault — they are the read command. Follow these to avoid the retries: 1. **Never pipe `-o json` through `2>&1`.** A wrapped/proxied CLI (RTK and similar) prints a banner to stderr; `2>&1 | jq` merges it into the JSON and `jq` dies with `parse error: Invalid numeric literal at line 1, column 6`. Redirect stderr away *before* `jq`, or save to a file first: ```bash databricks jobs list -p <profile> -o json 2>/dev/null | jq '...' # or, more robust for multi-step reads: databricks jobs list -p <profile> -o json 2>/dev/null > /tmp/jobs.json ``` Reading to a file once (then querying the file) also avoids re-issuing the same call and is the safest pattern when a proxy is in play. 2. **Resolve `<name>` → ID with the right per-type command, then read the detail.** The user names assets; the APIs key on IDs. Resolve first: | Resource | List command | Match on | ID field | Detail read | |---|---|---|---|---| | job | `jobs list` | `.settings.name` | `job_id` | `jobs get <id>` | | pipeline | `pipelines list-pipelines` | `.name` | `pipeline_id` | `pipelines get <id>` | | cluster | `clusters list` | `.cluster_name` | `cluster_id` | `clusters get <id>` | | sql_warehouse | `warehouses list` | `.name` | `id` | `warehouses get <id>` | | alert | `alerts-v2 list-alerts` | `.display_name` | `id` | `alerts-v2 get-alert <id>` | | dashboard | `lakeview list` | `.display_name` | `dashboard_id` | `lakeview get <id>` | | genie_space | `genie list-spaces` (`.spaces[]`) | `.title` | `space_id` | see rule 3 | 3. **Genie `serialized_space` is NOT returned by `genie get-space`.** That subcommand returns only `title`/`description`/`warehouse_id`/`parent_path` — copying from it silently produces an empty space. Get the body with either: ```bash databricks bundle generate genie-space --existing-id <space_id> --key <name> # writes <name>.geniespace.json # or the raw API with the include flag: databricks api get "/api/2.0/genie/spaces/<space_id>?include_serialized_space=true" -p <profile> 2>/dev/null ``` The dashboard equivalent (`lakeview get <id>`) *does* include `serialized_dashboard` inline — no extra flag needed. ## CI/CD action contract Every generated CI/CD pipeline (regardless of tool) must: 1. **Install the Databricks CLI** — use the official installer/action for the tool (see `cicd/<tool>.md`). 2. **Run `databricks bundle validate --output json`** — fail the pipeline on non-zero exit. The JSON output should be uploaded as a build artifact when the tool supports it. 3. **Run `databricks bundle deploy`** — only on the deploy pipelines, not on PR validation. Every bundle command passes the target explicitly with `-t <target>`. Also set `DATABRICKS_BUNDLE_ENV` to the same target and keep the two in sync. Without either, the CLI silently selects the `default: true` target (`dev`). CI never sets `BUNDLE_VAR_catalog`, `BUNDLE_VAR_schema` or any other `BUNDLE_VAR_*` for environment values. `BUNDLE_VAR_*` overrides `targets.<target>.variables`, so a shared CI value points every target at the same tables. Catalog, schema and other non-secret environment values are committed per target in `databricks.yml`; CI holds only the host and the service principal credentials. PR validation runs steps 1–2 only with `-t staging`. Staging deploys run 1–3 with `-t staging`. Prod deploys run 1–3 with `-t prod` and require manual approval / protected environment gating. Reference: [Databricks bundle jobs tutorial](https://docs.databricks.com/aws/en/dev-tools/bundles/jobs-tutorial). ## Hard rules — never do - **Never** generate, recommend, or run any `databricks repos` command. Bundle deploys do not need or use the Repos API; mixing them creates dual sources of truth. If the user asks for Git sync via Repos, refuse and point them at bundle deploys instead. - **Never** commit secrets, workspace tokens, or service principal credentials into generated YAML or workflow files. Use the CI tool's secret store (`${{ secrets.* }}` for GitHub, variable groups for Azure DevOps, etc.) and Databricks secret scopes (`{{secrets/scope/key}}`) for runtime. - **Never** hardcode workspace hosts, cluster IDs, warehouse IDs, catalog or schema names in resource YAML. Use bundle variables (`${var.catalog}`, `${var.schema}`, etc.) and override per target. - **Never** copy the concrete IDs a live read returns straight into resource YAML. Reading an existing asset returns *resolved* values — `warehouse_id: 3d885699...`, `cluster_id`, `instance_pool_id`, and fully-qualified `catalog.schema.table` (including inside SQL strings such as an alert's `query_text` or a dashboard dataset query). Replace each with the matching bundle variable (`${var.warehouse_id}`, `${var.catalog}`, `${var.schema}`, …) and declare it. This is the hardcode rule above applied to the migration read path — `alerts`, `dashboards`, `genie_spaces`, and `quality_monitors` are the usual offenders because they all carry a `warehouse_id`. - **Never** copy an environment value from a live read into a target it does not belong to. Every catalog, schema, warehouse ID, cluster ID, pipeline ID, job ID, dashboard ID, model, serving endpoint, volume or storage path, secret scope, principal and notification recipient is a bundle variable set per target, or a bundle reference such as `${resources.pipelines.<key>.id}`. The live read's values fill only the source target. - **Never** give `catalog` or `schema` a top-level default, and never let `dev` resolve to the production catalog or schema. Each target sets both under `targets.<target>.variables`; `dev` uses the user's dev catalog and `dev_${workspace.current_user.short_name}`. Stop generation if dev and prod would match. - **Never** copy a literal `UNPAUSED` schedule, trigger or alert status. A source that is active uses `${var.schedule_pause_status}`, which is `PAUSED` in `dev`. A source that is paused stays `PAUSED` everywhere. - **Never** copy `run_as` from a live read. Only the `staging` and `prod` targets set `run_as`, from `${var.staging_sp_app_id}` / `${var.prod_sp_app_id}`. `dev` runs as the developer. - **Never** let `dev` notify production recipients: notification recipients use `${var.notification_emails}`, which is `[]` in `dev`. - **Never** set `BUNDLE_VAR_catalog`, `BUNDLE_VAR_schema` or any other `BUNDLE_VAR_*` for environment values in generated CI. `BUNDLE_VAR_*` overrides the per-target values. - **Never** reference a `${var.<name>}` in `databricks.yml` without declaring it in a `variables:` block. The generated `databricks.yml` wires `workspace.host: ${var.workspace_host}` and `run_as.service_principal_name: ${var.staging_sp_app_id}` / `${var.prod_sp_app_id}`; all three must be declared under top-level `variables:` (see `templates/databricks.yml.tmpl`). An undeclared reference fails with `Error: reference does not exist: ${var.staging_sp_app_id}` on `bundle validate -t staging`/`-t prod` — and `-t dev` usually passes because it doesn't use them, so the bug ships silently until someone validates staging/prod. - **Never** put more than one resource definition per YAML file in `resources/`. One asset, one file. - **Never** deploy to `prod` from a dev machine. Prod deploys go through CI only. - **Never** use `../src/...` in resource YAML paths. Resource files live at `resources/<type>/<name>.yml` (two levels under the bundle root), so paths to `src/` must be `../../src/<name>/...`. Using a single `..` resolves to `resources/src/...` and the deploy fails. - **Never** mismatch the library entry kind to the source extension in a `pipelines` resource's `libraries:` block. Spark Declarative Pipelines (SDP) require `notebook:` for `.py` sources and `file:` for `.sql` sources. Mismatch produces `Error: expected a file for "resources.pipelines.<name>.libraries[0].file.path"` (or the symmetric notebook variant). - **Never** use `target:` on new pipelines. SDP uses `schema:` — `target` is the legacy DLT field and is deprecated.