| name | dabs-migrator |
| description | Use to scaffold a Databricks Asset Bundle (DABs) project from existing workspace assets. Trigger when the user says "migrate to DABs", "generate a DAB", "convert my job/pipeline to a bundle", or names Databricks resources (jobs, pipelines, dashboards, etc.) and asks for project code. Produces a complete repo — databricks.yml, resources/*.yml per asset, src/ stubs, tests/, requirements.txt, and CI/CD pipeline files for the user's chosen tool (GitHub Actions by default; Azure DevOps, GitLab CI, Bitbucket, Jenkins, CircleCI also supported). |
DABs Migrator
Generates a Databricks Asset Bundle project from a list of Databricks resource names. Outputs a ready-to-deploy repo following the conventions in the canonical DABs example projects (sts-dabs-demo, flights-simple, flights-advanced).
When to invoke this skill
The user wants to bootstrap a DABs project. They will typically:
- name one or more existing Databricks assets (e.g.
@my_job_1, @my_pipeline_1)
- optionally name a CI/CD tool (GitHub Actions, Azure DevOps, GitLab CI, Bitbucket Pipelines, Jenkins, CircleCI)
- optionally name target environments (dev/staging/prod)
If any of these are missing, ask once before generating; default to GitHub Actions and dev/staging/prod if the user is happy with defaults.
Inputs
| Input | Required | Default | Notes |
|---|
| Project name | yes | — | becomes bundle.name and the root folder |
| List of resources | yes | — | each item is <resource_type>:<resource_name> e.g. job:my_job_1. Supported types listed in resources/ — see the official supported-resources table |
| CI/CD tool | no | github-actions | one of: github-actions, azure-devops, gitlab-ci, bitbucket, jenkins, circleci. See cicd/ |
| Targets | no | dev, staging, prod | bundle deployment environments |
| Workspace host(s) | no | ${var.workspace_host} placeholder | one per target, can be filled in later |
Output: project layout
<project_name>/
├── .github/workflows/ # or .azure-pipelines/, .gitlab-ci.yml, etc.
│ ├── deploy_to_staging.yml
│ ├── deploy_to_prod.yml
│ └── pr_validate.yml
├── resources/
│ ├── jobs/<job_name>.yml # one file per job resource
│ ├── pipelines/<pipeline_name>.yml # one file per pipeline resource
│ └── <other_type>/<name>.yml # see resources/ reference
├── src/
│ ├── <job_name>/ # stubs matching the resource key
│ │ └── notebook.py
│ └── <pipeline_name>/
│ ├── bronze.py
│ ├── silver.py
│ └── gold.py
├── tests/
│ └── test_unity_catalog.py # single pytest file for Unity Catalog validation
├── databricks.yml # bundle entrypoint
├── requirements.txt # local dev deps
├── .gitignore
└── README.md
Workflow
- Parse inputs. Extract project name, the
<type>:<name> resource list, and CI/CD tool. Confirm any missing required fields before proceeding.
- Detect mode — fresh start vs. incremental. Check whether
databricks.yml already exists in the target project folder.
- Fresh start (file absent): proceed through all steps below.
- Incremental (file present): the project already exists. Skip steps 3, 5, and 6. Go directly to step 4 for the new resources only. Never overwrite existing files — if a resource file for the named asset already exists, report a conflict and stop for that asset.
- Create root folder named after the project and generate
databricks.yml from templates/databricks.yml.tmpl — fills in bundle.name, include: globs, and targets (dev/staging/prod with ${var.workspace_host} placeholder per target). (Fresh start only.)
- For each resource in the input list, open
resources/<type>.md and use its ## Complete schema reference as the authoritative field catalogue. Build resources/<type_plural>/<name>.yml by mapping the asset's actual existing attributes onto the schema — include only the fields the asset uses, using the correct field names and types from the schema. Do not copy the complete schema verbatim and do not invent placeholder values for fields the asset does not have. If the resource type owns source code (jobs, pipelines, apps, dashboards), also populate src/<name>/:
- If migrating an existing asset (the default case — the user named a real workspace asset): pull the original notebook(s) / script(s) and copy their content verbatim into
src/<name>/. Preserve filenames, structure, comments, and logic exactly. Do not add headers like # Originally sourced from: <path> or replace any block with # TODO: Replace with actual ingestion logic. See the corresponding hard rule below.
- If starting from scratch (only when the user explicitly says so): create minimal stub files and populate only the required fields (marked
REQUIRED in the schema reference).
- Generate CI/CD files from the user's chosen tool's reference under
cicd/<tool>.md. Always emit at least: PR validation pipeline, staging deploy pipeline, prod deploy pipeline. All pipelines must follow the CI/CD action contract below. (Fresh start only.)
- Write supporting files:
requirements.txt (databricks-cli, pytest, ruff baseline), .gitignore (Python + DABs .databricks/), tests/test_unity_catalog.py (copied from templates/test_unity_catalog.py), and a minimal README.md documenting how to deploy. (Fresh start only.)
- Report what was generated: tree of created/modified files. In incremental mode, explicitly list which files were added and confirm that no existing files were touched.
CI/CD action contract
Every generated CI/CD pipeline (regardless of tool) must:
- Install the Databricks CLI — use the official installer/action for the tool (see
cicd/<tool>.md).
- Run
databricks bundle validate --output json — fail the pipeline on non-zero exit. The JSON output should be uploaded as a build artifact when the tool supports it.
- Run
databricks bundle deploy — only on the deploy pipelines, not on PR validation. The target is inferred from the DATABRICKS_BUNDLE_ENV environment variable (never use -t).
Every step must also set BUNDLE_VAR_catalog and BUNDLE_VAR_schema (sourced from CI/CD variables named catalog and schema) so the CLI resolves the bundle variables defined in databricks.yml.
PR validation runs steps 1–2 only with DATABRICKS_BUNDLE_ENV=staging. Staging deploys run 1–3 with DATABRICKS_BUNDLE_ENV=staging. Prod deploys run 1–3 with DATABRICKS_BUNDLE_ENV=prod and require manual approval / protected environment gating.
Reference: Databricks bundle jobs tutorial.
Hard rules — never do
- Never generate, recommend, or run any
databricks repos command. Bundle deploys do not need or use the Repos API; mixing them creates dual sources of truth. If the user asks for Git sync via Repos, refuse and point them at bundle deploys instead.
- Never commit secrets, workspace tokens, or service principal credentials into generated YAML or workflow files. Use the CI tool's secret store (
${{ secrets.* }} for GitHub, variable groups for Azure DevOps, etc.) and Databricks secret scopes ({{secrets/scope/key}}) for runtime.
- Never hardcode workspace hosts, cluster IDs, warehouse IDs, or catalog names in resource YAML. Use bundle variables (
${var.catalog}, ${var.schema}, etc.) and override per target.
- Never put more than one resource definition per YAML file in
resources/. One asset, one file.
- Never deploy to
prod from a dev machine. Prod deploys go through CI only.
- Never use
../src/... in resource YAML paths. Resource files live at resources/<type>/<name>.yml (two levels under the bundle root), so paths to src/ must be ../../src/<name>/.... Using a single .. resolves to resources/src/... and the deploy fails.
- Never mismatch the library entry kind to the source extension in a
pipelines resource's libraries: block. Spark Declarative Pipelines (SDP) require notebook: for .py sources and file: for .sql sources. Mismatch produces Error: expected a file for "resources.pipelines.<name>.libraries[0].file.path" (or the symmetric notebook variant).
- Never use
target: on new pipelines. SDP uses schema: — target is the legacy DLT field and is deprecated.
- Never replace migrated source files with placeholder, stub, sample, or "blueprint" code. When migrating an existing Databricks asset (job, pipeline, app, dashboard, or any resource that owns code), clone the original notebook/script/file content verbatim into
src/<name>/. Do not insert # TODO: Replace with actual ingestion logic..., # Originally sourced from: <path> headers, or simplified example code. The user's working logic must survive the migration intact — anything else silently breaks production behavior. Only generate stub files when the user explicitly states they are starting from scratch (no original asset to migrate).
Supported resource types
One reference doc per resource type lives under resources/. Read the relevant file when generating the YAML for that resource. Full list (from the Databricks supported-resources table):
CI/CD tools
Templates
Example interactions
Fresh start
User: "I want to migrate @my_job_1 and @my_pipeline_1 to DABs, generate the project and for the CI/CD tool use GitHub Actions."
Output: create the tree at the top of this file, with:
resources/jobs/my_job_1.yml from resources/jobs.md following the complete schema available fields.resources/pipelines/my_pipeline_1.yml from resources/pipelines.md following the complete schema available fields.src/my_job_1/notebook.py and src/my_pipeline_1/{bronze,silver,gold}.py stubs.github/workflows/{deploy_to_staging,deploy_to_prod,pr_validate}.yml from cicd/github-actions.mddatabricks.yml, requirements.txt, .gitignore, README.md, tests/test_unity_catalog.py
Report the tree back and list TODOs (workspace host, CI auth secrets, fill in actual notebook logic).
Incremental (project already exists)
User: "Add @my_alert_1 to the project."
Output: detect that databricks.yml already exists, enter incremental mode, and only create:
resources/alerts/my_alert_1.yml (using the asset's actual attributes mapped against resources/alerts.md)
All existing files — databricks.yml, CI/CD workflows, other resource YAMLs, src/ directories, tests/ — are left untouched. Report only the newly added files.
