| name | sdp-meta |
| description | Use this skill for metadata-driven Databricks pipelines built with SDP-META (the framework in this repo; formerly DLT-META, package databricks-labs-sdp-meta, CLI `databricks labs sdp-meta`). Triggers: (1) 'sdp-meta', 'sdp meta', 'dlt-meta', 'dlt meta', or 'metadata-driven pipeline'; (2) onboarding a dataflowspec / writing an onboarding JSON/YAML that describes bronze/silver tables as metadata; (3) generating many bronze/silver Lakeflow SDP pipelines from config instead of hand-writing pipeline code; (4) `sdp-meta onboard`, `sdp-meta deploy`, or `sdp-meta bundle-init/bundle-add-flow/bundle-validate`; (5) scaffolding an sdp-meta Asset Bundle (DAB). This is the config-driven LAYER ON TOP OF Lakeflow Spark Declarative Pipelines — for hand-authored streaming tables / materialized views, that is plain SDP, not this framework. |
SDP-META — Metadata-Driven Lakeflow Pipelines
Overview
SDP-META (the framework in this repository) is a metadata-driven layer that
builds Databricks Lakeflow Spark Declarative Pipelines (SDP) from
configuration instead of per-table pipeline code. You describe each
source→bronze→silver flow once as a dataflowspec (an onboarding JSON/YAML
record); the framework generates and runs the bronze and silver SDP pipelines.
Add a table by adding a metadata entry — not by writing another pipeline.
- Package:
databricks-labs-sdp-meta (formerly dlt-meta).
- CLI:
databricks labs sdp-meta <command>.
- Canonical imports:
from databricks.labs.sdp_meta.dataflow_pipeline import DataflowPipeline
(v0.0.10 from src.* still resolves via the bundled compat shim).
- Relationship to plain SDP: SDP-META generates SDP pipelines. Use this
skill when the user wants config-driven fan-out across many tables/layers;
hand-authored streaming tables / materialized views are plain SDP.
When to use: the user has (or wants) an onboarding file describing many
tables, mentions sdp-meta/dlt-meta, or wants to onboard/deploy/scaffold a
metadata-driven pipeline. When NOT to use: a one-off pipeline with a couple
of hand-written tables.
The mental model (3 steps)
- Author an onboarding file: a list of dataflowspec records, one per source
table, describing bronze (and optionally silver) targets, reader options,
data-quality expectations, and transformations.
- Onboard (
sdp-meta onboard): reads the onboarding file and writes the
dataflowspec into Delta tables (the bronze/silver dataflowspec tables).
Metadata only — no data pipeline runs yet.
- Deploy (
sdp-meta deploy): creates a Lakeflow SDP pipeline whose runner
notebook reads the dataflowspec tables and materializes bronze/silver tables.
For infrastructure-as-code, bundle-init scaffolds a Databricks Asset Bundle
(DAB) wiring the onboarding job + the SDP pipeline + variables together.
Quick start
databricks labs sdp-meta onboard --profile <profile>
databricks labs sdp-meta deploy --profile <profile>
Prefer a bundle for anything beyond a one-off:
databricks labs sdp-meta bundle-init --quickstart=true --output-dir=.
databricks labs sdp-meta bundle-validate
databricks bundle deploy && databricks bundle run
Use the = syntax on boolean flags (--quickstart=true,
--build-and-upload-whl=true) — the Labs CLI string-flag parser otherwise
consumes the next token as the value.
Real, working configs live in this repo under examples/
(json/, yml/, and the sdp_meta_pipeline*.ipynb notebooks) and in
tests/resources/ (onboarding*.json,
silver_transformations*.json, dqe/). Copy from these rather than inventing
field names.
How agents drive SDP-META
Two complementary surfaces:
- MCP tools (scaffolding + inspection, no live workspace needed): the
framework ships its own stdio MCP server (
databricks labs sdp-meta mcp,
requires the mcp extra) with 5 tools — sdp_meta_bundle_init,
sdp_meta_bundle_add_flow, sdp_meta_bundle_validate,
sdp_meta_list_templates, sdp_meta_get_onboarding_template. See
references/mcp-tools.md.
- CLI (
onboard/deploy/bundle-*): the operational path that touches a
live workspace. See references/cli-and-bundles.md.
Recommended flow: scaffold + add flows + validate locally with the MCP tools (or
bundle-init), then hand off to CLI deploy for the live run.
Onboarding file essentials
A dataflowspec record keys targets by layer and environment. Environment-suffixed
fields (_dev / _staging / _prd) let one spec serve all environments.
{
"data_flow_id": "100",
"data_flow_group": "A1",
"source_format": "cloudFiles",
"source_details": { "source_database": "APP", "source_table": "CUSTOMERS",
"source_path_dev": "/Volumes/.../customers" },
"bronze_catalog_dev": "bronze_cl", "bronze_database_dev": "bronze",
"bronze_table": "customers",
"bronze_reader_options": { "cloudFiles.format": "json", "cloudFiles.inferColumnTypes": "true" },
- Data quality (
*_data_quality_expectations_json_*): expect_or_drop
(quarantine bad rows), expect_or_fail (halt), expect (warn).
- Silver transformations (
silver_transformation_json_*): a list of
{ target_table, select_exp[], where_clause[] }.
- CDC / SCD Type 2:
silver_cdc_apply_changes (and bronze CDC options).
Full field reference: references/onboarding-spec.md.
Reference files
Common issues
| Issue | Solution |
|---|
| Boolean flag "consumed" the next argument | Use = syntax: --quickstart=true, --build-and-upload-whl=true. |
deploy fails: no dataflowspec found | Run onboard first — deploy reads the dataflowspec Delta tables onboard writes. |
| Pipeline cluster can't reach PyPI to install the wheel | Use --build-and-upload-whl=true (or --whl-file-path=/Volumes/...) so the wheel is staged on a UC Volume and baked into the runner's %pip install. |
Serverless pipeline rejects a whl library | Serverless DLT does not support whl-typed pipeline libraries; the wheel is delivered via the runner notebook's %pip install, not a pipeline library. |
bundle-validate flags layer/topology or placeholder errors | Fix resources/variables.yml (catalog/schema/sdp_meta_dependency) and unresolved onboarding placeholders before databricks bundle deploy. |
