Menu
Profile Databricks data, design data-quality tests appropriate to what each column actually is, then build and deploy a Databricks DQ testing pipeline to Orchestra.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Builds Orchestra pipeline YAML using the native Data Reconciliation tasks (DATA_RECONCILIATION_MANUAL_QUERY, DATA_RECONCILIATION_CURSOR_FIELD) to prove two data stores match — the "did the migration land correctly" check between a source and destination integration (SNOWFLAKE, SQL_SERVER, DATABRICKS, any pairing). Use whenever the user wants to compare, validate, or reconcile data across two systems for a migration, replatform, cutover, or CDC/replication setup — phrases like "make sure the migration matches", "reconcile Snowflake and Databricks", "validate the cutover", "did we lose any rows moving to the new warehouse", "set up drift monitoring", or any mention of Data Reconciliation / DataRec. Produces a one-off full-match validation pipeline plus, optionally, an ongoing cursor-field drift monitor. Don't use create-orchestra-pipeline for this — Data Reconciliation tasks have sharp edges (single-scalar results, thresholds that silently no-op if omitted) covered in this skill's references.
Create, validate, and remediate Orchestra pipeline YAML files. Use when asked to build a new pipeline, add tasks to an existing pipeline, fix pipeline validation errors, or author Orchestra workflow definitions from a description. Trigger on phrases like "create a pipeline", "add a dbt task", "write orchestra yaml", "fix validate errors", or when editing files under orchestra/ or similar pipeline directories.
Profile BigQuery data, design data-quality tests appropriate to what each column actually is, then build and deploy a BigQuery DQ testing pipeline to Orchestra.
Profile ClickHouse data, design data-quality tests appropriate to what each column actually is, then build and deploy a ClickHouse DQ testing pipeline to Orchestra.
Profile Snowflake data, design data-quality tests appropriate to what each column actually is, then build and deploy a Snowflake DQ testing pipeline to Orchestra.
Finds Orchestra pipelines that are really the same process duplicated — across environments (`_prod`/`_staging`/`_dev`/`-uat` naming, or Orchestra's native Environments) or conceptually (same task graph copy-pasted per customer/region/business-unit under unrelated names) — then drafts one consolidated pipeline using Environment overlays, `${{ ENV.* }}`, inputs, conditionals, or a MetaEngine matrix instead of the duplication. Use when the user wants to "consolidate", "merge", "dedupe", or "unify" pipelines, asks "why do I have three copies of this pipeline", or wants to act on an account-health-check finding about one-pipeline-per-environment, duplicated tasks, or hardcoded environment values. Also trigger when handed an existing account-review report flagging those. Always shows evidence and a drafted YAML before touching anything, and asks per duplicate set whether to report, create/PR the unified pipeline, or also pause the originals — never merges, deletes, or pauses without that go-ahead.
| name | write-databricks-dq-tests |
| description | Profile Databricks data, design data-quality tests appropriate to what each column actually is, then build and deploy a Databricks DQ testing pipeline to Orchestra. |
Goal: inspect real Databricks data, design tests that fit what each column means (not a generic null check on everything), deploy them as an Orchestra pipeline, run it, and report what's actually wrong. A test that fails because the data is bad is the skill working correctly — do not tune thresholds until everything is green.
Flow: profile → design tests → write pipeline YAML → branch (or create pipeline if no git) → register → run → report.
../../references/orchestra/dq-tests/databricks.md — Databricks profiling SQL, the test
catalogue in Databricks SQL, the pipeline YAML, and Databricks-specific error causes.../../references/orchestra/dq-tests/workflow.md — the engine-agnostic rest of the workflow:
thresholds, the matrix/gating pattern, branching, registering, triggering, and how to interpret
results. Applies unchanged to Databricks.databricks.md §1–2.databricks.md §3, following the gating pattern and threshold
rules in workflow.md.workflow.md, using the Databricks-specific error causes and qualified-name
format from databricks.md §4 when interpreting results.