spark-sandbox

End-to-end interactive workflow to ship a Power BI report from a Delta Lake. Studies source data, builds dbt dims/facts, iterates on a matplotlib PBI-style mockup with the user, lays down semantic-model relationships + DAX measures (TMDL), and finally emits PBIR visual JSON so the report renders in Fabric. Skips the Power BI UI almost entirely.

Fix a failing CI run end-to-end: download logs, diagnose failures, apply code fixes, run local tests, push, poll CI, and iterate until green.

Run the spark-dbt regression testing loop. Iteratively install, lint, run, and test dbt projects against a local Spark environment until all models and tests pass.

Interactive workflow to design a Kimball STAR-schema dbml file from a directory of local Delta tables and a list of business questions. Profiles the source data with DuckDB, proposes conformed + local dims and fact tables, validates that every business question can be answered on the proposed model, surfaces bonus insights from sample data, and emits a committed authoring-style .dbml as the design spec for a new dbt project. The dbml is the only artefact that lands in git.

Interactive workflow to create Bronze-to-Silver Spark transformations. Discovers source data, catalogs existing transformers, validates schema design with the user, generates code (Constants, Loader, Transformer, Driver), registers tables for VACUUM, creates unit tests, and validates via spark-submit.

Create unit test YAML definitions that mock upstream model inputs and validate expected outputs for dbt models. Use when adding unit tests for a dbt model in projects/spark-dbt/ or practicing TDD on a new dim_* / fct_* before wiring it into the DAG. Includes the dbt-fabricspark Spark data-type caveat.

Retrieve and search dbt / dbt-fabricspark documentation in LLM-friendly Markdown. Use when looking up dbt configs (materializations, snapshots, sources, tests, contracts), dbt-fabricspark adapter options, or any docs.getdbt.com page.

Build and modify dbt models, write SQL transformations using ref() and source(), create tests, and validate results with dbt show. Use when doing any dbt work in projects/spark-dbt/ — building / modifying models, debugging errors, exploring sources, writing tests, or evaluating downstream impact. Tuned for Kimball STAR schema authoring on dbt-fabricspark.

Run the spark-scala regression testing loop. Iteratively compile, build, test, and validate Spark Drivers against a local VSCode Devcontainer environment until all tests are green.

Address PR review comments end-to-end: study comments, make code changes, run regression loops, push, poll CI, and resolve threads.