databricks-solution-builder

# Databricks Demo Generator A skill for creating and building compelling Databricks demos that Technical Solution Architects will show to enterprise customers across any industry. ## Purpose Generate a **coherent demo package** — a business story that showcases Databricks capabilities. The story must be compelling (a clear protagonist, challenge, and resolution in $), and the technical components must connect end-to-end (data → pipeline → dashboard → agent all align). One key value of this skill: everything generated is fully coherent — the synthetic data serves the story and works as input for every downstream consumer (dashboards, apps, Genie spaces, agents). ## How this skill is organized The main loop lives in this file (SKILL.md) — it describes **the flow**: stages, gates, what each stage produces, when to stop for user input. For **how to execute** a given stage, read the matching `DEMO_SKILL_DIR/stages/NN-*.md` at the moment you enter it. Keep SKILL.md in your context at all times — pull in a stage file **only** when you're actively executing that stage. | Stage | What | User gate at end | Execution guide | |-------|------|------------------|-----------------| | **0. Capture Intent** | Understand request, browse domain/pattern blocks, propose story ideas if vague | — (flows into stage 1) | Inline in SKILL.md | | **1. Design Story** | Write `resources.json` + `README.md` (batched in one message) | ✅ *"Approve the story?"* | `stages/01-design-story.md` | | **2. Write Specs** | Write `01-lakeflow.md`, then the other top-level specs, then the app spec (if app needed), coherence review | ✅ *"Ready to build?"* | `stages/02-write-specs.md` | | **3. Build** — *pick ONE fork (by mode):* | | | | | &nbsp;&nbsp;↳ **3.1 Build resources** *(default)* | Provision the Databricks resources via Databricks Agent Skills (DAS) | — (build completes) | `stages/03.1-build.md` | | &nbsp;&nbsp;↳ **3.1r Build on real tables** *(grounded / "use existing data")* | Build DIRECTLY on the user's real UC tables, **READ-ONLY** — no data generation | — (build completes) | `stages/03.1r-build-on-real.md` | | &nbsp;&nbsp;↳ **3.2 Build a workshop** *(workshop mode)* | Instead of provisioning, generate a notebook workshop (build-it-live via Genie Code) | — (package ready) | `stages/03.2-workshop.md` | | **4. Package as a DAB** (opt) | On user request only, post-build | — | `references/dab/dab.md` | **Cross-cutting (not a stage):** - **App creation** — folded into stages 2 + 3: `DEMO_SKILL_DIR/app/app.md` ### Three entry flows The opening message tells you which of three flows the user started (from the home page's tabs). The stages above are the **default**; two variants change where the run starts or how Build forks. **A — Build the resources (default).** Run stages 0→3.1: design the story, write specs, then **provision the real Databricks resources** (`stages/03.1-build.md`). Deployable demo; DAB optional only if the user asks at the end. > **"Use existing data" variant (build READ-ONLY on the user's real UC tables).** If the opening message says the user picked existing tables (and **`PROJECT/specifications/source-tables.md` exists**), that file holds the selected real tables' **fully-qualified names + schema (column names, types, comments)** — schema only. **Read it FIRST.** You **profile the tables yourself** at build time (read-only, off a warehouse) to get distinct counts / ranges / top values and derive the data capability signals (time columns, measures, dimensions, cross-table join keys) — see `references/grounding-table-stats.md`. If **`PROJECT/specifications/data-discovery.md`** also exists, **read it too** — it records what the user learned exploring these tables during data discovery: the use case they **already chose** (with its data-fit rationale), the alternatives they set aside, and the capability reasoning. Treat that use case as **AGREED SCOPE** — build it; do NOT re-pitch it, swap it for another idea, or contradict the fit findings. The demo is built **DIRECTLY on those real tables, READ-ONLY** — you query them **in place**; you do **NOT** generate synthetic data, do **NOT** copy or re-create them in another catalog, and you **NEVER** write to, alter, or drop them. Then: > - Derive **ONE** coherent **read-only analytics** use case that the real data actually supports. If `data-discovery.md` records a chosen use case, **that IS the one** — build it (don't re-derive a different one); the opening message may also include the user's own text and/or a story they picked — honor it. Ground it in the real column names / distributions in `source-tables.md`; do NOT invent a catalyst or columns the data doesn't have. > - Point **every** component (dashboards, Genie space, metric views, read-only AI functions) at the exact `catalog.schema.table` names from `source-tables.md`. The default capability set is read-only analytics — no ingest/SDP, no write-back app, no Lakebase, no ML training. > - **Build forks to 3.1r:** do NOT run `stages/03.1-build.md`; run **`stages/03.1r-build-on-real.md`** (verify the real tables are readable, then build the read-only components on them). > > **Data-write opt-in.** If — and only if — the opening message / `source-tables.md` header says the user opted into letting the demo add its own supporting data, you MAY generate **auxiliary** tables into the demo's **OWN** catalog/schema (clearly separated) so write-needing capabilities work. **Even then, the user's real tables above remain strictly READ-ONLY — never written to.** Absent that opt-in, create no tables at all. **B — Build a workshop** The SAME demo, delivered as prompts instead of resources: you generate a **package of clean notebooks whose cells are Genie Code prompts** that an SA (or customer) pastes into the Databricks Genie Assistant to build the demo **live, step by step** — raw data (Volume) → SDP → dashboard → Genie space. - **Stages 0–2 UNCHANGED** — story, `resources.json` + `README.md`, specs, as normal. (Capabilities are pre-scoped to workshop-ready ones — no ML/app/KA/MAS in V1.) - **Build forks to 3.2:** do NOT run `stages/03.1-build.md`; run **`stages/03.2-workshop.md`** — notebook workshop + volume-writing data-gen + Genie context, patterned on **`references/example-luxebeauty-workshop`**. - **Deliverable = the downloadable package** (notebooks + data-gen + specs + context), not a deployed demo. No DAB. **C — Architecture first** *(alternate start).* The user wants to draw the architecture before any story. Their text may be a brief, pasted meeting notes, or a transcript. - **Skip stages 0–1** — no story/`resources.json`/`README.md`/specs/build yet. - Read the **`databricks-architecture` skill** (`.claude/skills/databricks-architecture/SKILL.md`) — flat `nodes`/`edges` schema + component catalog + reference diagrams. - **Extract the components** the text implies (sources, pipeline, serving, dashboards/apps, agents), map each to a real catalog id. Start from the minimal example in that skill's **The format** section (or `reference/architecture-complete.jsonc` for the full platform) when intent matches, then patch in the named sources. - **Write ONLY `architecture.md`** at the project root (schema in a fenced ```json block), then **stop** with a one-liner inviting review/edit on the Architecture tab. The story comes later — the user clicks "Generate the solution from this architecture", which kicks off stage 1 *constrained to* the components they kept. ## Paths Your system prompt defines `PROJECT`, `SKILLS`, `DEMO_SKILL_DIR`, and `DEMO_SKILL` as absolute paths. This skill refers to sibling files like `DEMO_SKILL_DIR/stages/*.md`, `DEMO_SKILL_DIR/app/app.md`, `DEMO_SKILL_DIR/references/*`. **When spawning subagents**, substitute every placeholder (`DEMO_SKILL_DIR/…`, `PROJECT/…`, `SKILLS/…`) with its real absolute path before sending — the subagent has no system prompt defining them. The full spawn prompt is in `DEMO_SKILL_DIR/stages/03.1-build.md` → Step 2. --- ## Usage tracking Each stage fires one tracking event so we can see how the skill is used. Calls are inlined per stage below — run them as a single Bash command, ignore the result, move on. Opt out via `DBDEMOS_TRACKER_DISABLED=1`. --- ## Efficiency Batch independent tool calls into one message — the harness runs them concurrently, saving LLM round-trips. Load all reference blocks together; write independent files together. Don't overthink mechanical work (writing files, generating data) — call the real tools and execute, don't narrate when you think. **Subagent policy.** A subagent is used in Stage 3.1 (build) **for the App** — it's the longest task (~5 min) and runs in parallel while the parent builds everything else on main. Stages 0, 1, 2, and 4 run entirely on the main loop, and within Stage 3 only the App is delegated. The full spawn prompt is in `DEMO_SKILL_DIR/stages/03.1-build.md` → Step 2. ### Telling the user where you are Between phases, drop a one-liner so the user always knows where you are in the flow. Examples: - *"Story approved — writing all specs now (~2 min on the main loop)."* - *"Specs ready. Ready to build when you say go."* - *"Stage 3 building. Genie + dashboard subagent running, app subagent running. Continuing with governance meanwhile."* No drawn-out status dumps — one line, then the work continues. ### Output discipline Between tool calls, write about the **problem**, not the file you're about to create. If a sentence describes what the file will contain, put it in the file (comments, code) instead — don't preview it in chat. Don't narrate the act of writing ("writing the script…", "finishing the join…", "still generating…", "Building the materialized view...", Building the query output..." "Writing the daily aggregation view" etc.) — the tool call does that. When spawning a subagent or handing off context, **point at files, don't paraphrase them.** Listing absolute paths is cheaper than summarizing what's inside — your paraphrase costs tokens and goes stale, the subagent's read is fast. Don't pre-digest the spec or the README so the subagent "doesn't have to re-read" — that's the wrong economy. Reading is cheap for them; rewriting is expensive for you. Real thinking (surprising results, tradeoffs, ambiguity, errors) is welcome. File previews and progress updates aren't. --- ## Project Structure ``` ./README.md # Story overview, products showcased, walkthrough ./architecture.md # Architecture diagram schema (JSON) — built on demand (see Architecture Diagram), not by default ./resources.json # Selected capabilities + created resource IDs ./specifications/ # Detailed specs per component — the exact files depend on what the demo includes; there is no fixed list ``` ### resources.json Source of truth for what capabilities the demo includes. Created during spec phase with capabilities, updated during build with resource IDs. Structure mirrors `DEMO_SKILL_DIR/references/example-luxebeauty/resources.json`. You must keep this exact naming convention. **`created_resources` starts empty `{}` and grows one key at a time.** Add a resource's ID key **only after that resource is actually created and validated** (Stage 3, build loop step 5). **Never pre-seed a key** — not with a placeholder like `<your-dashboard-uuid>`, not with `""`, not by copying the example file's keys wholesale. The reference example below shows the *final* shape of a fully-built demo; it is a naming reference, **not a scaffold to paste in up front**. The UI's resource-link builder renders a clickable link for any present, non-empty ID, so a pre-seeded `dashboard_id`/`genie_space_id` becomes a dead link to a resource that doesn't exist yet. **After build** (populated with created resource IDs — do not add links here). **Use these exact key names**; the UI's resource-link builder is wired to them. Skip a section entirely when the demo doesn't include that capability, but do NOT rename keys. Authoritative reference: `DEMO_SKILL_DIR/references/example-luxebeauty/resources.json`. Lakebase sub-keys are defined in `DEMO_SKILL_DIR/app/app.md`. ```json { "capabilities": { "buildable": [...], "talking_track": [...] }, "created_resources": { "workspace_folder": "/Workspace/Users/<your-email>/luxebeauty_demo", "catalog": "luxebeauty", "schema": "demo_c360", "warehouse_id": "abc123def456...", "pipeline_id": "12ab34cd-5678-...", "metric_view_name": "luxebeauty.demo_c360.mv_returns", "dashboard_id": "01efab12cd34...", "genie_space_id": "abc123...", "knowledge_assistant_id": "ka-456...", "knowledge_assistant_endpoint": "ka-15956b19-endpoint", "multi_agent_supervisor_id": "mas-789...", "multi_agent_supervisor_endpoint": "mas-15956b19-endpoint", "ml_model_name": "luxebeauty.demo_c360.customer_premium_classifier", "mlflow_experiment_path": "/Workspace/Users/<your-email>/luxebeauty/experiments/premium_classifier", "app": { "name": "luxebeauty-demo", "id": "app-luxebeauty-1234", "deployment_note": "Deployed via `databricks apps deploy` — see app.md Step 6" }, "lakebase_project_id": "<uid from `databricks postgres get-project | jq -r .uid`>", "lakebase_project_slug": "dbdemos-asset-generator", "lakebase_database": "dbgen_luxebeauty" } } ``` Notes on the trickier keys: - **Multiple resources of the same type (e.g. a 2nd dashboard added when iterating).** Add another key that ENDS IN THE SAME CANONICAL SUFFIX with a descriptive prefix — e.g. a second dashboard is `labor_economics_dashboard_id` (NOT `dashboard_2` or `second_dashboard`), a second Genie space is `pipeline_health_genie_space_id`, a second job is `recovery_job_id`. The suffix (`_dashboard_id`, `_genie_space_id`, `_job_id`, …) is what the UI link-builder AND the cross-workspace ownership reconcile match on — a key that doesn't end in the canonical suffix is invisible to both, so the resource gets no link/tile and is NEVER re-homed to you (it stays owned by the deployer SP). This is essential on ITERATE turns: when you add a resource to an already-built demo, record it with a suffix-matching key so it's granted to the triggering user like everything else. - **`mlflow_experiment_path`** — required when the demo trains an ML model. Full workspace path passed to `mlflow.set_experiment(...)`. Without it the MLflow Experiment tile never appears in the resources grid (the UI resolves the path → numeric experiment_id via the SDK). - **`app` is nested** (`app.name`, `app.id`, `app.deployment_note`). **Record `app.name` as soon as the app's initial setup is done (scaffold + config) — do NOT wait for deploy.** `app.name` alone marks the app capability "built" in the UI, so a preview-only app that never deploys still counts as complete. Add `app.id`/`app.url` later, only after `databricks apps deploy`. If the deploy fails or is intentionally skipped, keep `app.name` and put the explanation in `deployment_note`. - **Lakebase keys are three flat fields**, not nested. See `app.md` for `lakebase_setup_db.sh` which prints them. - **buildable**: capabilities that require actual Databricks resources (pipelines, dashboards, agents, apps, etc.) - **talking_track**: capabilities mentioned in the demo narrative but don't require resource creation - **created_resources**: filled during build phase — keys match the resource type (e.g., `pipeline_id`, `dashboard_id`, `genie_space_id`) Capability IDs come from `DEMO_SKILL_DIR/references/platform_architecture.md`. When the user provides exact capabilities, use those directly — don't override with pattern suggestions. ### Architecture Diagram Don't write `./architecture.md` by default. Build it **on demand** — when the user asks for an architecture diagram (or starts architecture-first) — always by reading the **`databricks-architecture` skill** (`.claude/skills/databricks-architecture/SKILL.md`). --- ## Context Blocks & Platform Architecture **Always start by reading `DEMO_SKILL_DIR/references/platform_architecture.md`** — it shows the complete Databricks platform capabilities with dependencies, all product IDs and categories, and when to use each capability. `DEMO_SKILL_DIR/references/blocks/capabilities` details Databricks products — selling points, positioning, how to showcase unique value, common pitfalls. Read them when you need deeper product knowledge for story or spec writing (typically dashboards or ML model training) ---