databricks-solution-builder

name	databricks-solution-builder
description	Generate comprehensive specification files for building Databricks assets, demos or end 2 end projects. Use when users want to create a new demo, design a demo story, or need help structuring demo components, create an entire project. This skill creates prompts that another agent will execute to build the actual demo.

Databricks Demo Generator

A skill for creating and building compelling Databricks demos that Technical Solution Architects will show to enterprise customers across any industry.

Purpose

Generate a coherent demo package — a business story that showcases Databricks capabilities. The story must be compelling (a clear protagonist, challenge, and resolution in $), and the technical components must connect end-to-end (data → pipeline → dashboard → agent all align). One key value of this skill: everything generated is fully coherent — the synthetic data serves the story and works as input for every downstream consumer (dashboards, apps, Genie spaces, agents).

How this skill is organized

The main loop lives in this file (SKILL.md) — it describes the flow: stages, gates, what each stage produces, when to stop for user input. For how to execute a given stage, read the matching DEMO_SKILL_DIR/stages/NN-*.md at the moment you enter it. Keep SKILL.md in your context at all times — pull in a stage file only when you're actively executing that stage.

Stage	What	User gate at end	Execution guide
0. Capture Intent	Understand request, browse domain/pattern blocks, propose story ideas if vague	— (flows into stage 1)	Inline in SKILL.md
1. Design Story	Write resources.json + README.md (batched in one message)	✅ "Approve the story?"	stages/01-design-story.md
2. Write Specs	Write 01-lakeflow.md, then the other top-level specs, then the app spec (if app needed), coherence review	✅ "Ready to build?"	stages/02-write-specs.md
3. Build — pick ONE fork (by mode):
↳ 3.1 Build resources (default)	Provision the Databricks resources via Databricks Agent Skills (DAS)	— (build completes)	stages/03.1-build.md
↳ 3.1r Build on real tables (grounded / "use existing data")	Build DIRECTLY on the user's real UC tables, READ-ONLY — no data generation	— (build completes)	stages/03.1r-build-on-real.md
↳ 3.2 Build a workshop (workshop mode)	Instead of provisioning, generate a notebook workshop (build-it-live via Genie Code)	— (package ready)	stages/03.2-workshop.md
4. Package as a DAB (opt)	On user request only, post-build	—	references/dab/dab.md

Cross-cutting (not a stage):

• App creation — folded into stages 2 + 3: DEMO_SKILL_DIR/app/app.md

Three entry flows

The opening message tells you which of three flows the user started (from the home page's tabs). The stages above are the default; two variants change where the run starts or how Build forks.

A — Build the resources (default). Run stages 0→3.1: design the story, write specs, then provision the real Databricks resources (stages/03.1-build.md). Deployable demo; DAB optional only if the user asks at the end.

"Use existing data" variant (build READ-ONLY on the user's real UC tables). If the opening message says the user picked existing tables (and PROJECT/specifications/source-tables.md exists), that file holds the selected real tables' fully-qualified names + schema (column names, types, comments) — schema only. Read it FIRST. You profile the tables yourself at build time (read-only, off a warehouse) to get distinct counts / ranges / top values and derive the data capability signals (time columns, measures, dimensions, cross-table join keys) — see references/grounding-table-stats.md. If PROJECT/specifications/data-discovery.md also exists, read it too — it records what the user learned exploring these tables during data discovery: the use case they already chose (with its data-fit rationale), the alternatives they set aside, and the capability reasoning. Treat that use case as AGREED SCOPE — build it; do NOT re-pitch it, swap it for another idea, or contradict the fit findings. The demo is built DIRECTLY on those real tables, READ-ONLY — you query them in place; you do NOT generate synthetic data, do NOT copy or re-create them in another catalog, and you NEVER write to, alter, or drop them. Then:

• Derive ONE coherent read-only analytics use case that the real data actually supports. If data-discovery.md records a chosen use case, that IS the one — build it (don't re-derive a different one); the opening message may also include the user's own text and/or a story they picked — honor it. Ground it in the real column names / distributions in source-tables.md; do NOT invent a catalyst or columns the data doesn't have.
• Point every component (dashboards, Genie space, metric views, read-only AI functions) at the exact catalog.schema.table names from source-tables.md. The default capability set is read-only analytics — no ingest/SDP, no write-back app, no Lakebase, no ML training.
• Build forks to 3.1r: do NOT run stages/03.1-build.md; run stages/03.1r-build-on-real.md (verify the real tables are readable, then build the read-only components on them).

Data-write opt-in. If — and only if — the opening message / source-tables.md header says the user opted into letting the demo add its own supporting data, you MAY generate auxiliary tables into the demo's OWN catalog/schema (clearly separated) so write-needing capabilities work. Even then, the user's real tables above remain strictly READ-ONLY — never written to. Absent that opt-in, create no tables at all.

B — Build a workshop The SAME demo, delivered as prompts instead of resources: you generate a package of clean notebooks whose cells are Genie Code prompts that an SA (or customer) pastes into the Databricks Genie Assistant to build the demo live, step by step — raw data (Volume) → SDP → dashboard → Genie space.

• Stages 0–2 UNCHANGED — story, resources.json + README.md, specs, as normal. (Capabilities are pre-scoped to workshop-ready ones — no ML/app/KA/MAS in V1.)
• Build forks to 3.2: do NOT run stages/03.1-build.md; run stages/03.2-workshop.md — notebook workshop + volume-writing data-gen + Genie context, patterned on references/example-luxebeauty-workshop.
• Deliverable = the downloadable package (notebooks + data-gen + specs + context), not a deployed demo. No DAB.

C — Architecture first (alternate start). The user wants to draw the architecture before any story. Their text may be a brief, pasted meeting notes, or a transcript.

• Skip stages 0–1 — no story/resources.json/README.md/specs/build yet.
• Read the databricks-architecture skill (.claude/skills/databricks-architecture/SKILL.md) — flat nodes/edges schema + component catalog + reference diagrams.
• Extract the components the text implies (sources, pipeline, serving, dashboards/apps, agents), map each to a real catalog id. Start from the minimal example in that skill's The format section (or reference/architecture-complete.jsonc for the full platform) when intent matches, then patch in the named sources.
• Write ONLY architecture.md at the project root (schema in a fenced ```json block), then stop with a one-liner inviting review/edit on the Architecture tab. The story comes later — the user clicks "Generate the solution from this architecture", which kicks off stage 1 constrained to the components they kept.

Paths

Your system prompt defines PROJECT, SKILLS, DEMO_SKILL_DIR, and DEMO_SKILL as absolute paths. This skill refers to sibling files like DEMO_SKILL_DIR/stages/*.md, DEMO_SKILL_DIR/app/app.md, DEMO_SKILL_DIR/references/*.

When spawning subagents, substitute every placeholder (DEMO_SKILL_DIR/…, PROJECT/…, SKILLS/…) with its real absolute path before sending — the subagent has no system prompt defining them. The full spawn prompt is in DEMO_SKILL_DIR/stages/03.1-build.md → Step 2.

---

Usage tracking

Each stage fires one tracking event so we can see how the skill is used. Calls are inlined per stage below — run them as a single Bash command, ignore the result, move on. Opt out via DBDEMOS_TRACKER_DISABLED=1.

---

Efficiency

Batch independent tool calls into one message — the harness runs them concurrently, saving LLM round-trips. Load all reference blocks together; write independent files together.

Don't overthink mechanical work (writing files, generating data) — call the real tools and execute, don't narrate when you think.

Subagent policy. A subagent is used in Stage 3.1 (build) for the App — it's the longest task (~5 min) and runs in parallel while the parent builds everything else on main. Stages 0, 1, 2, and 4 run entirely on the main loop, and within Stage 3 only the App is delegated. The full spawn prompt is in DEMO_SKILL_DIR/stages/03.1-build.md → Step 2.

Telling the user where you are

Between phases, drop a one-liner so the user always knows where you are in the flow. Examples:

• "Story approved — writing all specs now (~2 min on the main loop)."
• "Specs ready. Ready to build when you say go."
• "Stage 3 building. Genie + dashboard subagent running, app subagent running. Continuing with governance meanwhile."

No drawn-out status dumps — one line, then the work continues.

Output discipline

Between tool calls, write about the problem, not the file you're about to create. If a sentence describes what the file will contain, put it in the file (comments, code) instead — don't preview it in chat. Don't narrate the act of writing ("writing the script…", "finishing the join…", "still generating…", "Building the materialized view...", Building the query output..." "Writing the daily aggregation view" etc.) — the tool call does that.

When spawning a subagent or handing off context, point at files, don't paraphrase them. Listing absolute paths is cheaper than summarizing what's inside — your paraphrase costs tokens and goes stale, the subagent's read is fast. Don't pre-digest the spec or the README so the subagent "doesn't have to re-read" — that's the wrong economy. Reading is cheap for them; rewriting is expensive for you.

Real thinking (surprising results, tradeoffs, ambiguity, errors) is welcome. File previews and progress updates aren't.

---

Project Structure

./README.md           # Story overview, products showcased, walkthrough
./architecture.md     # Architecture diagram schema (JSON) — built on demand (see Architecture Diagram), not by default
./resources.json      # Selected capabilities + created resource IDs
./specifications/     # Detailed specs per component — the exact files depend on what the demo includes; there is no fixed list

resources.json

Source of truth for what capabilities the demo includes. Created during spec phase with capabilities, updated during build with resource IDs. Structure mirrors DEMO_SKILL_DIR/references/example-luxebeauty/resources.json. You must keep this exact naming convention.

created_resources starts empty {} and grows one key at a time. Add a resource's ID key only after that resource is actually created and validated (Stage 3, build loop step 5). Never pre-seed a key — not with a placeholder like <your-dashboard-uuid>, not with "", not by copying the example file's keys wholesale. The reference example below shows the final shape of a fully-built demo; it is a naming reference, not a scaffold to paste in up front. The UI's resource-link builder renders a clickable link for any present, non-empty ID, so a pre-seeded dashboard_id/genie_space_id becomes a dead link to a resource that doesn't exist yet.

After build (populated with created resource IDs — do not add links here). Use these exact key names; the UI's resource-link builder is wired to them. Skip a section entirely when the demo doesn't include that capability, but do NOT rename keys. Authoritative reference: DEMO_SKILL_DIR/references/example-luxebeauty/resources.json. Lakebase sub-keys are defined in DEMO_SKILL_DIR/app/app.md.

{
  "capabilities": { "buildable": [...], "talking_track": [...] },
  "created_resources": {
    "workspace_folder": "/Workspace/Users/<your-email>/luxebeauty_demo",
    "catalog": "luxebeauty",
    "schema": "demo_c360",
    "warehouse_id": "abc123def456...",
    "pipeline_id": "12ab34cd-5678-...",
    "metric_view_name": "luxebeauty.demo_c360.mv_returns",
    "dashboard_id": "01efab12cd34...",
    "genie_space_id": "abc123...",
    "knowledge_assistant_id": "ka-456...",
    "knowledge_assistant_endpoint": "ka-15956b19-endpoint",
    "multi_agent_supervisor_id": "mas-789...",
    "multi_agent_supervisor_endpoint": "mas-15956b19-endpoint",
    "ml_model_name": "luxebeauty.demo_c360.customer_premium_classifier",
    "mlflow_experiment_path": "/Workspace/Users/<your-email>/luxebeauty/experiments/premium_classifier",
    "app": {
      "name": "luxebeauty-demo",
      "id": "app-luxebeauty-1234",
      "deployment_note": "Deployed via `databricks apps deploy` — see app.md Step 6"
    },
    "lakebase_project_id": "<uid from `databricks postgres get-project | jq -r .uid`>",
    "lakebase_project_slug": "dbdemos-asset-generator",
    "lakebase_database": "dbgen_luxebeauty"
  }
}

Notes on the trickier keys:

• Multiple resources of the same type (e.g. a 2nd dashboard added when iterating). Add another key that ENDS IN THE SAME CANONICAL SUFFIX with a descriptive prefix — e.g. a second dashboard is labor_economics_dashboard_id (NOT dashboard_2 or second_dashboard), a second Genie space is pipeline_health_genie_space_id, a second job is recovery_job_id. The suffix (_dashboard_id, _genie_space_id, _job_id, …) is what the UI link-builder AND the cross-workspace ownership reconcile match on — a key that doesn't end in the canonical suffix is invisible to both, so the resource gets no link/tile and is NEVER re-homed to you (it stays owned by the deployer SP). This is essential on ITERATE turns: when you add a resource to an already-built demo, record it with a suffix-matching key so it's granted to the triggering user like everything else.

• mlflow_experiment_path — required when the demo trains an ML model. Full workspace path passed to mlflow.set_experiment(...). Without it the MLflow Experiment tile never appears in the resources grid (the UI resolves the path → numeric experiment_id via the SDK).

• app is nested (app.name, app.id, app.deployment_note). Record app.name as soon as the app's initial setup is done (scaffold + config) — do NOT wait for deploy. app.name alone marks the app capability "built" in the UI, so a preview-only app that never deploys still counts as complete. Add app.id/app.url later, only after databricks apps deploy. If the deploy fails or is intentionally skipped, keep app.name and put the explanation in deployment_note.

• Lakebase keys are three flat fields, not nested. See app.md for lakebase_setup_db.sh which prints them.

• buildable: capabilities that require actual Databricks resources (pipelines, dashboards, agents, apps, etc.)

• talking_track: capabilities mentioned in the demo narrative but don't require resource creation

• created_resources: filled during build phase — keys match the resource type (e.g., pipeline_id, dashboard_id, genie_space_id)

Capability IDs come from DEMO_SKILL_DIR/references/platform_architecture.md.

When the user provides exact capabilities, use those directly — don't override with pattern suggestions.

Architecture Diagram

Don't write ./architecture.md by default. Build it on demand — when the user asks for an architecture diagram (or starts architecture-first) — always by reading the databricks-architecture skill (.claude/skills/databricks-architecture/SKILL.md).

---

Context Blocks & Platform Architecture

Always start by reading DEMO_SKILL_DIR/references/platform_architecture.md — it shows the complete Databricks platform capabilities with dependencies, all product IDs and categories, and when to use each capability.

DEMO_SKILL_DIR/references/blocks/capabilities details Databricks products — selling points, positioning, how to showcase unique value, common pitfalls. Read them when you need deeper product knowledge for story or spec writing (typically dashboards or ML model training)