databricks-architecture

name

databricks-architecture

description

Create or edit a Databricks solution-architecture diagram — a Lucidchart-style platform diagram (sources → Lakeflow/Genie pipeline → lakehouse/lakebase → dashboards, Genie, apps → end user, on a governed platform). Use when someone wants to draw, generate, or edit an architecture diagram, visualize a Databricks solution's components and data flow, or produce an architecture image. Produces a self-contained HTML you render to a PNG and iterate on.

Databricks Architecture Diagram

Section map (jump to the one canonical home for each topic):

• Designing from a request → Step 0 read the request's shape · Step 0.5 where extra context goes (desc/note/label/ai_reasoning) · Step 1 which diagram kind · Step 2 how much to infer · Step 2.5 how to draw the data layer.
• Workflow → copy a renderer HTML, render to PNG, iterate.
• The format → the JSON schema, worked example, Tabs, Positioning (placement resolution order), Columns/rowGrid, Relational placement, field tables (A node / An edge), Containers, physical layout, Annotations, Custom logos.
• Component catalog → the per-component wiring: map (the authoritative edge guide) + icon bank + Sources.
• Authoring rules → the terse do/don't checklist. Reference files → the .jsonc examples.

A Databricks solution architecture, drawn to fit what the request actually describes. That might be:

• a left→right flow (sources → pipeline → compute → dashboards/Genie/apps), the common shape for a single use-case; OR
• a layered platform — Databricks added ALONGSIDE an existing stack, or an ask written as explicit LAYERS (ingestion / governance / intelligence / agents / apps) → the layers organize the diagram (usually into lanes); OR
• a governance / infra containment picture (UC hierarchy, or cloud/VPC boxes).

Do NOT force every request into the sources→serve funnel. The funnel is one shape among several — match the diagram to the request's own structure (see Designing the architecture → Step 0). A prompt that lists layers, names external (non-Databricks) systems, or asks for an integration seam wants the layered shape, not the funnel.

The architecture must ultimately make sense. A user describes what they want in their own words and often leaves out the connective tissue — if they name dashboards and agents but no data layer or compute, ADD what's needed so the story holds together; a diagram missing its ingestion / compute / governance backbone isn't right. Respect what they asked for and keep their emphasis. Equally, don't over-engineer: if a detail wasn't asked for and doesn't help the story, leave it out. Aim for the smallest architecture that is complete and correct for THIS ask.

Overarching principle — fidelity scales with how specific the ask is. Two goals are in tension: do exactly what the user said and make the architecture make sense. Which wins depends on how precise and how recent the instruction is:

• A loose first description (a use-case, pasted notes, a rough vision — often hand-wavy, sometimes not 100% internally consistent) → your job is interpretive: honor the intent and emphasis as much as possible, but fill the gaps and quietly reconcile contradictions so the result holds together. Here make-it-make-sense leads. Don't refuse or flag every inconsistency — resolve it into a coherent diagram.
• A specific instruction ("connect X to Y", "put the model upstream", "drop the dashboard", "these three are one block") → respect it literally, even if you'd have drawn it differently. Precise beats your sense of a tidier architecture.
• A follow-up correction is the STRONGEST signal there is. When the user edits or corrects the diagram, do exactly that and do NOT silently revert it toward what seems cleaner to you — if their change looks odd, keep it and record the why in ai_reasoning, don't overrule it. Later + more-specific always wins over earlier + vaguer (and over your own defaults).

The rule of thumb: the vaguer the ask, the more you shape it; the more specific or corrective the ask, the more you obey it verbatim.

The file is a flat list of nodes + edges — a node is on the canvas iff it's in nodes (no visibility/state/diffing). Author STRUCTURE, not pixels: place with col/row lanes, relational fields, wraps boxes, and pin banners — you almost never write at. These compose in a fixed resolution order — full rules in Positioning below.

Edges follow the catalog wiring: lines. Each component's wiring: line (in the Component catalog) is the authoritative map of what it consumes/depends on: a plain entry is the normal edge (draw it when both components are present), one marked (optional) is added when the story calls for it. They're the typical connections, not an exhaustive whitelist — a genuine edge the story needs is fine; just don't invent a connection a component wouldn't actually have. Read the relevant rows before wiring.

Avoid floating components. A participating component — one that's IN the data flow — should connect to something; by default it doesn't sit on the canvas unwired. sql-lakehouse reads from the medallion; ai-bi-dashboard/genie read from the lakehouse; an app reads from lakebase/features and fronts a model; lakebase syncs from the pipeline and powers the app. If you place one of these and can't trace an edge into or out of it (following its wiring: line), you either forgot the edge or the tile doesn't belong — fix one or the other. The exception is GLOBAL / cross-cutting things that are ABOUT the whole architecture, not a step in it: a type:"note"/text annotation, the db-platform banner, a wrapping box, and platform-spanning governance (unity-catalog/governance-block) legitimately have no data-flow edge — they frame the diagram rather than participate in it. But a consumption/app/state tile (ai-bi-dashboard, databricks-apps, lakebase, genie, sql-lakehouse, a model) is NOT global — it should be wired. This is the failure to watch for in wide multi-lane layouts: those tiles land far from the data they read and get left stranded. Wiring is a design step, not an afterthought: after placing tiles, trace the flow end-to-end — walk each source through the pipeline to a consumer/entry point and draw every hop. When a global/governing tile DOES draw an edge, it attaches to what it actually governs — ai-gateway to MODEL calls (model-serving, apps), unity-catalog to data — never chained through whatever tile happens to sit next to it.

Designing the architecture from a request (architecture-first)

Design a coherent, functionally-complete architecture that solves the ask from the user's request (their prompt / pasted text) — not from any pre-existing file or default, not a word-for-word transcription, not a fixed template.

Step 0 — READ THE REQUEST'S OWN STRUCTURE FIRST (do this before anything)

The request usually TELLS you the shape. Extract it before you pick components:

• Does it list LAYERS / SECTIONS? ("Layer 1 — ingestion, Layer 2 — governance, Layer 3 — intelligence, …", or numbered/bulleted sections.) → Use the layered shape (not the funnel), honoring the request's order and names. Layers often map to left→right lanes, but a layer→column mapping is a starting point, not a rule: a governance/cross-cutting layer usually SPANS the others (a pinned bar/band, not a lane), thin adjacent layers can share a lane, and a layer can run top→bottom. Choose the fewest lanes that keep the flow legible and the connected tiles near each other — many sparse lanes push consumers away from the data they read and leave them stranded. Do NOT collapse a rich multi-layer ask into the 5-tile funnel — that's the #1 failure mode. When the ask doesn't dictate its own order, a sane default spine is sources → data processing → domain + governance → compute → ML / agentic → apps → Genie One — a starting skeleton only; the ask's own structure and names always win. Keep each layer coherent — group same-role tiles into the same lane rather than scattering them (e.g. compute sits as one layer between data and the consumers). This is a preference, not a rule: a layered ask mentioning related tiles in different sentences isn't a reason to split them across lanes unless the ask actually wants them separated.
• Does it name EXTERNAL (non-Databricks) systems? (AWS Lambda/Kinesis/S3, an existing app, a third-party bus, "keep X, add Databricks alongside".) → Draw the external side in its OWN boundary box and Databricks in ANOTHER; the gap between them is the integration seam. Show it both ways if the ask mentions callbacks/bi-directional (subscribe IN + call APIs back OUT). Use file:cloud/<provider>/… logos for the external side.
• Does it map future-state to current gaps, name reference customers, or carry other context? → Those are type:"note" post-its by default (a text/box annotation only when you specifically want a plain caption or a titled badge), not core components — place them near what they describe (beside the relevant tile, or below/beside the diagram), just don't let them overlap or crowd the tiles.

Honor the structure the user handed you — if they wrote the layers/sections, use those names and order.

Step 0.5 — where does the EXTRA CONTEXT go? (keep the diagram dense)

A rich request carries far more detail than there are components. Don't drop it, and don't inflate the diagram with it — route each piece to the right lightweight slot, keeping every line short so the picture stays scannable:

• Context ABOUT a component (what a source/domain/table holds, what a tile is for) → a desc line under it (title + one short subtitle, like "Lakebase" / "Managed Postgres for app state"). Works on catalog tiles (override only when the default can't say it), and on source and logo nodes (set desc + it shows; a logo desc is a muted 2nd caption line). One tight phrase, not a sentence — the tile stays compact and the layout reserves room for it. If it needs a paragraph, it's not a desc.
• Context from the REQUEST that isn't about any one component (a rationale, a "keep X / we'll add Y", future-state wins, reference customers, an SLA, a caveat, a team/ownership fact) → a VISIBLE type:"note" post-it, placed beside/below the component it relates to. This is the DEFAULT for request context — if the user wrote it and it matters, it belongs on the canvas as a note, not compressed into a tile desc and NEVER parked in the hidden ai_reasoning field. Lean toward MORE notes: add one per distinct point rather than cramming several into one or dropping them. (Use a plain text/box annotation only for a genuine caption/badge; never invent a fake component.)
• Context about a CONNECTION (why A feeds B, what flows, an order/step, a non-obvious hop) → a short edge label drawn ON the edge. Use it whenever the relationship isn't self-evident from position — but keep it to a few words ("Subscribe (streams)", "Query shared context", "Trigger APIs after decision") so edges stay legible and the diagram dense.
• Your OWN reasoning you want to keep but NOT show (why you chose a handle, a "don't add X" caution, why a row/col) → the node/edge ai_reasoning field — it round-trips verbatim and never renders (see rule 10). This is ONLY for your authoring rationale, never for request content the user should see — that goes to a visible post-it above.

The bar: every component and edge that isn't obvious gets a short label or desc; everything narrative goes to a note/annotation; nothing becomes a bogus tile, and no line runs long. Dense and self-explanatory beats sparse-but-cryptic or cluttered-with-paragraphs.

Step 1 — which KIND of diagram is this?

Pick the shape that fits what Step 0 found. Most SINGLE-use-case asks are (1); a multi-layer / "alongside an existing stack" ask is (2).

1. Flow (solution / demo) — a left→right DATA-FLOW story: a few sources (default ~4, see Sources) → Lakeflow+Genie → lakehouse/Lakebase → dashboard/Genie/app → Genie One → user. The common shape for ONE use-case ("predictive maintenance", "customer 360") or a data+AI feature ask ("Lakeflow Connect + SDP → a model endpoint"). Uses the catalog tiles + composites; how much to infer → Step 2; which reference(s) to learn from → Pick a starting point. Layout conventions: one box wraps the whole flow (usually not the raw sources) = "the Databricks Platform" (auto-renders behind its children — see Containers); db-platform + governance-block pin to its top-left/top-right (never a raw at); Genie One fronts the consumption tiles with auto-arrows.
2. Layered platform / integration — the request is organized as layers, and/or Databricks sits alongside an existing (non-Databricks) system. The layers organize the diagram in the ask's order (usually lanes, but see Step 0 — governance spans, thin layers merge, pick the fewest legible lanes); external systems get their own boundary box with an integration seam to the Databricks boundary box; governance (Unity Catalog) SPANS the layers as a pinned top bar or full-height band, not one lane tile. Build it from the ask's actual layers (boundary boxes + seam + layers-as-columns). This is the shape for enterprise / "add Databricks to our stack" asks — do NOT reduce it to the funnel. Logical DOMAINS / knowledge areas (a "five-domain context", a shared semantic layer, a set of business subject areas any agent can query) → draw each as a type:"logo" with icon:"file:vendor/genie-ontology" (the Genie Ontology mark = logical domain / knowledge), wrapped in a box titled for the layer.
3. Physical / governance — the Unity Catalog HIERARCHY: workspace → metastore → catalogs → schemas → tables. A containment picture (nested boxes), NOT a flow. Trigger: the ask is about UC objects / org structure. Use the container-box presets — see Databricks physical layout below.
4. Infra / networking — cloud/account topology: VPC / subnets / PrivateLink. Also containment (nested cloud boxes, the Containers pattern). Trigger: the ask is about networking / deployment / cloud accounts.

If the ask blends kinds (e.g. "the layered flow, inside our VPC"), compose them.

Step 2 — (solution/demo only) how much to infer

• Broad / use-case ask ("predictive maintenance", "fraud detection", "a governed data platform"): infer the full end-to-end shape the use-case implies. Predictive maintenance is a MODEL story → sensors/history → pipeline → ML training → registry → serving → scoring + dashboards. Don't under-scope to the couple of nouns typed — add the components that make it actually work.
• Named-component ask ("Lakeflow Connect + SDP into UC, served by a model endpoint"): assemble exactly those into a working whole — honor what they named + add the connective tissue that wires it (platform box, ingest ports, an entry point, the edges). Don't over-build past what they asked, don't under-build to isolated unconnected tiles.

Either way, components that belong together, wired so the flow reads correctly, on the governed platform.

Don't copy a reference verbatim — adapt it to THIS ask. References are for inspiration. Read the reference(s) to learn the PATTERN (which components connect, how the layout/handles/pins work), then build a fresh diagram for THIS ask. Read as many references as are relevant — most real asks mix several: take the ML platform's serving lane, add the agent-bricks supervisor, drop the governance bar on top, wrap it in the layered shape's boundary boxes, swap the sources. Pick whichever references inform the ask, learn from each, and compose your own. A reference matches the ask 1:1 only rarely — even then, adapt it (names, sources, which components) rather than reproduce it. See Pick a starting point for what each offers.

Step 2.5 — pick how to draw the DATA / INGEST layer (3 ways)

The bronze→silver→gold data layer can be drawn three ways. Pick by what the story emphasizes — don't default to one:

1. lakeflow-genie-block (or lakeflow-block) — the big unified block. Shows the whole ingest story in ONE block: Connect · Zerobus · raw-file landing on the left rail, the SDP medallion (bronze→silver→gold), Delta/Iceberg, and (genie variant) a "Built with Genie Code" footer. Use when ingestion IS part of the story — you want to showcase how data lands + gets processed, all products in one tidy block. Optional bronze_desc/silver_desc/gold_desc params add a short caption under each layer.

   { "id": "data", "type": "lakeflow-genie-block", "col": "pipeline",
     "params": { "gold_desc": "Business marts + metrics" } }
   

2. medallion-table — the simpler block. Just bronze→silver→gold in one compact tile, no ingest rail. Use when ingestion is NOT the focus but you still need the medallion — especially when you want the Feature Store / Metric Views forks off gold (params:{feature_store,metric_views}, wired via @out-fs/@out-mv/@out-gold). Same *_desc layer-caption params.

   { "id": "med", "type": "medallion-table", "col": "pipeline",
     "params": { "feature_store": true, "metric_views": true } }
   

3. DIY — compose it yourself. When you want to go into DETAIL and list the actual tables: a type:"box" per layer (title "Bronze"/"Silver"/"Gold") wrapping type:"logo" tiles (icon bronzeLayer/silverLayer/goldLayer, text = the table name). Tiles use col/row inside the box's column; each box needs no col/pad/size/z — it auto-sizes around its tiles (z is automatic — see Containers). Flow edges box→box. Optionally wrap all three in a parent box titled "Lakeflow Spark Declarative Pipelines" with the SDP logo (titleIcon:"sdpBrand") — a box titleIcon accepts ANY icon-library key.

   { "id": "b1", "type": "logo", "icon": "bronzeLayer", "text": "orders_raw",   "caption": "right", "col": "bronze", "row": 1 },
   { "id": "b2", "type": "logo", "icon": "bronzeLayer", "text": "events_raw",   "caption": "right", "col": "bronze", "row": 2 },
   { "id": "s1", "type": "logo", "icon": "silverLayer", "text": "orders",       "caption": "right", "col": "silver", "row": 1 },
   { "id": "s2", "type": "logo", "icon": "silverLayer", "text": "events",       "caption": "right", "col": "silver", "row": 2 },
   { "id": "g1", "type": "logo", "icon": "goldLayer",   "text": "customer_360", "caption": "right", "col": "gold",   "row": 1,
     "ai_reasoning": "only add ai_reasoning when there's a real authoring choice to record — don't invent" },
   { "id": "bronze-box", "type": "box", "title": "Bronze", "wraps": ["b1", "b2"] },