databricks-architecture

# Databricks Architecture Diagram **Section map** (jump to the one canonical home for each topic): - *Designing from a request* → **Step 0** read the request's shape · **Step 0.5** where extra context goes (`desc`/`note`/`label`/`ai_reasoning`) · **Step 1** which diagram kind · **Step 2** how much to infer · **Step 2.5** how to draw the data layer. - *Workflow* → copy a renderer HTML, render to PNG, iterate. - *The format* → the JSON schema, worked example, **Tabs**, **Positioning** (placement resolution order), **Columns/rowGrid**, **Relational placement**, field tables (**A node** / **An edge**), **Containers**, **physical layout**, **Annotations**, **Custom logos**. - *Component catalog* → the per-component `wiring:` map (the authoritative edge guide) + icon bank + **Sources**. - *Authoring rules* → the terse do/don't checklist. *Reference files* → the `.jsonc` examples. A Databricks solution architecture, drawn to fit **what the request actually describes**. That might be: - a left→right **flow** (sources → pipeline → compute → dashboards/Genie/apps), the common shape for a single use-case; OR - a **layered platform** — Databricks added ALONGSIDE an existing stack, or an ask written as explicit LAYERS (ingestion / governance / intelligence / agents / apps) → the layers organize the diagram (usually into lanes); OR - a **governance / infra** containment picture (UC hierarchy, or cloud/VPC boxes). **Do NOT force every request into the sources→serve funnel.** The funnel is one shape among several — match the diagram to the request's own structure (see *Designing the architecture* → Step 0). A prompt that lists layers, names external (non-Databricks) systems, or asks for an integration seam wants the **layered** shape, not the funnel. **The architecture must ultimately make sense.** A user describes what they want in their own words and often leaves out the connective tissue — if they name dashboards and agents but no data layer or compute, ADD what's needed so the story holds together; a diagram missing its ingestion / compute / governance backbone isn't right. Respect what they asked for and keep their emphasis. Equally, don't over-engineer: if a detail wasn't asked for and doesn't help the story, leave it out. Aim for the **smallest architecture that is complete and correct for THIS ask.** **Overarching principle — fidelity scales with how specific the ask is.** Two goals are in tension: *do exactly what the user said* and *make the architecture make sense*. Which wins depends on how precise and how recent the instruction is: - **A loose first description** (a use-case, pasted notes, a rough vision — often hand-wavy, sometimes not 100% internally consistent) → your job is **interpretive**: honor the intent and emphasis as much as possible, but fill the gaps and quietly reconcile contradictions so the result holds together. Here *make-it-make-sense* leads. Don't refuse or flag every inconsistency — resolve it into a coherent diagram. - **A specific instruction** ("connect X to Y", "put the model upstream", "drop the dashboard", "these three are one block") → **respect it literally**, even if you'd have drawn it differently. Precise beats your sense of a tidier architecture. - **A follow-up correction is the STRONGEST signal there is.** When the user edits or corrects the diagram, do exactly that and do NOT silently revert it toward what seems cleaner to you — if their change looks odd, keep it and record the why in `ai_reasoning`, don't overrule it. Later + more-specific always wins over earlier + vaguer (and over your own defaults). The rule of thumb: **the vaguer the ask, the more you shape it; the more specific or corrective the ask, the more you obey it verbatim.** The file is a **flat list of `nodes` + `edges`** — a node is on the canvas iff it's in `nodes` (no visibility/state/diffing). **Author STRUCTURE, not pixels:** place with `col`/`row` lanes, relational fields, `wraps` boxes, and `pin` banners — you almost never write `at`. These compose in a fixed resolution order — full rules in *Positioning* below. **Edges follow the catalog `wiring:` lines.** Each component's `wiring:` line (in the *Component catalog*) is the authoritative map of what it consumes/depends on: a plain entry is the **normal** edge (draw it when both components are present), one marked *(optional)* is added when the story calls for it. They're the typical connections, not an exhaustive whitelist — a genuine edge the story needs is fine; just don't invent a connection a component wouldn't actually have. **Read the relevant rows before wiring.** **Avoid floating components.** A **participating** component — one that's IN the data flow — **should** connect to something; by default it doesn't sit on the canvas unwired. `sql-lakehouse` reads from the medallion; `ai-bi-dashboard`/`genie` read from the lakehouse; an app reads from `lakebase`/features and fronts a model; `lakebase` syncs from the pipeline and powers the app. If you place one of these and can't trace an edge into or out of it (following its `wiring:` line), you either forgot the edge or the tile doesn't belong — fix one or the other. The **exception is GLOBAL / cross-cutting things that are ABOUT the whole architecture, not a step in it**: a `type:"note"`/`text` annotation, the `db-platform` banner, a wrapping `box`, and platform-spanning governance (`unity-catalog`/`governance-block`) legitimately have no data-flow edge — they frame the diagram rather than participate in it. But a consumption/app/state tile (`ai-bi-dashboard`, `databricks-apps`, `lakebase`, `genie`, `sql-lakehouse`, a model) is NOT global — it should be wired. This is the failure to watch for in wide multi-lane layouts: those tiles land far from the data they read and get left stranded. **Wiring is a design step, not an afterthought:** after placing tiles, trace the flow end-to-end — walk each source through the pipeline to a consumer/entry point and draw every hop. When a global/governing tile DOES draw an edge, it attaches to what it actually governs — `ai-gateway` to MODEL calls (`model-serving`, apps), `unity-catalog` to data — never chained through whatever tile happens to sit next to it. ## Designing the architecture from a request (architecture-first) Design a **coherent, functionally-complete architecture that solves the ask** from **the user's request** (their prompt / pasted text) — not from any pre-existing file or default, not a word-for-word transcription, not a fixed template. ### Step 0 — READ THE REQUEST'S OWN STRUCTURE FIRST (do this before anything) The request usually TELLS you the shape. Extract it before you pick components: - **Does it list LAYERS / SECTIONS?** ("Layer 1 — ingestion, Layer 2 — governance, Layer 3 — intelligence, …", or numbered/bulleted sections.) → Use the **layered shape** (not the funnel), honoring the request's order and names. Layers often map to left→right lanes, but a layer→column mapping is a starting point, not a rule: a governance/cross-cutting layer usually SPANS the others (a pinned bar/band, not a lane), thin adjacent layers can share a lane, and a layer can run top→bottom. Choose the fewest lanes that keep the flow legible and the connected tiles near each other — many sparse lanes push consumers away from the data they read and leave them stranded. Do NOT collapse a rich multi-layer ask into the 5-tile funnel — that's the #1 failure mode. When the ask *doesn't* dictate its own order, a sane default spine is **sources → data processing → domain + governance → compute → ML / agentic → apps → Genie One** — a starting skeleton only; the ask's own structure and names always win. **Keep each layer coherent** — group same-role tiles into the same lane rather than scattering them (e.g. compute sits as one layer between data and the consumers). This is a preference, not a rule: a layered ask mentioning related tiles in different sentences isn't a reason to split them across lanes unless the ask actually wants them separated. - **Does it name EXTERNAL (non-Databricks) systems?** (AWS Lambda/Kinesis/S3, an existing app, a third-party bus, "keep X, add Databricks alongside".) → Draw the external side in its OWN boundary `box` and Databricks in ANOTHER; the gap between them is the **integration seam**. Show it **both ways** if the ask mentions callbacks/bi-directional (subscribe IN + call APIs back OUT). Use `file:cloud/<provider>/…` logos for the external side. - **Does it map future-state to current gaps, name reference customers, or carry other context?** → Those are **`type:"note"` post-its by default** (a `text`/`box` annotation only when you specifically want a plain caption or a titled badge), not core components — place them near what they describe (beside the relevant tile, or below/beside the diagram), just don't let them overlap or crowd the tiles. **Honor the structure the user handed you** — if they wrote the layers/sections, use those names and order. ### Step 0.5 — where does the EXTRA CONTEXT go? (keep the diagram dense) A rich request carries far more detail than there are components. Don't drop it, and don't inflate the diagram with it — route each piece to the right lightweight slot, keeping every line **short** so the picture stays scannable: - **Context ABOUT a component** (what a source/domain/table holds, what a tile is for) → a **`desc`** line under it (title + one short subtitle, like "Lakebase" / "Managed Postgres for app state"). Works on catalog tiles (override only when the default can't say it), and on **`source`** and **`logo`** nodes (set `desc` + it shows; a logo desc is a muted 2nd caption line). **One tight phrase, not a sentence** — the tile stays compact and the layout reserves room for it. If it needs a paragraph, it's not a `desc`. - **Context from the REQUEST that isn't about any one component** (a rationale, a "keep X / we'll add Y", future-state wins, reference customers, an SLA, a caveat, a team/ownership fact) → a **VISIBLE `type:"note"` post-it, placed beside/below the component it relates to. This is the DEFAULT for request context** — if the user wrote it and it matters, it belongs on the canvas as a note, not compressed into a tile `desc` and NEVER parked in the hidden `ai_reasoning` field. **Lean toward MORE notes:** add one per distinct point rather than cramming several into one or dropping them. (Use a plain `text`/`box` annotation only for a genuine caption/badge; never invent a fake component.) - **Context about a CONNECTION** (why A feeds B, what flows, an order/step, a non-obvious hop) → a short **edge `label`** drawn ON the edge. Use it whenever the relationship isn't self-evident from position — but keep it to a few words ("Subscribe (streams)", "Query shared context", "Trigger APIs after decision") so edges stay legible and the diagram dense. - **Your OWN reasoning you want to keep but NOT show** (why you chose a handle, a "don't add X" caution, why a row/col) → the node/edge **`ai_reasoning`** field — it round-trips verbatim and never renders (see rule 10). This is ONLY for your authoring rationale, never for request content the user should see — that goes to a visible post-it above. The bar: every component and edge that isn't obvious gets a short label or `desc`; everything narrative goes to a note/annotation; nothing becomes a bogus tile, and no line runs long. Dense and self-explanatory beats sparse-but-cryptic or cluttered-with-paragraphs. ### Step 1 — which KIND of diagram is this? Pick the shape that fits what Step 0 found. Most SINGLE-use-case asks are (1); a multi-layer / "alongside an existing stack" ask is (2). 1. **Flow (solution / demo)** — a left→right DATA-FLOW story: a few sources (default ~4, see *Sources*) → Lakeflow+Genie → lakehouse/Lakebase → dashboard/Genie/app → Genie One → user. The common shape for ONE use-case ("predictive maintenance", "customer 360") or a data+AI feature ask ("Lakeflow Connect + SDP → a model endpoint"). Uses the catalog **tiles** + composites; how much to infer → Step 2; which reference(s) to learn from → *Pick a starting point*. **Layout conventions:** one `box` `wraps` the whole flow (usually not the raw sources) = "the Databricks Platform" (auto-renders behind its children — see *Containers*); `db-platform` + `governance-block` `pin` to its `top-left`/`top-right` (never a raw `at`); Genie One fronts the consumption tiles with auto-arrows. 2. **Layered platform / integration** — the request is organized as **layers**, and/or Databricks sits **alongside an existing (non-Databricks) system**. The layers organize the diagram in the ask's order (usually lanes, but see Step 0 — governance spans, thin layers merge, pick the fewest legible lanes); external systems get their own **boundary box** with an **integration seam** to the Databricks boundary box; governance (Unity Catalog) SPANS the layers as a pinned top bar or full-height band, not one lane tile. Build it from the ask's actual layers (boundary boxes + seam + layers-as-columns). This is the shape for enterprise / "add Databricks to our stack" asks — do NOT reduce it to the funnel. **Logical DOMAINS / knowledge areas** (a "five-domain context", a shared semantic layer, a set of business subject areas any agent can query) → draw each as a `type:"logo"` with **`icon:"file:vendor/genie-ontology"`** (the Genie Ontology mark = logical domain / knowledge), wrapped in a `box` titled for the layer. 3. **Physical / governance** — the Unity Catalog HIERARCHY: workspace → metastore → catalogs → schemas → tables. A **containment** picture (nested boxes), NOT a flow. Trigger: the ask is about UC objects / org structure. Use the container-box presets — see *Databricks physical layout* below. 4. **Infra / networking** — cloud/account topology: VPC / subnets / PrivateLink. Also containment (nested cloud boxes, the *Containers* pattern). Trigger: the ask is about networking / deployment / cloud accounts. If the ask blends kinds (e.g. "the layered flow, inside our VPC"), compose them. ### Step 2 — (solution/demo only) how much to infer - **Broad / use-case ask** ("predictive maintenance", "fraud detection", "a governed data platform"): **infer the full end-to-end shape** the use-case implies. Predictive maintenance is a MODEL story → sensors/history → pipeline → ML training → registry → serving → scoring + dashboards. **Don't under-scope** to the couple of nouns typed — add the components that make it actually work. - **Named-component ask** ("Lakeflow Connect + SDP into UC, served by a model endpoint"): **assemble exactly those into a working whole** — honor what they named + add the connective tissue that wires it (platform `box`, ingest ports, an entry point, the edges). **Don't over-build** past what they asked, **don't under-build** to isolated unconnected tiles. Either way, components that belong together, wired so the flow reads correctly, on the governed platform. **Don't copy a reference verbatim — adapt it to THIS ask. References are for inspiration.** Read the reference(s) to learn the PATTERN (which components connect, how the layout/handles/pins work), then build a fresh diagram for THIS ask. **Read as many references as are relevant** — most real asks mix several: take the ML platform's serving lane, add the agent-bricks supervisor, drop the governance bar on top, wrap it in the layered shape's boundary boxes, swap the sources. Pick whichever references inform the ask, learn from each, and compose your own. A reference matches the ask 1:1 only rarely — even then, adapt it (names, sources, which components) rather than reproduce it. See **Pick a starting point** for what each offers. ### Step 2.5 — pick how to draw the DATA / INGEST layer (3 ways) The bronze→silver→gold data layer can be drawn three ways. Pick by **what the story emphasizes** — don't default to one: 1. **`lakeflow-genie-block` (or `lakeflow-block`) — the big unified block.** Shows the whole ingest story in ONE block: Connect · Zerobus · raw-file landing on the left rail, the SDP medallion (bronze→silver→gold), Delta/Iceberg, and (genie variant) a "Built with Genie Code" footer. **Use when ingestion IS part of the story** — you want to showcase how data lands + gets processed, all products in one tidy block. Optional `bronze_desc`/`silver_desc`/`gold_desc` params add a short caption under each layer. ```json { "id": "data", "type": "lakeflow-genie-block", "col": "pipeline", "params": { "gold_desc": "Business marts + metrics" } } ``` 2. **`medallion-table` — the simpler block.** Just bronze→silver→gold in one compact tile, no ingest rail. **Use when ingestion is NOT the focus** but you still need the medallion — especially when you want the **Feature Store / Metric Views** forks off gold (`params:{feature_store,metric_views}`, wired via `@out-fs`/`@out-mv`/`@out-gold`). Same `*_desc` layer-caption params. ```json { "id": "med", "type": "medallion-table", "col": "pipeline", "params": { "feature_store": true, "metric_views": true } } ``` 3. **DIY — compose it yourself.** When you want to go into DETAIL and list the actual tables: a `type:"box"` per layer (title `"Bronze"`/`"Silver"`/`"Gold"`) wrapping `type:"logo"` tiles (icon `bronzeLayer`/`silverLayer`/`goldLayer`, `text` = the table name). Tiles use `col`/`row` inside the box's column; each box needs no `col`/`pad`/size/`z` — it auto-sizes around its tiles (z is automatic — see *Containers*). Flow edges box→box. Optionally wrap all three in a **parent box** titled "Lakeflow Spark Declarative Pipelines" with the SDP logo (`titleIcon:"sdpBrand"`) — a box `titleIcon` accepts ANY icon-library key. ```json { "id": "b1", "type": "logo", "icon": "bronzeLayer", "text": "orders_raw", "caption": "right", "col": "bronze", "row": 1 }, { "id": "b2", "type": "logo", "icon": "bronzeLayer", "text": "events_raw", "caption": "right", "col": "bronze", "row": 2 }, { "id": "s1", "type": "logo", "icon": "silverLayer", "text": "orders", "caption": "right", "col": "silver", "row": 1 }, { "id": "s2", "type": "logo", "icon": "silverLayer", "text": "events", "caption": "right", "col": "silver", "row": 2 }, { "id": "g1", "type": "logo", "icon": "goldLayer", "text": "customer_360", "caption": "right", "col": "gold", "row": 1, "ai_reasoning": "only add ai_reasoning when there's a real authoring choice to record — don't invent" }, { "id": "bronze-box", "type": "box", "title": "Bronze", "wraps": ["b1", "b2"] },