| name | medallion-layer-design |
| description | Playbook for designing bronze/silver/gold medallion layers on OneLake — per-layer V-Order, file-size targeting, partitioning vs Liquid Clustering, deletion vectors, and materialized lake view decisions. |
Medallion Layer Design
When to Use This Skill
Use when setting up a new Lakehouse or when auditing an existing one for performance, cost, or correctness issues in the medallion structure.
1. Layer Purpose and Constraints
| Layer | What it holds | Key constraints |
|---|
| Bronze | Raw, immutable ingested data — exact copy of source | Never alter; append-only; no V-Order; schema may be loose |
| Silver | Cleansed, conformed, deduplicated | V-Order optional (helps reads, costs writes); apply Liquid Clustering; deletion vectors for MERGE-heavy tables |
| Gold | Business-ready aggregates shaped for consumption | V-Order ON (required for Direct Lake); narrow schema; no PII; served to semantic models and SQL endpoint |
Never serve bronze to Direct Lake or the SQL endpoint — this is a hard rule (CLAUDE.md §3 #3).
2. V-Order Settings by Layer
df.write.format("delta").option("delta.parquet.vorder.enabled", "false").save(bronze_path)
df.write.format("delta").option("delta.parquet.vorder.enabled", "true").save(gold_path)
V-Order adds ~15% write overhead and reduces read cost — the trade pays off at gold where reads dominate.
3. File Size Targeting
| Layer | Target file size | Rationale |
|---|
| Bronze | 128–256 MB | Minimize small-file problem from streaming/micro-batch ingestion |
| Silver | 256–512 MB | Balance query parallelism vs file count |
| Gold | 128–256 MB | Smaller files suit Direct Lake framing (reads one file per column stripe) |
Use OPTIMIZE + VACUUM as a maintenance job — not inline in the ingestion notebook:
OPTIMIZE schema.gold_sales ZORDER BY (customer_id, order_date);
VACUUM schema.gold_sales RETAIN 168 HOURS;
4. Partitioning vs Liquid Clustering
| Scenario | Choice | Why |
|---|
| High-cardinality column (customer_id, order_id) | Liquid Clustering | Static partitions explode file count |
| Low-cardinality, time-based (year/month) + high daily volume | Static partition by year/month | Reduces scan on date-range queries |
| Mixed (date range + customer filter) | Liquid Clustering on both columns | Handles cross-dimension skipping |
| Direct Lake gold tables | Liquid Clustering preferred | Avoids over-partitioned directories that confuse framing |
spark.sql("""
ALTER TABLE schema.silver_orders
CLUSTER BY (customer_id, order_date)
""")
5. Deletion Vectors
Enable on any silver or gold table with MERGE, UPDATE, or DELETE operations:
spark.sql("""
ALTER TABLE schema.silver_orders
SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')
""")
Without deletion vectors, a MERGE rewrites entire Parquet files. With them, only a small bitmap sidecar is written — dramatically reducing write amplification on CDC/upsert pipelines.
6. Materialized Lake View vs Notebook Gold Shaping
| Factor | Materialized Lake View (MLV) | Notebook gold |
|---|
| Refresh trigger | Defined cadence / on-demand | Pipeline-scheduled notebook |
| Compute cost | Fabric CU (no separate Spark job) | Spark session startup overhead |
| Complexity | Simple SELECT projections + aggregations | Full Spark/Python for complex logic |
| ALM | Git-tracked as workspace item | Notebook .ipynb in Git |
Choose MLV for straightforward projections of silver into gold. Choose a notebook for multi-step transformations, ML feature engineering, or complex business logic.
7. Layer Checklist
Bronze:
Silver:
Gold:
Pitfalls
- Serving bronze to a semantic model — query performance degrades and PII leaks
- Enabling V-Order on bronze — adds write overhead to append-heavy ingestion with no read benefit
- Static partitioning on a high-cardinality column — creates millions of directories; worse than no partitioning
- Skipping deletion vectors on a CDC table — MERGE rewrites entire files; cost spikes on high-frequency updates
See Also
