| license | Apache-2.0 |
| name | lakehouse-architect |
| description | Delta Lake, Apache Iceberg, Hudi for ACID transactions on object storage. Activate on: lakehouse, Delta Lake, Iceberg, Hudi, table format, ACID on S3, time travel, data lake, open table format. NOT for: warehouse query tuning (use data-warehouse-optimizer), streaming ingestion (use streaming-pipeline-architect). |
| allowed-tools | Read,Write,Edit,Bash(npm:*,npx:*,python:*,spark-submit:*) |
| category | Data & Analytics |
| tags | ["lakehouse","delta-lake","iceberg","data-lake","object-storage"] |
| pairs-with | [{"skill":"data-warehouse-optimizer","reason":"Lakehouses complement or replace traditional warehouses"},{"skill":"streaming-pipeline-architect","reason":"Streaming data lands in lakehouse tables"},{"skill":"data-cost-optimizer","reason":"Lakehouse architecture dramatically reduces storage costs"}] |
Lakehouse Architect
Design data lakehouse architectures using Delta Lake, Apache Iceberg, or Apache Hudi for ACID transactions, time travel, and schema evolution on object storage.
Activation Triggers
Activate on: "lakehouse", "Delta Lake", "Apache Iceberg", "Hudi", "table format", "ACID on S3", "time travel", "data lake", "open table format", "Databricks", "catalog"
NOT for: Warehouse query tuning → data-warehouse-optimizer | Streaming pipeline design → streaming-pipeline-architect | Data quality rules → data-quality-guardian
Quick Start
- Choose table format — Iceberg (vendor-neutral, widest engine support), Delta Lake (Databricks ecosystem), Hudi (Uber/AWS)
- Set up catalog — Unity Catalog, AWS Glue, Nessie, or Hive Metastore for table discovery
- Design medallion layers — Bronze (raw), Silver (cleaned), Gold (business-ready)
- Enable compaction — schedule file compaction to avoid small file problem
- Configure retention — time travel retention + vacuum/expire snapshots for cost control
Core Capabilities
| Domain | Technologies |
|---|
| Table Formats | Apache Iceberg 1.7+, Delta Lake 3.x, Apache Hudi 1.x |
| Compute | Spark 3.5+, Trino, DuckDB, Snowflake (Iceberg), Flink |
| Catalogs | Unity Catalog, AWS Glue, Nessie, REST Catalog, Polaris |
| Storage | S3, GCS, ADLS, MinIO |
| Managed | Databricks Lakehouse, AWS Lake Formation, Snowflake Iceberg |
Architecture Patterns
Medallion Architecture
Bronze (Raw) Silver (Cleaned) Gold (Business)
───────────── ──────────────── ───────────────
Raw JSON/CSV/Parquet → Typed, deduplicated → Aggregated, modeled
Append-only → Merge/upsert → Materialized views
Schema-on-read → Schema enforced → Star schema
Full history → Latest + SCD Type 2 → Pre-aggregated metrics
Storage: S3/GCS All layers use Iceberg/Delta
Format: Parquet ACID transactions at each layer
Apache Iceberg Table Management
