data-lake-architecture

name	data-lake-architecture
description	Guides agents through data lake architecture design. Use when defining raw, refined, curated, or publish layers; storage organization; retention; and operational boundaries for a data lake.

Data Lake And Zone Architecture

Overview

Use this skill when the storage platform needs structure before pipelines scale into chaos. It helps agents design clear lake zones, dataset boundaries, ownership, lifecycle rules, and publish-safe storage conventions.

When to Use

• designing a new data lake
• reorganizing raw, staging, refined, or curated zones
• defining object storage layout and lifecycle rules
• separating landing, transformation, and publish responsibilities
• reducing data swamp behavior in shared lake storage

Do not use this to justify creating extra layers with no operational purpose.

Workflow

1. Define the lake purpose and consumers. Clarify:

   • source landing needs
   • internal producer teams
   • publish consumers
   • compliance and retention expectations

2. Define the zone model intentionally. Typical zones include:

   • raw or landing
   • standardized or staging
   • refined or modeled
   • publish or serving

3. Assign responsibilities to each zone. Decide:

   • who writes to it
   • who reads from it
   • what quality guarantees exist
   • whether mutation is allowed

4. Design storage conventions. Include:

   • path or catalog naming
   • partition strategy
   • retention lifecycle
   • file-size expectations
   • ownership tags and metadata

5. Keep publish rules separate from lake convenience. Not every dataset in the lake is ready for shared consumption.

Medallion Pattern (Bronze / Silver / Gold)

A common specialization of the zone model. The names only matter if each layer carries distinct responsibilities:

Layer	Purpose	Quality bar
Bronze	Raw or lightly standardized landing	Source-faithful, append-only, no business logic
Silver	Cleaned, conformed, contract-aware	Deduplicated, typed, validated, merge-safe
Gold	Business-facing, publish-ready	Aggregated or wide, stable schema, dashboard-safe

Rules that prevent medallion from becoming decoration:

• Bronze is permissive, Gold is disciplined — they need different operating rules.
• Movement between layers requires explicit validation, not just a copy.
• Layer names without contracts and quality bars create confusion, not design.
• Gold tables fed directly from unstable raw landing logic skip the quality gate that makes them trustworthy.
• Merge semantics (SCD1, SCD2, append, overwrite) must be explicit per Silver table — defer to data-lake-open-table-format for format-specific MERGE and compaction behavior.
• Maintenance (compaction, retention, snapshot expiry) is architecture, not an afterthought — defer to data-lake-open-table-format for the operational schedule.

Batch and streaming into the same layer: if both target Bronze, they must not conflict on commits or partitions. Design one as the sole writer per partition, or use a staging table that the batch MERGE reads from. Details in data-lake-open-table-format Step 4.

Cross-Cloud Architecture

Use references/cloud-data-engineering-architecture-patterns.md when the task is not only zone design, but choosing the overall cloud architecture pattern across lake, warehouse, lakehouse, streaming, and hybrid shapes.

Common Rationalizations

Rationalization	Reality
"We can dump everything into one bucket or container and organize later."	That is how data lakes turn into data swamps.
"More zones always means better governance."	Extra layers without distinct purpose add complexity and slow teams down.
"If the file exists in the lake, it is available for analytics."	Raw landing data rarely has the quality or contract guarantees needed for shared use.
"Bronze, silver, and gold are enough architecture by themselves."	Layer names without contracts and movement rules create confusion, not design clarity.
"We can merge everything directly into gold."	Business-facing layers need stronger validation and stability than raw ingestion paths can guarantee.

Red Flags

• zone meanings overlap or are undocumented
• ownership is unclear at the dataset or zone level
• publish and landing data are mixed together
• retention, cleanup, or lifecycle policy is absent
• medallion layers exist but have no explicit responsibilities or quality bars
• gold tables are fed directly from raw landing without a validation stage
• merge semantics for CDC or late-arriving data are undefined

Verification

• The zone model has clear purposes and boundaries
• Ownership, read/write expectations, and quality guarantees are explicit
• Storage conventions and lifecycle rules are documented
• Shared publish datasets are separated from raw landing data
• Each medallion layer has a distinct quality bar and movement rules to the next
• Streaming and batch interaction within a layer is operationally safe