| name | aws-native-lakehouse |
| description | Reference for the Amazon SageMaker Lakehouse architecture — unified data access across S3 data lakes and Redshift warehouses using Apache Iceberg, AWS Glue Data Catalog, and Lake Formation governance. Use when designing pipelines that target the lakehouse, configuring catalog integration, or setting up cross-account data sharing. |
AWS Native Lakehouse (SageMaker Lakehouse Architecture)
What It Is
The lakehouse architecture of Amazon SageMaker unifies data across Amazon S3 data lakes and Amazon Redshift data warehouses into a single governed catalog. It eliminates data movement between systems — one copy of data, queried by multiple engines, secured by one permission model.
Built on AWS Glue Data Catalog and AWS Lake Formation, it exposes all data as Apache Iceberg tables accessible from Athena, Redshift Spectrum, EMR, AWS Glue 5.0, and any Iceberg-compatible engine.
Key Components
| Component | Description |
|---|
| Storage | Amazon S3 (general purpose), S3 Tables (managed Iceberg), Redshift Managed Storage (RMS) |
| Catalog | Logical container organizing schemas, tables, and views. Two types: federated (mounts existing sources) and managed (new data in S3/RMS) |
| Database | Organizes metadata tables within a catalog |
| Table/View | Defines schema, partitions, storage location, format, and access SQL |
| Lake Formation | Fine-grained permissions (catalog, database, table, column, cell level) enforced across all engines |
| Glue Data Catalog | Centralized metadata repository — Iceberg REST Catalog API for third-party engine connectivity |
Data Sources
The lakehouse integrates data from:
- Amazon S3 Tables — First cloud object store with built-in Apache Iceberg support. Stored in table buckets optimized for analytics.
- Amazon Redshift Managed Storage — Federated access to Redshift tables without ETL or data duplication.
- Amazon S3 data lakes — Direct integration of existing S3-based assets organized in raw/processed/curated zones.
- Federated catalogs — External sources (Google BigQuery, Snowflake, DynamoDB, PostgreSQL, MySQL, Oracle, SQL Server, Aurora) queryable in-place without data movement.
Apache Iceberg Integration
The lakehouse uses Apache Iceberg as the standard table format:
- ACID transactions — Reliable concurrent read/write operations
- Schema evolution — Add/rename/drop columns without breaking queries
- Time travel — Query historical snapshots for auditing and rollback
- Row-level operations — Efficient MERGE, UPDATE, DELETE on S3
- Iceberg REST Catalog — Standard API enabling any Iceberg-compatible engine to connect
- Table optimization — Compaction, snapshot expiry, and NDV statistics via Glue Catalog
Data Organization (S3 Data Lakes)
| Zone | Purpose |
|---|
| Raw | Ingested data in original format — long-term retention, compliance |
| Processed | Cleaned, transformed, enriched — optimized for analytics |
| Curated | Business-ready datasets — organized for specific use cases |
Best practices:
- Partition by frequently queried dimensions (date, region, category)
- Use columnar formats (Parquet, ORC) for compression and query performance
- Apply S3 lifecycle policies for cost-effective tiering
Governance Model
Lake Formation provides unified governance:
- LF-Tags — Tag-based access control for logical grouping by domain and sensitivity
- Named resources — Direct grants on catalogs, databases, tables, columns
- Cross-account sharing — Share via AWS RAM without data duplication
- Column-level security — Restrict sensitive columns per principal
- Consistent enforcement — Same permissions apply across Athena, EMR, Glue, Redshift Spectrum
Cross-Account Data Sharing
Architecture: Producer account (Account A) shares catalog resources with Consumer account (Account B) via Lake Formation + AWS RAM.
Key steps:
- Producer registers Redshift namespace/S3 location with Lake Formation
- Producer grants database/table permissions to consumer account
- Consumer accepts AWS RAM share
- Consumer creates catalog link container + resource links pointing to shared data
- Consumer grants permissions to execution roles (e.g., Glue-execution-role)
- Consumer queries shared tables using three-part notation in Athena/Spark
Required consumer role policies: AWSGlueServiceRole, AmazonRedshiftDataFullAccess, plus lakeformation:GetDataAccess and redshift-serverless:GetCredentials.
Integration with AWS Glue 5.0
AWS Glue 5.0 Spark jobs access lakehouse data with:
--datalake-formats = iceberg
--enable-lakeformation-fine-grained-access = true
This enables PySpark/Spark SQL to join tables across federated catalogs, S3 Iceberg tables, and Redshift — all governed by Lake Formation permissions.
Relevance to This Agent
The datalake-build-agent generates Bronze/Silver/Gold pipeline configurations and Glue jobs that target this architecture:
- Bronze jobs land data into S3 as Iceberg tables registered in the Glue Data Catalog
- Silver jobs cleanse and deduplicate using Iceberg MERGE operations
- Gold jobs aggregate and produce curated datasets for consumption via Athena/Redshift
- Governance — All tables are registered with Lake Formation for fine-grained access
- Catalog integration — Jobs use
MSCK REPAIR TABLE and Iceberg REST Catalog for discoverability
When to Use This Skill
- Designing how pipeline outputs integrate with the SageMaker Lakehouse
- Configuring Glue Data Catalog registration for generated tables
- Setting up Lake Formation permissions for pipeline-produced data
- Planning cross-account data sharing of pipeline outputs
- Choosing between S3 Tables (managed Iceberg) vs self-managed Iceberg on S3
- Understanding how federated catalogs connect external sources to the lakehouse
