菜单
Data catalogs for lakehouse architectures: Iceberg catalogs (Hive Metastore, AWS Glue, REST/Tabular), using DuckDB as a lightweight multi-source catalog, and comparisons of open-source metadata tools (Amundsen, DataHub, OpenMetadata).
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Access cloud storage (S3, GCS, Azure) in Python using fsspec, pyarrow.fs, or obstore. Includes DataFrame integrations (Polars, DuckDB, Pandas, PyArrow), performance optimization, patterns for incremental loading, partitioned writes, and cross-cloud copy.
Exploratory data analysis and visualization: profiling datasets, choosing appropriate charts, applying statistical tests, and creating effective visualizations for insight communication. Use when understanding data structure, exploring distributions and relationships, selecting visualization libraries, or producing analysis-ready charts.
Data quality validation and observability for data pipelines. Combines Great Expectations and Pandera for data validation with OpenTelemetry and Prometheus for monitoring and alerting.
File formats and lakehouse table formats for data lakes: Parquet, Arrow, Lance, Zarr, Avro, ORC, Delta Lake, Apache Iceberg, and Apache Hudi. Covers compression, partitioning, ACID transactions, schema evolution, and format selection.
AI/ML production workflows: embedding generation, vector storage, RAG patterns, LLM monitoring, and batch inference.
Feature engineering for machine learning: encoding categorical variables, scaling numeric features, datetime transformations, text features, and leakage-safe preprocessing pipelines. Use when preparing data for modeling or improving model performance through better representations.
| name | managing-data-catalogs |
| description | Data catalogs for lakehouse architectures: Iceberg catalogs (Hive Metastore, AWS Glue, REST/Tabular), using DuckDB as a lightweight multi-source catalog, and comparisons of open-source metadata tools (Amundsen, DataHub, OpenMetadata). |
| dependsOn | ["@designing-data-storage","@accessing-cloud-storage"] |
Guide to selecting, configuring, and using data catalog systems for data discovery, governance, and unified table access across multiple query engines.
Use this skill when:
Do not use this skill for:
@designing-data-storage@accessing-cloud-storage@accessing-cloud-storage@building-data-pipelines| Catalog | Backend | Managed? | Best For |
|---|---|---|---|
| Hive Metastore | RDBMS (Postgres/MySQL) | Self-hosted | Existing Hadoop, high partition counts |
| AWS Glue | AWS-managed serverless | AWS-managed | AWS-native stacks (Athena, EMR) |
| Tabular/REST | SaaS (Nessie-backed) | Vendor-managed | Iceberg-native, Git-like branching |
| DuckDB (embedded) | Local file/Postgres | Self-hosted | Single-user, PoC, small teams |
| Scenario | Recommended Catalog | Why |
|---|---|---|
| AWS-native (Athena, Redshift Spectrum) | AWS Glue | Serverless, IAM integration |
| Self-hosted Hadoop/Spark | Hive Metastore | Battle-tested, no vendor lock-in |
| Iceberg-first, multi-cloud | Tabular or Hive | Native Iceberg features or flexibility |
| Small team, PoC, analytics | DuckDB | Zero infrastructure, SQL-native |
| LinkedIn-scale metadata | DataHub | Enterprise lineage, scale |
| Governance-heavy workflows | OpenMetadata | Built-in workflows, data quality |
Catalog (name) → Table identifier → Metadata location → Data files
(schema, snapshots, partitions)
Key insight: The catalog stores only metadata pointers. Actual data lives in object storage (S3, GCS, Azure).
# Same table, different engines
table = catalog.load_table("db.events")
# PyIceberg → Pandas DataFrame
df = table.scan().to_pandas()
# Spark SQL (via same catalog)
spark.table("db.events") # Same underlying data
# DuckDB (via ATTACH or REST catalog)
SELECT * FROM iceberg.db.events
| Guide | Covers | When to Read |
|---|---|---|
| Hive Metastore | Docker deployment, PyIceberg integration, pros/cons | Self-hosting Hadoop/Spark |
| AWS Glue Catalog | GlueCatalog setup, crawlers, IAM, Unity Catalog federation | AWS-native stacks |
| REST Catalog & Tabular | Tabular SaaS, Nessie patterns, Git-like branching | Iceberg-first, multi-cloud |
| DuckDB Multi-Source | ATTACH patterns, unified views, limitations | Single-user/PoC catalog |
| Open Source Tools | Amundsen vs DataHub vs OpenMetadata comparison | Metadata discovery/governance |
from pyiceberg.catalog import load_catalog
# Hive Metastore
catalog = load_catalog("hive", **{
"type": "hive",
"uri": "thrift://localhost:9083",
"warehouse": "s3://bucket/warehouse/"
})
# AWS Glue
catalog = load_catalog("glue", **{
"type": "glue",
"region": "us-east-1",
"warehouse": "s3://bucket/warehouse/"
})
# REST/Tabular
catalog = load_catalog("rest", **{
"type": "rest",
"uri": "https://api.tabular.io/ws/...",
"token": "tabular-token-...",
"warehouse": "s3://bucket/warehouse/"
})
# Create and query table
table = catalog.create_table("db.events", schema=schema)
table.append(data)
df = table.scan().to_pandas()
@designing-data-storage - Delta Lake, Iceberg table formats, file format selection@accessing-cloud-storage - fsspec, pyarrow.fs, obstore for storage access@building-data-pipelines - ETL patterns using catalog-registered tables