メニュー
Arquitetura Medallion (Bronze/Silver/Gold) com regras mandatórias por camada no Lakeflow/SDP, padrões cross-platform Fabric↔Databricks, JSON de Job Databricks e checklist de qualidade de pipeline. Use ao desenhar ou revisar um novo pipeline ETL/ELT.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | pipeline-design |
| description | Arquitetura Medallion (Bronze/Silver/Gold) com regras mandatórias por camada no Lakeflow/SDP, padrões cross-platform Fabric↔Databricks, JSON de Job Databricks e checklist de qualidade de pipeline. Use ao desenhar ou revisar um novo pipeline ETL/ELT. |
Fonte (CSV / API / Stream)
│
▼
┌─────────────┐
│ BRONZE │ STREAMING TABLE via cloud_files() (Auto Loader).
│ (Raw Zone) │ Append-only, sem transformação. Formato: Delta.
└─────────────┘ Metadados: _ingest_timestamp, _metadata.file_path
│
▼
┌─────────────┐
│ SILVER │ STREAMING TABLE consumindo via stream(bronze_table).
│ (Clean Zone)│ SCD Tipo 2 nativo via AUTO CDC INTO (CREATE FLOW).
└─────────────┘ ⚠️ PROIBIDO: MATERIALIZED VIEW, LAG/LEAD manual, SHA2.
│
▼
┌─────────────┐
│ GOLD │ MATERIALIZED VIEW para Star Schema e agregações.
│ (Serve Zone)│ Otimizado para consulta (CLUSTER BY / Z-ORDER).
└─────────────┘
| Camada | Tipo SDP Obrigatório | Padrão de Ingestão | Proibido |
|---|---|---|---|
| Bronze | STREAMING TABLE | cloud_files() (Auto Loader) | read_files, batch reads, schemaHints |
| Silver | STREAMING TABLE + CREATE FLOW | stream(bronze_table) + AUTO CDC INTO | MATERIALIZED VIEW, Window Functions LAG/LEAD para SCD2 |
| Gold | MATERIALIZED VIEW | Leitura direta das tabelas Silver | — |
OneLake (Fabric) Databricks
┌─────────────────┐ ┌─────────────────────────────┐
│ CSV / Parquet │──────▶│ Bronze: ingestão via ABFSS │
│ no Lakehouse │ │ Silver: transformação Spark │
└─────────────────┘ │ Gold: tabela Unity Catalog │
└─────────────────────────────┘
Estratégias de conectividade:
{
"name": "pipeline_vendas_daily",
"tasks": [
{
"task_key": "ingest",
"notebook_task": {
"notebook_path": "/Workspace/pipelines/01_bronze_ingest",
"source": "WORKSPACE"
},
"existing_cluster_id": "<cluster-id>"
},
{
"task_key": "transform",
"depends_on": [{"task_key": "ingest"}],
"notebook_task": {
"notebook_path": "/Workspace/pipelines/02_silver_transform"
},
"existing_cluster_id": "<cluster-id>"
}
],
"schedule": {
"quartz_cron_expression": "0 0 6 * * ?",
"timezone_id": "America/Sao_Paulo"
},
"max_retries": 2,
"min_retry_interval_millis": 300000
}
Índice de skills Python: FastAPI, pandas/polars, pytest, packaging, asyncio e CLIs. Use ao trabalhar com APIs REST, transformações de dados, testes, publicação de pacotes, código async ou ferramentas de linha de comando.
Engenharia de Ontologias OWL no Microsoft Fabric — import, export, design e integração com OneLake e Delta Lake.
Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → chunk → index → query).
Create Databricks AI/BI dashboards. Use when creating, updating, or deploying Lakeview dashboards. CRITICAL: You MUST test ALL SQL queries via execute_sql BEFORE deploying. Follow guidelines strictly.
Builds Python-based Databricks applications using Dash, Streamlit, Gradio, Flask, FastAPI, or Reflex. Handles OAuth authorization (app and user auth), app resources, SQL warehouse and Lakebase connectivity, model serving integration, foundation model APIs, LLM integration, and deployment. Use when building Python web apps, dashboards, ML demos, or REST APIs for Databricks, or when the user mentions Streamlit, Dash, Gradio, Flask, FastAPI, Reflex, or Databricks app.
Patterns and best practices for Lakebase Autoscaling (next-gen managed PostgreSQL). Use when creating or managing Lakebase Autoscaling projects, configuring autoscaling compute or scale-to-zero, working with database branching for dev/test workflows, implementing reverse ETL via synced tables, or connecting applications to Lakebase with OAuth credentials.