메뉴
Arquitetura Medallion (Bronze/Silver/Gold) com regras mandatórias por camada no Lakeflow/SDP, padrões cross-platform Fabric↔Databricks, JSON de Job Databricks e checklist de qualidade de pipeline. Use ao desenhar ou revisar um novo pipeline ETL/ELT.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Índice de skills Python: FastAPI, pandas/polars, pytest, packaging, asyncio e CLIs. Use ao trabalhar com APIs REST, transformações de dados, testes, publicação de pacotes, código async ou ferramentas de linha de comando.
Engenharia de Ontologias OWL no Microsoft Fabric — import, export, design e integração com OneLake e Delta Lake.
Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → chunk → index → query).
Create Databricks AI/BI dashboards. Use when creating, updating, or deploying Lakeview dashboards. CRITICAL: You MUST test ALL SQL queries via execute_sql BEFORE deploying. Follow guidelines strictly.
Builds Python-based Databricks applications using Dash, Streamlit, Gradio, Flask, FastAPI, or Reflex. Handles OAuth authorization (app and user auth), app resources, SQL warehouse and Lakebase connectivity, model serving integration, foundation model APIs, LLM integration, and deployment. Use when building Python web apps, dashboards, ML demos, or REST APIs for Databricks, or when the user mentions Streamlit, Dash, Gradio, Flask, FastAPI, Reflex, or Databricks app.
Patterns and best practices for Lakebase Autoscaling (next-gen managed PostgreSQL). Use when creating or managing Lakebase Autoscaling projects, configuring autoscaling compute or scale-to-zero, working with database branching for dev/test workflows, implementing reverse ETL via synced tables, or connecting applications to Lakebase with OAuth credentials.
| name | pipeline-design |
| description | Arquitetura Medallion (Bronze/Silver/Gold) com regras mandatórias por camada no Lakeflow/SDP, padrões cross-platform Fabric↔Databricks, JSON de Job Databricks e checklist de qualidade de pipeline. Use ao desenhar ou revisar um novo pipeline ETL/ELT. |
Fonte (CSV / API / Stream)
│
▼
┌─────────────┐
│ BRONZE │ STREAMING TABLE via cloud_files() (Auto Loader).
│ (Raw Zone) │ Append-only, sem transformação. Formato: Delta.
└─────────────┘ Metadados: _ingest_timestamp, _metadata.file_path
│
▼
┌─────────────┐
│ SILVER │ STREAMING TABLE consumindo via stream(bronze_table).
│ (Clean Zone)│ SCD Tipo 2 nativo via AUTO CDC INTO (CREATE FLOW).
└─────────────┘ ⚠️ PROIBIDO: MATERIALIZED VIEW, LAG/LEAD manual, SHA2.
│
▼
┌─────────────┐
│ GOLD │ MATERIALIZED VIEW para Star Schema e agregações.
│ (Serve Zone)│ Otimizado para consulta (CLUSTER BY / Z-ORDER).
└─────────────┘
| Camada | Tipo SDP Obrigatório | Padrão de Ingestão | Proibido |
|---|---|---|---|
| Bronze | STREAMING TABLE | cloud_files() (Auto Loader) | read_files, batch reads, schemaHints |
| Silver | STREAMING TABLE + CREATE FLOW | stream(bronze_table) + AUTO CDC INTO | MATERIALIZED VIEW, Window Functions LAG/LEAD para SCD2 |
| Gold | MATERIALIZED VIEW | Leitura direta das tabelas Silver | — |
OneLake (Fabric) Databricks
┌─────────────────┐ ┌─────────────────────────────┐
│ CSV / Parquet │──────▶│ Bronze: ingestão via ABFSS │
│ no Lakehouse │ │ Silver: transformação Spark │
└─────────────────┘ │ Gold: tabela Unity Catalog │
└─────────────────────────────┘
Estratégias de conectividade:
{
"name": "pipeline_vendas_daily",
"tasks": [
{
"task_key": "ingest",
"notebook_task": {
"notebook_path": "/Workspace/pipelines/01_bronze_ingest",
"source": "WORKSPACE"
},
"existing_cluster_id": "<cluster-id>"
},
{
"task_key": "transform",
"depends_on": [{"task_key": "ingest"}],
"notebook_task": {
"notebook_path": "/Workspace/pipelines/02_silver_transform"
},
"existing_cluster_id": "<cluster-id>"
}
],
"schedule": {
"quartz_cron_expression": "0 0 6 * * ?",
"timezone_id": "America/Sao_Paulo"
},
"max_retries": 2,
"min_retry_interval_millis": 300000
}