메뉴
Pipeline orchestration and workflow management with Prefect, Dagster, and dbt. Covers scheduling, dependency management, retries, and integration patterns.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Access cloud storage (S3, GCS, Azure) in Python using fsspec, pyarrow.fs, or obstore. Includes DataFrame integrations (Polars, DuckDB, Pandas, PyArrow), performance optimization, patterns for incremental loading, partitioned writes, and cross-cloud copy.
Exploratory data analysis and visualization: profiling datasets, choosing appropriate charts, applying statistical tests, and creating effective visualizations for insight communication. Use when understanding data structure, exploring distributions and relationships, selecting visualization libraries, or producing analysis-ready charts.
Data quality validation and observability for data pipelines. Combines Great Expectations and Pandera for data validation with OpenTelemetry and Prometheus for monitoring and alerting.
File formats and lakehouse table formats for data lakes: Parquet, Arrow, Lance, Zarr, Avro, ORC, Delta Lake, Apache Iceberg, and Apache Hudi. Covers compression, partitioning, ACID transactions, schema evolution, and format selection.
AI/ML production workflows: embedding generation, vector storage, RAG patterns, LLM monitoring, and batch inference.
Feature engineering for machine learning: encoding categorical variables, scaling numeric features, datetime transformations, text features, and leakage-safe preprocessing pipelines. Use when preparing data for modeling or improving model performance through better representations.
| name | orchestrating-data-pipelines |
| description | Pipeline orchestration and workflow management with Prefect, Dagster, and dbt. Covers scheduling, dependency management, retries, and integration patterns. |
| dependsOn | ["@building-data-pipelines"] |
Workflow orchestration tools for data pipelines: Prefect, Dagster, and dbt. These tools handle scheduling, dependency resolution, retries, monitoring, and state management for production data pipelines.
| Tool | Paradigm | Best For | Learning Curve |
|---|---|---|---|
| Prefect | Flow-based | Pythonic workflows, quick prototypes, cloud-first | Moderate |
| Dagster | Asset-based | Data asset lineage, reproducibility, type checking | Steeper |
| dbt | SQL transformations | Analytics engineering, ELT, data warehouses | Low (SQL-focused) |
| FlowerPower | Hamilton DAGs | Lightweight batch ETL, configuration-driven pipelines | Low-Moderate |
Prefect: You want Python code flexibility, Prefect Cloud UI, and quick setup. Good for general-purpose data pipelines, ETL, and API integrations.
Dagster: You care about data asset observability, type safety, and reproducibility. Good for complex data platforms with clear asset dependencies.
dbt: Your transformations are primarily SQL and you're building analytics marts in a data warehouse. Great for analytics engineering teams.
Assumes familiarity with:
@building-data-pipelines - Polars, DuckDB, PyArrow@accessing-cloud-storage - Cloud storage for intermediate dataRelated:
@assuring-data-pipelines - Data validation integrated into orchestration@assuring-data-pipelines - Monitoring and tracing@designing-data-storage - Delta/Iceberg for state managementSee: @orchestrating-data-pipelines/prefect.md
See: @orchestrating-data-pipelines/dagster.md
See: @orchestrating-data-pipelines/dbt.md
FlowerPower is a lightweight DAG orchestration framework built on Apache Hamilton, ideal for batch ETL and data transformation scripts without the overhead of full orchestrators.
Key characteristics:
When to choose FlowerPower over Prefect/Dagster:
When NOT to use:
FlowerPower limitations vs. Prefect/Dagster:
| Feature | Prefect/Dagster | FlowerPower |
|---|---|---|
| Scheduling | Native (cron, intervals) | External (cron/systemd) |
| State persistence | Database/cloud | None (ephemeral) |
| Retry policies | Configurable per task | Per-pipeline via YAML |
| Observability | Rich UI, lineage | Basic Hamilton UI |
| Production readiness | High | Moderate (batch jobs) |
Integration with data-engineering stack:
@building-data-pipelines)@designing-data-storage)@accessing-cloud-storage)@assuring-data-pipelines)@building-data-pipelines)Skill reference: @flowerpower - Complete guide to FlowerPower with advanced production patterns (watermarks, data quality, incremental loads, cloud deployment).
See: @orchestrating-data-pipelines/integrations/cloud-storage.md
# Prefect: @task(retries=3, retry_delay_seconds=60)
# Dagster: @asset(retry_policy=RetryPolicy(...))
# dbt: --fail-fast flag + custom macro retry logic
All orchestrators assume idempotent operations - running twice should produce identical results. Design your INSERT, UPDATE, MERGE operations to be idempotent.
dbt_run_results.json; uses SELECT + INSERT by defaulttask1 >> task2)@asset(depends_on=[other_asset]))ref() calls in models