| name | data-pipeline-audit |
| description | Audit data engineering workloads for compute waste — Spark shuffle explosions, small-files problems, missing partition pruning and predicate pushdown, row formats where columnar belongs, full-refresh pipelines that should be incremental, and dbt models rebuilding unchanged upstreams. Use this skill whenever the user shares Spark/ETL/warehouse jobs, dbt projects, Airflow DAGs, query plans from Snowflake/BigQuery/Databricks, or asks why data pipelines are slow, expensive, or heavy. Part of Lean Agentic AI Skills; emits lean-findings.json. |
Data Pipeline Audit
Producer skill. Input: pipeline code (Spark/SQL/dbt), DAG definitions, query plans/profiles, table layouts and stats. Output: lean-findings.json.
Data pipelines are some of the heaviest scheduled compute in most organizations, and they re-run daily whether or not anything upstream changed. Waste here compounds on a schedule.
Subject type: emit subject.type: "data-pipeline" in findings.
Signatures
- Full refresh where incremental serves — pipelines recomputing entire history when only the newest partition changed; dbt models without incremental materialization on append-mostly sources. High (daily × full-history). Fix: incremental strategy with a correctness note (late-arriving data handling must be designed, not ignored — name the lookback window).
- Rebuilding unchanged upstreams — orchestration that runs every model/task on every schedule with no freshness/state check; dbt full runs where state-based selection (
--select state:modified+) serves. High.
- Missing partition pruning — queries scanning all partitions because filters don't match partition keys, or tables partitioned on the wrong column for the dominant query pattern. High. Evidence: bytes-scanned from query plans/profiles — real numbers, report them.
- No predicate/projection pushdown — filters applied after full reads (e.g. filtering a DataFrame after loading whole tables; SELECT * feeding a job that uses three columns). High.
- Row formats at rest — JSON/CSV in the lake where Parquet/ORC belongs; no compression on text formats. High: columnar + compression cuts both scan E and storage M. Fix names the format and the migration path.
- Shuffle explosions — wide joins/groupBys without co-partitioning, skewed keys sending one executor 90% of the data, crossJoins hiding in plain sight. High. Evidence: shuffle read/write sizes from Spark UI stats when provided.
- Small-files problem — thousands of tiny files per partition (streaming sinks, over-parallel writes); every job pays per-file open overhead. Medium-high. Fix: compaction jobs, target file sizes, auto-optimize features where the platform has them.
- Oversized clusters idling — fixed-size clusters for bursty jobs; no autoscaling/auto-terminate on interactive clusters. Medium-high; route sizing specifics to right-size-compute, record the pipeline-level finding here.
- Materializing the unread — tables/marts no downstream consumer has queried (from access logs/information_schema where available). Medium; flag for owner confirmation, never assert abandonment.
- Chained recomputation of shared logic — the same expensive CTE/subquery computed in N models instead of one upstream materialization. Medium.
Honesty rules
Bytes-scanned, shuffle sizes, run durations, and file counts from the user's plans/UI/stats are real — report them and cite where they came from. Cost figures only from the user's billing data. Incremental-conversion recommendations always name the correctness trade-off (late data, backfills, idempotency). Scheduled pipelines are classically shiftable — always mention the carbon-aware-scheduler pairing.
Cost signal (countable)
Cost drivers: bytes scanned, compute-hours per job run, and job frequency. Bytes-scanned in particular is exposed by every serverless warehouse and is the honest cost driver to quote.
Not this skill's job
Warehouse/OLTP query tuning outside pipelines (db-efficiency-audit), storage tiering of the lake (storage-lifecycle-audit), what data to ingest at all (data-minimization-designer).
