data-pipeline-audit

Stars4

Forks0

UpdatedJuly 18, 2026 at 15:25

Audit data engineering workloads for compute waste — Spark shuffle explosions, small-files problems, missing partition pruning and predicate pushdown, row formats where columnar belongs, full-refresh pipelines that should be incremental, and dbt models rebuilding unchanged upstreams. Use this skill whenever the user shares Spark/ETL/warehouse jobs, dbt projects, Airflow DAGs, query plans from Snowflake/BigQuery/Databricks, or asks why data pipelines are slow, expensive, or heavy. Part of Lean Agentic AI Skills; emits lean-findings.json.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

navveenb

navveenb/lean-agentic-ai-skills

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

File Explorer

2 files

SKILL.md

readonly

name

data-pipeline-audit

description

Data Pipeline Audit

Producer skill. Input: pipeline code (Spark/SQL/dbt), DAG definitions, query plans/profiles, table layouts and stats. Output: lean-findings.json.

Data pipelines are some of the heaviest scheduled compute in most organizations, and they re-run daily whether or not anything upstream changed. Waste here compounds on a schedule.

Subject type: emit subject.type: "data-pipeline" in findings.

Signatures

Full refresh where incremental serves — pipelines recomputing entire history when only the newest partition changed; dbt models without incremental materialization on append-mostly sources. High (daily × full-history). Fix: incremental strategy with a correctness note (late-arriving data handling must be designed, not ignored — name the lookback window).
Rebuilding unchanged upstreams — orchestration that runs every model/task on every schedule with no freshness/state check; dbt full runs where state-based selection (--select state:modified+) serves. High.
Missing partition pruning — queries scanning all partitions because filters don't match partition keys, or tables partitioned on the wrong column for the dominant query pattern. High. Evidence: bytes-scanned from query plans/profiles — real numbers, report them.
No predicate/projection pushdown — filters applied after full reads (e.g. filtering a DataFrame after loading whole tables; SELECT * feeding a job that uses three columns). High.
Row formats at rest — JSON/CSV in the lake where Parquet/ORC belongs; no compression on text formats. High: columnar + compression cuts both scan E and storage M. Fix names the format and the migration path.
Shuffle explosions — wide joins/groupBys without co-partitioning, skewed keys sending one executor 90% of the data, crossJoins hiding in plain sight. High. Evidence: shuffle read/write sizes from Spark UI stats when provided.
Small-files problem — thousands of tiny files per partition (streaming sinks, over-parallel writes); every job pays per-file open overhead. Medium-high. Fix: compaction jobs, target file sizes, auto-optimize features where the platform has them.
Oversized clusters idling — fixed-size clusters for bursty jobs; no autoscaling/auto-terminate on interactive clusters. Medium-high; route sizing specifics to right-size-compute, record the pipeline-level finding here.
Materializing the unread — tables/marts no downstream consumer has queried (from access logs/information_schema where available). Medium; flag for owner confirmation, never assert abandonment.
Chained recomputation of shared logic — the same expensive CTE/subquery computed in N models instead of one upstream materialization. Medium.

Honesty rules

Bytes-scanned, shuffle sizes, run durations, and file counts from the user's plans/UI/stats are real — report them and cite where they came from. Cost figures only from the user's billing data. Incremental-conversion recommendations always name the correctness trade-off (late data, backfills, idempotency). Scheduled pipelines are classically shiftable — always mention the carbon-aware-scheduler pairing.

Cost signal (countable)

Cost drivers: bytes scanned, compute-hours per job run, and job frequency. Bytes-scanned in particular is exposed by every serverless warehouse and is the honest cost driver to quote.

Not this skill's job

Warehouse/OLTP query tuning outside pipelines (db-efficiency-audit), storage tiering of the lake (storage-lifecycle-audit), what data to ingest at all (data-minimization-designer).

More from this repository

same repository

agentic-loop-auditor

navveenb/lean-agentic-ai-skills

Audit agent frameworks and agentic workflows for token-burning loop pathologies — unbounded retries, excessive planning depth, tool-call thrashing, context snowballing, missing loop budgets and kill switches. Use this skill whenever the user shares agent configs or traces (LangChain/LangGraph, CrewAI, AutoGen, custom loops), complains an agent is slow or expensive, or is designing any autonomous/multi-step agent. Part of Lean Agentic AI Skills; emits lean-findings.json.

2026-07-184

ai-eval-efficiency-audit

navveenb/lean-agentic-ai-skills

Audit AI evaluation infrastructure for wasted compute — full benchmark suites re-run on unchanged cases, LLM-as-judge grading without caching, oversized judge models, redundant eval passes per commit, and missing result reuse. Use this skill whenever the user shares eval harness configs or CI eval steps, complains that evals are slow or expensive, mentions LLM-as-judge costs, or runs benchmark suites on every change. Part of Lean Agentic AI Skills; emits lean-findings.json.

2026-07-184

api-efficiency-audit

navveenb/lean-agentic-ai-skills

Audit API design for chattiness and over-fetching — REST endpoints requiring N calls per screen, GraphQL resolvers with N+1 fan-out, default-everything payloads, missing pagination and field selection, provider-side polling where webhooks serve, and uncompressed responses. Use this skill whenever the user shares API schemas (OpenAPI, GraphQL SDL), resolver code, endpoint designs, or asks why clients make too many calls or payloads are heavy. Part of Lean Agentic AI Skills; emits lean-findings.json.

2026-07-184

caching-strategy-designer

navveenb/lean-agentic-ai-skills

Design caching layers so work is done once, not per request — HTTP/CDN caching, application-level caches (Redis/Memcached), computed-result caching, and cache-invalidation strategy. Use this skill whenever the user wants to add caching, asks why the same work is recomputed, designs a read-heavy system, or an audit skill (db-efficiency-audit, web-carbon-audit) flags uncached repeated work. Part of Lean Agentic AI Skills; emits a cache design plus lean-findings.json.

2026-07-184

carbon-aware-scheduler

navveenb/lean-agentic-ai-skills

Design time-shifting of flexible workloads (batch jobs, CI, training, backups, crons) to hours and regions when grid electricity is cleanest, using carbon-intensity signals like Electricity Maps, WattTime, or the Carbon Aware SDK. Use this skill whenever the user mentions carbon-aware computing, scheduling flexible/deferrable jobs, demand shifting, running workloads when energy is clean, or asks how to reduce the carbon intensity (not just amount) of their compute. Part of Lean Agentic AI Skills; emits schedule designs plus lean-findings.json.

2026-07-184

carbon-budget-designer

navveenb/lean-agentic-ai-skills

Design per-feature and per-service efficiency budgets — page-weight budgets, token budgets per AI feature, query budgets per endpoint, GPU-hour budgets per training project — with owners, review checkpoints, and enforcement hooks. Use this skill whenever the user wants sustainability targets for teams, asks how to keep efficiency from regressing organizationally, mentions performance-budget-style governance for carbon/energy, or is setting engineering standards. Part of Lean Agentic AI Skills; an advisor emitting budget definitions plus lean-findings.json.

2026-07-184

name

data-pipeline-audit

description

Data Pipeline Audit

Producer skill. Input: pipeline code (Spark/SQL/dbt), DAG definitions, query plans/profiles, table layouts and stats. Output: lean-findings.json.

Data pipelines are some of the heaviest scheduled compute in most organizations, and they re-run daily whether or not anything upstream changed. Waste here compounds on a schedule.

Subject type: emit subject.type: "data-pipeline" in findings.

Signatures

Full refresh where incremental serves — pipelines recomputing entire history when only the newest partition changed; dbt models without incremental materialization on append-mostly sources. High (daily × full-history). Fix: incremental strategy with a correctness note (late-arriving data handling must be designed, not ignored — name the lookback window).
Rebuilding unchanged upstreams — orchestration that runs every model/task on every schedule with no freshness/state check; dbt full runs where state-based selection (--select state:modified+) serves. High.
Missing partition pruning — queries scanning all partitions because filters don't match partition keys, or tables partitioned on the wrong column for the dominant query pattern. High. Evidence: bytes-scanned from query plans/profiles — real numbers, report them.
No predicate/projection pushdown — filters applied after full reads (e.g. filtering a DataFrame after loading whole tables; SELECT * feeding a job that uses three columns). High.
Row formats at rest — JSON/CSV in the lake where Parquet/ORC belongs; no compression on text formats. High: columnar + compression cuts both scan E and storage M. Fix names the format and the migration path.
Shuffle explosions — wide joins/groupBys without co-partitioning, skewed keys sending one executor 90% of the data, crossJoins hiding in plain sight. High. Evidence: shuffle read/write sizes from Spark UI stats when provided.
Small-files problem — thousands of tiny files per partition (streaming sinks, over-parallel writes); every job pays per-file open overhead. Medium-high. Fix: compaction jobs, target file sizes, auto-optimize features where the platform has them.
Oversized clusters idling — fixed-size clusters for bursty jobs; no autoscaling/auto-terminate on interactive clusters. Medium-high; route sizing specifics to right-size-compute, record the pipeline-level finding here.
Materializing the unread — tables/marts no downstream consumer has queried (from access logs/information_schema where available). Medium; flag for owner confirmation, never assert abandonment.
Chained recomputation of shared logic — the same expensive CTE/subquery computed in N models instead of one upstream materialization. Medium.

Honesty rules

Cost signal (countable)

Cost drivers: bytes scanned, compute-hours per job run, and job frequency. Bytes-scanned in particular is exposed by every serverless warehouse and is the honest cost driver to quote.

Not this skill's job

Warehouse/OLTP query tuning outside pipelines (db-efficiency-audit), storage tiering of the lake (storage-lifecycle-audit), what data to ingest at all (data-minimization-designer).