| license | Apache-2.0 |
| name | batch-processing-optimizer |
| description | Spark, pandas, polars, DuckDB optimization for batch data processing. Activate on: batch processing, Spark optimization, polars, DuckDB, pandas performance, data frame, shuffle, partition, memory optimization. NOT for: streaming pipelines (use streaming-pipeline-architect), warehouse queries (use data-warehouse-optimizer). |
| allowed-tools | Read,Write,Edit,Bash(npm:*,npx:*,python:*,spark-submit:*) |
| category | Backend & Infrastructure |
| tags | ["batch-processing","spark","polars","duckdb","performance"] |
| pairs-with | [{"skill":"data-warehouse-optimizer","reason":"Batch outputs often load into warehouses"},{"skill":"airflow-dag-orchestrator","reason":"Airflow schedules batch processing jobs"},{"skill":"lakehouse-architect","reason":"Batch jobs read/write lakehouse tables"}] |
Batch Processing Optimizer
Optimize batch data processing workloads using Spark, Polars, DuckDB, and pandas with focus on memory efficiency, parallelism, and cost reduction.
Activation Triggers
Activate on: "batch processing", "Spark optimization", "Polars", "DuckDB", "pandas performance", "data frame", "shuffle optimization", "partition skew", "memory optimization", "out of memory"
NOT for: Real-time streaming → streaming-pipeline-architect | Warehouse SQL tuning → data-warehouse-optimizer | Pipeline orchestration → airflow-dag-orchestrator
Quick Start
- Choose the right tool — DuckDB for single-node analytics, Polars for DataFrames, Spark for distributed
- Profile first — identify bottlenecks (shuffle, skew, memory) before optimizing
- Reduce data early — filter and select columns as early as possible in the pipeline
- Avoid shuffles — broadcast small tables, pre-partition data, use map-side joins
- Right-size resources — match executor memory/cores to actual data size
Core Capabilities
| Domain | Technologies |
|---|
| Distributed | Apache Spark 3.5+, Dask, Ray |
| Single-Node | DuckDB 1.1+, Polars 1.x, pandas 2.2+ |
| File Formats | Parquet, Arrow IPC, Delta Lake, Iceberg |
| Optimization | AQE (Spark), lazy evaluation (Polars), columnar scans |
| Cloud | Databricks, EMR, Dataproc, serverless Spark |
Architecture Patterns
Tool Selection Decision Tree
Data Size?
├─ < 10 GB → DuckDB (SQL) or Polars (DataFrame)
│ Single machine, zero setup, fastest iteration
│
├─ 10-100 GB → Polars (lazy) or DuckDB (out-of-core)
│ Still single machine with spill-to-disk
│
└─ > 100 GB → Spark (distributed)
Multi-node cluster, shuffle-based joins
Complexity?
├─ SQL-centric → DuckDB (fastest SQL engine for analytics)
├─ DataFrame → Polars (10x faster than pandas, lazy evaluation)
└─ Complex ML → Spark + MLlib or Spark + Ray
Spark Optimization Patterns
pyspark.sql SparkSession
pyspark.sql.functions F
spark = SparkSession.builder \
.config(, ) \
.config(, ) \
.config(, ) \
.getOrCreate()
pyspark.sql.functions broadcast
result = large_df.join(broadcast(small_dim_df), )
orders = spark.read.parquet() \
.(F.col() >= ) \
.select(, , )
result.repartition() \
.write.mode() \
.partitionBy() \
.parquet()
