batch-processing-optimizer

license	Apache-2.0
name	batch-processing-optimizer
description	Spark, pandas, polars, DuckDB optimization for batch data processing. Activate on: batch processing, Spark optimization, polars, DuckDB, pandas performance, data frame, shuffle, partition, memory optimization. NOT for: streaming pipelines (use streaming-pipeline-architect), warehouse queries (use data-warehouse-optimizer).
allowed-tools	Read,Write,Edit,Bash(npm:*,npx:*,python:*,spark-submit:*)
category	Backend & Infrastructure
tags	["batch-processing","spark","polars","duckdb","performance"]
pairs-with	[{"skill":"data-warehouse-optimizer","reason":"Batch outputs often load into warehouses"},{"skill":"airflow-dag-orchestrator","reason":"Airflow schedules batch processing jobs"},{"skill":"lakehouse-architect","reason":"Batch jobs read/write lakehouse tables"}]

Batch Processing Optimizer

Optimize batch data processing workloads using Spark, Polars, DuckDB, and pandas with focus on memory efficiency, parallelism, and cost reduction.

Activation Triggers

Activate on: "batch processing", "Spark optimization", "Polars", "DuckDB", "pandas performance", "data frame", "shuffle optimization", "partition skew", "memory optimization", "out of memory"

NOT for: Real-time streaming → streaming-pipeline-architect | Warehouse SQL tuning → data-warehouse-optimizer | Pipeline orchestration → airflow-dag-orchestrator

Quick Start

1. Choose the right tool — DuckDB for single-node analytics, Polars for DataFrames, Spark for distributed
2. Profile first — identify bottlenecks (shuffle, skew, memory) before optimizing
3. Reduce data early — filter and select columns as early as possible in the pipeline
4. Avoid shuffles — broadcast small tables, pre-partition data, use map-side joins
5. Right-size resources — match executor memory/cores to actual data size

Core Capabilities

Domain	Technologies
Distributed	Apache Spark 3.5+, Dask, Ray
Single-Node	DuckDB 1.1+, Polars 1.x, pandas 2.2+
File Formats	Parquet, Arrow IPC, Delta Lake, Iceberg
Optimization	AQE (Spark), lazy evaluation (Polars), columnar scans
Cloud	Databricks, EMR, Dataproc, serverless Spark

Architecture Patterns

Tool Selection Decision Tree

Data Size?
  ├─ < 10 GB     → DuckDB (SQL) or Polars (DataFrame)
  │                 Single machine, zero setup, fastest iteration
  │
  ├─ 10-100 GB   → Polars (lazy) or DuckDB (out-of-core)
  │                 Still single machine with spill-to-disk
  │
  └─ > 100 GB    → Spark (distributed)
                    Multi-node cluster, shuffle-based joins

Complexity?
  ├─ SQL-centric  → DuckDB (fastest SQL engine for analytics)
  ├─ DataFrame    → Polars (10x faster than pandas, lazy evaluation)
  └─ Complex ML   → Spark + MLlib or Spark + Ray

Spark Optimization Patterns

 pyspark.sql  SparkSession
 pyspark.sql.functions  F

spark = SparkSession.builder \
    .config(, ) \
    .config(, ) \
    .config(, ) \
    .getOrCreate()


 pyspark.sql.functions  broadcast
result = large_df.join(broadcast(small_dim_df), )


orders = spark.read.parquet() \
    .(F.col() >= ) \
    .select(, , )  





result.repartition() \
    .write.mode() \
    .partitionBy() \
    .parquet()