Blazingly fast DataFrame library written in Rust. Features a multi-threaded query engine, lazy evaluation, and efficient memory usage via Apache Arrow. Designed for high-performance data processing on a single machine. Use for large datasets (1GB-100GB+), fast data transformations, Parquet/CSV processing, complex query pipelines, memory-efficient operations, and when speed is critical (10-100x faster than pandas).
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Blazingly fast DataFrame library written in Rust. Features a multi-threaded query engine, lazy evaluation, and efficient memory usage via Apache Arrow. Designed for high-performance data processing on a single machine. Use for large datasets (1GB-100GB+), fast data transformations, Parquet/CSV processing, complex query pipelines, memory-efficient operations, and when speed is critical (10-100x faster than pandas).
version
0.2
license
MIT
Polars - High-Performance Dataframes
Polars is designed for speed. Unlike pandas, which processes data sequentially on a single CPU core, Polars parallelizes operations across all available cores. Its "Lazy API" allows it to optimize queries before execution, significantly reducing memory overhead and processing time.
When to Use
Processing large datasets (1GB - 100GB+) that struggle in pandas.
When execution speed is a priority (Polars is often 10-100x faster than pandas).
Working with complex data transformation pipelines (Lazy evaluation).
Systems with limited RAM (Polars is more memory-efficient than pandas).
Situations requiring strict type safety and consistent null handling.
Reading/writing large Parquet, CSV, or Avro files.
Eager: Operations are executed immediately (like pandas).
Lazy: Operations are queued into a query plan. Polars optimizes the plan (e.g., predicate pushdown, projection pushdown) and executes it only when called.
The Expression API
Polars uses a declarative syntax. Instead of writing loops or complex lambdas, you write expressions using pl.col(). These expressions are highly optimized and run in parallel.
Apache Arrow
Polars stores data in the Apache Arrow format, enabling zero-copy data exchange with other tools like PyArrow and DuckDB.
Quick Reference
Installation
pip install polars
# For Excel/Cloud support
pip install 'polars[all]'
Prefer Lazy API (scan_*) - This allows Polars to optimize memory and skip unnecessary data.
Use Expressions - Always use pl.col("name") instead of selecting columns via strings or indices.
Method Chaining - Polars is built for clean, readable pipelines.
Specify Schema - When reading CSVs, providing a schema prevents type inference errors and speeds up loading.
Use collect(streaming=True) - For datasets larger than RAM, streaming allows Polars to process data in chunks.
Parquet over CSV - Use Parquet for permanent storage; it is significantly faster and stores type information.
❌ DON'T
Avoid .apply() - Custom Python functions are slow because they break the Rust/parallel optimization. Use built-in expressions.
Don't use inplace=True - Polars (like JAX) favors immutability; transformations return new DataFrames.
Don't convert to pandas early - Keep data in Polars as long as possible to maintain speed.
Avoid Row Iteration - for row in df is an anti-pattern; use vectorized expressions.
Anti-Patterns (NEVER)
import polars as pl
# ❌ BAD: Using Python lambdas for simple math# df.select(pl.col("val").map_elements(lambda x: x * 2)) # Slow!# ✅ GOOD: Use expressions
df.select(pl.col("val") * 2) # Fast, parallelized in Rust# ❌ BAD: Filtering after a heavy operation# df.group_by("id").mean().filter(pl.col("id") == 5)# ✅ GOOD: Lazy API will automatically "push down" the filter
(pl.scan_csv("data.csv")
.filter(pl.col("id") == 5) # Optimized to read only id=5
.group_by("id").mean())
# ❌ BAD: Converting to pandas just to check .head()# df.to_pandas().head() # ✅ GOOD: Polars has its own fast .head() and rich printingprint(df.head())
Expression API Deep Dive
Selection and Transformation
df.select([
pl.col("name"),
pl.col("price") * 1.2, # Scalar math
pl.col("category").str.to_uppercase(), # String methods
pl.col("date").dt.year().alias("year") # Date methods
])
Instead of creating one column at a time, use with_columns to run multiple calculations in parallel.
# All 3 columns are calculated simultaneously in different threads
df = df.with_columns([
(pl.col("a") + pl.col("b")).alias("sum"),
(pl.col("a") * pl.col("b")).alias("prod"),
pl.col("c").str.len().alias("c_len")
])
Column Selection via Dtypes
Rapidly apply transformations to groups of columns.
# Multiply all float columns by 100
df = df.with_columns(
pl.col(pl.Float64) * 100
)
Common Pitfalls and Solutions
The .apply() Trap
Python functions in .map_elements() (formerly .apply()) are slow.
# ❌ Problem: Using custom Python code# df.select(pl.col("txt").map_elements(my_custom_func))# ✅ Solution: Use Polars native expressions or pl.when()
df.select(
pl.when(pl.col("score") > 50).then(pl.lit("Pass")).otherwise(pl.lit("Fail"))
)
Memory Errors on Large Files
If you hit OOM with .collect(), you might be trying to load too much data into memory.
# ✅ Solution: # 1. Use .filter() early in the Lazy plan.# 2. Use streaming: .collect(streaming=True).# 3. Select only the columns you need.
String vs Categorical
For low-cardinality strings (like "City" or "Gender"), use Categorical.
# ✅ Solution: Saves massive amounts of RAM and speeds up joins
df = df.with_columns(pl.col("category").cast(pl.Categorical))
Best Practices
Always use Lazy API for large files - Start with scan_csv() or scan_parquet() instead of read_csv() or read_parquet().
Build complete query plans before collecting - Let Polars optimize the entire pipeline.
Use expressions over Python functions - Leverage pl.col() expressions for maximum performance.
Specify schemas when reading CSVs - Prevents type inference overhead and errors.
Use streaming for out-of-memory datasets - Enable streaming=True in collect().
Prefer Parquet format - Faster reads/writes and preserves type information.
Cast to Categorical for low-cardinality strings - Significant memory and performance gains.
Use with_columns for multiple transformations - Parallelizes column creation.
Filter early in lazy queries - Predicate pushdown reduces data scanned.
Avoid converting to pandas - Stay in Polars ecosystem for maximum speed.
Polars is the new gold standard for single-node data processing. By combining the safety of Rust with the flexibility of Python, it provides a seamless and incredibly fast experience for modern data science.
