Polars is a Rust-implemented DataFrame library on top of Apache Arrow's columnar memory format. It exposes two execution modes: eager DataFrame (immediate compute, pandas-like) and lazy LazyFrame (query plan, optimized before execution). The point of using Polars over pandas is not "pandas with a faster backend" — it is a different programming model based on expressions that compose, parallelize across cores, and feed a cost-based query optimizer.
Polars wins decisively on large files, multi-key group-bys, window functions, and ETL with lazy IO. Pandas wins on tiny data, on libraries (sklearn / matplotlib / statsmodels) that consume pandas natively, and when the analyst writes ad-hoc one-liners in a notebook. The two coexist via / at boundaries.
to_pandas()
from_pandas()
Capabilities
LazyFrame, the query optimizer, and .collect()
pl.scan_csv / pl.scan_parquet / pl.scan_ndjson return a LazyFrame. Operations on it build a query plan; nothing executes until .collect(). The optimizer runs predicate pushdown, projection pushdown, slice pushdown, common-subplan elimination, expression simplification, join reordering, and type coercion. Inspect the plan with .explain(). See references/lazy-vs-eager.md.
Expression API — the core of Polars
Expressions (pl.col, pl.lit, pl.when().then().otherwise(), arithmetic, .alias, .over) are the lingua franca of Polars. They are declarative, composable, and the optimizer reasons over them. Expressions run inside contexts: select, with_columns, filter, group_by(...).agg, and .over(...) for window form. See references/expressions-api.md.
Streaming engine for out-of-core data
For datasets that exceed RAM, .collect(engine="streaming") and sink_parquet / sink_csv / sink_ipc stream chunks through the plan. Many but not all operations are streamable; the engine falls back where not supported. See references/streaming.md.
Group-by and aggregations with expression bodies
df.group_by("k").agg(pl.col("x").sum(), pl.col("y").mean()) — aggregations are arbitrary expressions, including filtered and conditional ones. group_by_dynamic handles timeseries bucketing. See references/groupby-aggregations.md.
Joins, including as-of
.join(other, on=, how="inner|left|outer|cross|anti|semi", validate=, suffix=) and .join_asof for time-aligned merges. See references/joins.md.
Window expressions via .over()
The polars-idiomatic way to do per-group rank, lag/lead, cumulative aggregates is pl.col("x").rank().over("k") — no groupby().apply(). This is where Polars dramatically outperforms pandas. See references/window-functions.md.
Strict, Arrow-backed type system
Polars dtypes (Int64, Float64, String, Boolean, Date, Datetime, Duration, Categorical, Enum, Struct, List, Array) are Arrow-native. Null semantics differ from pandas NaN — None / Null is a first-class value, not float NaN. See references/data-types.md.
Interop with pandas, NumPy, Arrow, DuckDB, PyTorch
to_pandas(), from_pandas(), to_arrow(), from_arrow(), to_numpy(). DuckDB reads/writes Polars directly. PyTorch dataloaders consume Polars via Arrow or NumPy. See references/interop.md.
IO surface — scan_* vs read_*, hive partitioning, databases
scan_* returns lazy frames with pushdown into IO. read_* is eager. pl.read_database / write_database use ConnectorX or ADBC. Hive partitions are auto-discovered on parquet scans. See references/io.md.
Migration from pandas
The single densest reference in this skill. Translates read_csv → scan_csv/read_csv, .loc[mask, cols] → .filter(...).select(...), .apply → expression, .rolling → pl.col().rolling_*, .groupby().agg(dict) → .group_by().agg([exprs]), dtype names, NaN ↔ Null. See references/migration-from-pandas.md.
Behavioral Traits
Always reaches for expressions before reaching for Python loops or .map_elements / apply.
Always uses scan_* + .collect() over read_* for files larger than a few hundred MB.
Always names lazy plans and inspects them with .explain() when performance is in question.
Always specifies schema= (or schema_overrides=) on scans when columns are known — avoids slow inference and silent type drift.
Prefers sink_parquet for outputs that don't need to live in memory.
Prefers .with_columns([...]) batched in a single call over chained one-column updates (single physical plan node).
Uses engine="streaming" for datasets that don't fit RAM; verifies the operation is streamable by reading .explain(streaming=True).
Treats Polars Null and pandas NaN as different concepts; converts explicitly at boundaries.
Important Constraints
NEVER iterate a DataFrame row-by-row in Python — use expressions or, last resort, .map_elements(strategy="thread_local") with a documented justification.
NEVER assume pandas NaN semantics. In Polars, Null is a distinct sentinel; is_null() is the test, not is_nan() (which only applies to floats).
NEVER shadow the pl import (pl = some_lazyframe is a classic foot-gun in notebooks).
NEVER load entire multi-GB Parquet datasets with pl.read_parquet when pl.scan_parquet(...).filter(...).select(...).collect(engine="streaming") would push the work down.
NEVER mix .groupby (pandas) into Polars code — the Polars API is .group_by (1.x) and the old .groupby alias was removed.
NEVER call .collect() more than once on the same LazyFrame if you can avoid it — cache with .cache() or materialize once and re-.lazy().
ALWAYS run .explain() on a slow query before assuming the optimizer is wrong; usually a missing filter or unnecessary with_columns is the cause.
ALWAYS prefer pl.col("x").over("k") for per-group transforms over group_by + join_back.
Related Skills
Parent / sibling data tooling
✓ python — language foundation (typing, packaging, environments)
How to use: open the specific topic file. migration-from-pandas.md is the densest single document for engineers coming from pandas. wrong-vs-right.md is the fastest correction loop for code review.
