Skip to main content Home Creators phoroth agentic polars
polars Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/phoroth/AGENTIC --skill polarsThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name polars description Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex. license https://github.com/pola-rs/polars/blob/main/LICENSE metadata {"skill-author":"K-Dense Inc."} risk unknown source community
Polars
When to Use
You need a faster in-memory DataFrame workflow than pandas for data that still fits in RAM.
You are building ETL, analytics, or transformation pipelines that benefit from lazy evaluation and parallel execution.
You want expression-based tabular operations on top of Apache Arrow semantics.
Overview
Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.
Quick Start
Installation and Basic Usage Basic DataFrame creation and operations:
import polars as pl
df = pl.DataFrame({
"name" : ["Alice" , "Bob" , "Charlie" ],
"age" : [25 , 30 , 35 ],
"city" : ["NY" , "LA" , "SF" ]
})
df.select("name" , "age" )
df.filter (pl.col("age" ) > 25 )
df.with_columns(
age_plus_10=pl.col("age" ) + 10
)
Core Concepts
Expressions Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.
Use pl.col("column_name") to reference columns
Chain methods to build complex transformations
Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)
df.select(
pl.col("name" ),
(pl.col("age" ) * 12 ).alias("age_in_months" )
)
Lazy vs Eager Evaluation Eager (DataFrame): Operations execute immediately
df = pl.read_csv("file.csv" )
result = df.filter (pl.col("age" ) > 25 )
Lazy (LazyFrame): Operations build a query plan, optimized before execution
lf = pl.scan_csv("file.csv" )
result = lf.filter (pl.col("age" ) > 25 ).select("name" , "age" )
df = result.collect()
Working with large datasets
Complex query pipelines
When only some columns/rows are needed
Performance is critical
Benefits of lazy evaluation:
Automatic query optimization
Predicate pushdown
Projection pushdown
Parallel execution
For detailed concepts, load references/core_concepts.md.
Common Operations
Select Select and manipulate columns:
df.select("name" , "age" )
df.select(
pl.col("name" ),
(pl.col("age" ) * 2 ).alias("double_age" )
)
df.select(pl.col("^.*_id$" ))
Filter Filter rows by conditions:
df.filter (pl.col("age" ) > 25 )
df.filter (
pl.col("age" ) > 25 ,
pl.col("city" ) == "NY"
)
df.filter (
(pl.col("age" ) > 25 ) | (pl.col("city" ) == "LA" )
)
With Columns Add or modify columns while preserving existing ones:
df.with_columns(
age_plus_10=pl.col("age" ) + 10 ,
name_upper=pl.col("name" ).str .to_uppercase()
)
df.with_columns(
pl.col("value" ) * 10 ,
pl.col("value" ) * 100 ,
)
Group By and Aggregations Group data and compute aggregations:
df.group_by("city" ).agg(
pl.col("age" ).mean().alias("avg_age" ),
pl.len ().alias("count" )
)
df.group_by("city" , "department" ).agg(
pl.col("salary" ).sum ()
)
df.group_by("city" ).agg(
(pl.col("age" ) > 30 ).sum ().alias("over_30" )
)
For detailed operation patterns, load references/operations.md.
Aggregations and Window Functions
Aggregation Functions Common aggregations within group_by context:
pl.len() - count rowspl.col("x").sum() - sum valuespl.col("x").mean() - averagepl.col("x").min() / pl.col("x").max() - extremespl.first() / pl.last() - first/last values
Window Functions with over() Apply aggregations while preserving row count:
df.with_columns(
avg_age_by_city=pl.col("age" ).mean().over("city" ),
rank_in_city=pl.col("salary" ).rank().over("city" )
)
df.with_columns(
group_avg=pl.col("value" ).mean().over("category" , "region" )
)
group_to_rows (default): Preserves original row orderexplode: Faster but groups rows togetherjoin: Creates list columns
Data I/O
Supported Formats Polars supports reading and writing:
CSV, Parquet, JSON, Excel
Databases (via connectors)
Cloud storage (S3, Azure, GCS)
Google BigQuery
Multiple/partitioned files
Common I/O Operations
df = pl.read_csv("file.csv" )
df.write_csv("output.csv" )
lf = pl.scan_csv("file.csv" )
result = lf.filter (...).select(...).collect()
Parquet (recommended for performance):
df = pl.read_parquet("file.parquet" )
df.write_parquet("output.parquet" )
df = pl.read_json("file.json" )
df.write_json("output.json" )
For comprehensive I/O documentation, load references/io_guide.md.
Transformations
Joins
df1.join(df2, on="id" , how="inner" )
df1.join(df2, on="id" , how="left" )
df1.join(df2, left_on="user_id" , right_on="id" )
Concatenation
pl.concat([df1, df2], how="vertical" )
pl.concat([df1, df2], how="horizontal" )
pl.concat([df1, df2], how="diagonal" )
Pivot and Unpivot
df.pivot(values="sales" , index="date" , columns="product" )
df.unpivot(index="id" , on=["col1" , "col2" ])
For detailed transformation examples, load references/transformations.md.
Pandas Migration Polars offers significant performance improvements over pandas with a cleaner API. Key differences:
Conceptual Differences
No index : Polars uses integer positions onlyStrict typing : No silent type conversionsLazy evaluation : Available via LazyFrameParallel by default : Operations parallelized automatically
Common Operation Mappings Operation Pandas Polars Select column df["col"]df.select("col")Filter df[df["col"] > 10]df.filter(pl.col("col") > 10)Add column df.assign(x=...)df.with_columns(x=...)Group by df.groupby("col").agg(...)df.group_by("col").agg(...)Window df.groupby("col").transform(...)df.with_columns(...).over("col")
Key Syntax Patterns Pandas sequential (slow):
df.assign(
col_a=lambda df_: df_.value * 10 ,
col_b=lambda df_: df_.value * 100
)
df.with_columns(
col_a=pl.col("value" ) * 10 ,
col_b=pl.col("value" ) * 100 ,
)
For comprehensive migration guide, load references/pandas_migration.md.
Best Practices
Performance Optimization
Use lazy evaluation for large datasets:
lf = pl.scan_csv("large.csv" )
result = lf.filter (...).select(...).collect()
Avoid Python functions in hot paths:
Stay within expression API for parallelization
Use .map_elements() only when necessary
Prefer native Polars operations
Use streaming for very large data:
lf.collect(streaming=True )
Select only needed columns early:
lf.select("col1" , "col2" ).filter (...)
lf.filter (...).select("col1" , "col2" )
Use appropriate data types:
Categorical for low-cardinality strings
Appropriate integer sizes (i32 vs i64)
Date types for temporal data
Expression Patterns pl.when(condition).then(value).otherwise(other_value)
Column operations across multiple columns:
df.select(pl.col("^.*_value$" ) * 2 )
pl.col("x" ).fill_null(0 )
pl.col("x" ).is_null()
pl.col("x" ).drop_nulls()
For additional best practices and patterns, load references/best_practices.md.
Resources This skill includes comprehensive reference documentation:
references/
core_concepts.md - Detailed explanations of expressions, lazy evaluation, and type systemoperations.md - Comprehensive guide to all common operations with examplespandas_migration.md - Complete migration guide from pandas to Polarsio_guide.md - Data I/O operations for all supported formatstransformations.md - Joins, concatenation, pivots, and reshaping operationsbest_practices.md - Performance optimization tips and common patterns Load these references as needed when users require detailed information about specific topics.
Limitations
Use this skill only when the task clearly matches the scope described above.
Do not treat the output as a substitute for enprojectnment-specific validation, testing, or expert review.
Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
