Skip to main content Accueil Créateurs synthetic-sciences openscience polars
polars Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/synthetic-sciences/openscience --skill polarsLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... Métiers associés SOC
Basé sur la classification professionnelle SOC
Explorateur de fichiers
7 fichiers name polars description Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex. category data-engineering license https://github.com/pola-rs/polars/blob/main/LICENSE metadata {"skill-author":"Synthetic Sciences"} version 1.0.0 author Synthetic Sciences tags ["DataFrames","ETL","Data Processing","Performance"] dependencies ["polars>=1.0.0"]
Polars
Overview
Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.
Quick Start
Installation and Basic Usage
Install Polars:
uv pip install polars
Basic DataFrame creation and operations:
import polars as pl
df = pl.DataFrame({
"name" : ["Alice" , "Bob" , "Charlie" ],
"age" : [25 , 30 , 35 ],
"city" : ["NY" , "LA" , "SF" ]
})
df.select("name" , "age" )
df.filter (pl.col("age" ) > 25 )
df.with_columns(
age_plus_10=pl.col("age" ) + 10
)
Core Concepts
Expressions
Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.
Key principles:
Use pl.col("column_name") to reference columns
Chain methods to build complex transformations
Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)
Example:
df.select(
pl.col("name" ),
(pl.col("age" ) * 12 ).alias("age_in_months"
Lazy vs Eager Evaluation Eager (DataFrame): Operations execute immediately
df = pl.read_csv("file.csv" )
result = df.filter (pl.col("age" ) > 25 )
Lazy (LazyFrame): Operations build a query plan, optimized before execution
lf = pl.scan_csv("file.csv" )
result = lf.filter (pl.col("age" ) > 25 ).select("name" , "age" )
df = result.collect()
Working with large datasets
Complex query pipelines
When only some columns/rows are needed
Performance is critical
Benefits of lazy evaluation:
Automatic query optimization
Predicate pushdown
Projection pushdown
Parallel execution
For detailed concepts, load references/core_concepts.md.
Common Operations
Select Select and manipulate columns:
df.select("name" , "age" )
df.select(
pl.col("name" ),
(pl.col("age" ) * 2 ).alias("double_age" )
)
df.select(pl.col("^.*_id$" ))
Filter Filter rows by conditions:
df.filter (pl.col("age" ) > 25 )
df.filter (
pl.col("age" ) > 25 ,
pl.col("city" ) == "NY"
)
df.filter (
(pl.col("age" ) > 25 ) | (pl.col("city" ) == "LA" )
)
With Columns Add or modify columns while preserving existing ones:
df.with_columns(
age_plus_10=pl.col("age" ) + 10 ,
name_upper=pl.col("name" ).str .to_uppercase()
)
df.with_columns(
pl.col("value" ) * 10 ,
pl.col("value" ) * 100 ,
)
Group By and Aggregations Group data and compute aggregations:
df.group_by("city" ).agg(
pl.col("age" ).mean().alias("avg_age" ),
pl.len ().alias("count" )
)
df.group_by("city" , "department" ).agg(
pl.col("salary" ).sum ()
)
df.group_by("city" ).agg(
(pl.col("age" ) > 30 ).sum ().alias("over_30" )
)
For detailed operation patterns, load references/operations.md.
Aggregations and Window Functions
Aggregation Functions Common aggregations within group_by context:
pl.len() - count rowspl.col("x").sum() - sum valuespl.col("x").mean() - averagepl.col("x").min() / pl.col("x").max() - extremespl.first() / pl.last() - first/last values
Window Functions with over() Apply aggregations while preserving row count:
df.with_columns(
avg_age_by_city=pl.col("age" ).mean().over("city" ),
rank_in_city=pl.col("salary" ).rank().over("city" )
)
df.with_columns(
group_avg=pl.col("value" ).mean().over("category" , "region" )
)
group_to_rows (default): Preserves original row orderexplode: Faster but groups rows togetherjoin: Creates list columns
Data I/O
Supported Formats Polars supports reading and writing:
CSV, Parquet, JSON, Excel
Databases (via connectors)
Cloud storage (S3, Azure, GCS)
Google BigQuery
Multiple/partitioned files
Common I/O Operations
df = pl.read_csv("file.csv" )
df.write_csv("output.csv" )
lf = pl.scan_csv("file.csv" )
result = lf.filter (...).select(...).collect()
Parquet (recommended for performance):
df = pl.read_parquet("file.parquet" )
df.write_parquet("output.parquet" )
df = pl.read_json("file.json" )
df.write_json("output.json" )
For comprehensive I/O documentation, load references/io_guide.md.
Transformations
Joins
df1.join(df2, on="id" , how="inner" )
df1.join(df2, on="id" , how="left" )
df1.join(df2, left_on="user_id" , right_on="id" )
Concatenation
pl.concat([df1, df2], how="vertical" )
pl.concat([df1, df2], how="horizontal" )
pl.concat([df1, df2], how="diagonal" )
Pivot and Unpivot
df.pivot(values="sales" , index="date" , columns="product" )
df.unpivot(index="id" , on=["col1" , "col2" ])
For detailed transformation examples, load references/transformations.md.
Pandas Migration Polars offers significant performance improvements over pandas with a cleaner API. Key differences:
Conceptual Differences
No index : Polars uses integer positions onlyStrict typing : No silent type conversionsLazy evaluation : Available via LazyFrameParallel by default : Operations parallelized automatically
Common Operation Mappings Operation Pandas Polars Select column df["col"]df.select("col")Filter df[df["col"] > 10]df.filter(pl.col("col") > 10)Add column df.assign(x=...)df.with_columns(x=...)Group by df.groupby("col").agg(...)df.group_by("col").agg(...)Window df.groupby("col").transform(...)df.with_columns(...).over("col")
Key Syntax Patterns Pandas sequential (slow):
df.assign(
col_a=lambda df_: df_.value * 10 ,
col_b=lambda df_: df_.value * 100
)
df.with_columns(
col_a=pl.col("value" ) * 10 ,
col_b=pl.col("value" ) * 100 ,
)
For comprehensive migration guide, load references/pandas_migration.md.
Best Practices
Performance Optimization
Use lazy evaluation for large datasets:
lf = pl.scan_csv("large.csv" )
result = lf.filter (...).select(...).collect()
Avoid Python functions in hot paths:
Stay within expression API for parallelization
Use .map_elements() only when necessary
Prefer native Polars operations
Use streaming for very large data:
lf.collect(streaming=True )
Select only needed columns early:
lf.select("col1" , "col2" ).filter (...)
lf.filter (...).select("col1" , "col2" )
Use appropriate data types:
Categorical for low-cardinality strings
Appropriate integer sizes (i32 vs i64)
Date types for temporal data
Expression Patterns pl.when(condition).then(value).otherwise(other_value)
Column operations across multiple columns:
df.select(pl.col("^.*_value$" ) * 2 )
pl.col("x" ).fill_null(0 )
pl.col("x" ).is_null()
pl.col("x" ).drop_nulls()
For additional best practices and patterns, load references/best_practices.md.
Resources This skill includes comprehensive reference documentation:
references/
core_concepts.md - Detailed explanations of expressions, lazy evaluation, and type systemoperations.md - Comprehensive guide to all common operations with examplespandas_migration.md - Complete migration guide from pandas to Polarsio_guide.md - Data I/O operations for all supported formatstransformations.md - Joins, concatenation, pivots, and reshaping operationsbest_practices.md - Performance optimization tips and common patterns Load these references as needed when users require detailed information about specific topics.
