Skip to main content Inicio Creadores synthetic-sciences openscience polars
polars Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.
Ir a la instalación Skills Marketplace Descubre y explora habilidades de IA creadas por la comunidad.
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Mostrar detalles del prompt Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.
npx skills add https://github.com/synthetic-sciences/openscience --skill polarsEl comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Descargar Zip Descargando... Más de este repositorio Diffusion-based molecular docking. Predict protein-ligand binding poses from PDB/SMILES, confidence scores, virtual screening, for structure-based drug design. Not for affinity prediction.
Fast inference and fine-tuning platform with serverless and on-demand GPU deployments. OpenAI-compatible API for chat completions, embeddings, function calling, vision, and structured output. Supports SFT, DPO, and RL fine-tuning. SOC2 + HIPAA compliant.
Serverless inference, fine-tuning, embeddings, image generation, and batch processing on 200+ open-source models via an OpenAI-compatible API. Use when you need fast, cost-effective access to open-source LLMs without managing infrastructure.
Ocupaciones relacionadas SOC
Basado en la clasificación ocupacional SOC
Explorador de archivos
7 archivos name polars description Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex. category data-engineering license https://github.com/pola-rs/polars/blob/main/LICENSE metadata {"skill-author":"Synthetic Sciences"} version 1.0.0 author Synthetic Sciences tags ["DataFrames","ETL","Data Processing","Performance"] dependencies ["polars>=1.0.0"]
Polars
Overview
Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.
Quick Start
Installation and Basic Usage
Install Polars:
uv pip install polars
Basic DataFrame creation and operations:
import polars as pl
df = pl.DataFrame({
"name" : ["Alice" , "Bob" , "Charlie" ],
"age" : [25 , 30 , 35 ],
"city" : ["NY" , "LA" , "SF" ]
})
df.select("name" , "age" )
df.filter (pl.col("age" ) > 25 )
df.with_columns(
age_plus_10=pl.col("age" ) + 10
)
Core Concepts
Expressions
Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.
Key principles:
Use pl.col("column_name") to reference columns
Chain methods to build complex transformations
Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)
Example:
df.select(
pl.col("name" ),
(pl.col("age" ) * 12 ).alias("age_in_months"
Lazy vs Eager Evaluation Eager (DataFrame): Operations execute immediately
df = pl.read_csv("file.csv" )
result = df.filter (pl.col("age" ) > 25 )
Lazy (LazyFrame): Operations build a query plan, optimized before execution
lf = pl.scan_csv("file.csv" )
result = lf.filter (pl.col("age" ) > 25 ).select("name" , "age" )
df = result.collect()
Working with large datasets
Complex query pipelines
When only some columns/rows are needed
Performance is critical
Benefits of lazy evaluation:
Automatic query optimization
Predicate pushdown
Projection pushdown
Parallel execution
For detailed concepts, load references/core_concepts.md.
Common Operations
Select Select and manipulate columns:
df.select("name" , "age" )
df.select(
pl.col("name" ),
(pl.col("age" ) * 2 ).alias("double_age" )
)
df.select(pl.col("^.*_id$" ))
Filter Filter rows by conditions:
df.filter (pl.col("age" ) > 25 )
df.filter (
pl.col("age" ) > 25 ,
pl.col("city" ) == "NY"
)
df.filter (
(pl.col("age" ) > 25 ) | (pl.col("city" ) == "LA" )
)
With Columns Add or modify columns while preserving existing ones:
df.with_columns(
age_plus_10=pl.col("age" ) + 10 ,
name_upper=pl.col("name" ).str .to_uppercase()
)
df.with_columns(
pl.col("value" ) * 10 ,
pl.col("value" ) * 100 ,
)
Group By and Aggregations Group data and compute aggregations:
df.group_by("city" ).agg(
pl.col("age" ).mean().alias("avg_age" ),
pl.len ().alias("count" )
)
df.group_by("city" , "department" ).agg(
pl.col("salary" ).sum ()
)
df.group_by("city" ).agg(
(pl.col("age" ) > 30 ).sum ().alias("over_30" )
)
For detailed operation patterns, load references/operations.md.
Aggregations and Window Functions
Aggregation Functions Common aggregations within group_by context:
pl.len() - count rowspl.col("x").sum() - sum valuespl.col("x").mean() - averagepl.col("x").min() / pl.col("x").max() - extremespl.first() / pl.last() - first/last values
Window Functions with over() Apply aggregations while preserving row count:
df.with_columns(
avg_age_by_city=pl.col("age" ).mean().over("city" ),
rank_in_city=pl.col("salary" ).rank().over("city" )
)
df.with_columns(
group_avg=pl.col("value" ).mean().over("category" , "region" )
)
group_to_rows (default): Preserves original row orderexplode: Faster but groups rows togetherjoin: Creates list columns
Data I/O
Supported Formats Polars supports reading and writing:
CSV, Parquet, JSON, Excel
Databases (via connectors)
Cloud storage (S3, Azure, GCS)
Google BigQuery
Multiple/partitioned files
Common I/O Operations
df = pl.read_csv("file.csv" )
df.write_csv("output.csv" )
lf = pl.scan_csv("file.csv" )
result = lf.filter (...).select(...).collect()
Parquet (recommended for performance):
df = pl.read_parquet("file.parquet" )
df.write_parquet("output.parquet" )
df = pl.read_json("file.json" )
df.write_json("output.json" )
For comprehensive I/O documentation, load references/io_guide.md.
Transformations
Joins
df1.join(df2, on="id" , how="inner" )
df1.join(df2, on="id" , how="left" )
df1.join(df2, left_on="user_id" , right_on="id" )
Concatenation
pl.concat([df1, df2], how="vertical" )
pl.concat([df1, df2], how="horizontal" )
pl.concat([df1, df2], how="diagonal" )
Pivot and Unpivot
df.pivot(values="sales" , index="date" , columns="product" )
df.unpivot(index="id" , on=["col1" , "col2" ])
For detailed transformation examples, load references/transformations.md.
Pandas Migration Polars offers significant performance improvements over pandas with a cleaner API. Key differences:
Conceptual Differences
No index : Polars uses integer positions onlyStrict typing : No silent type conversionsLazy evaluation : Available via LazyFrameParallel by default : Operations parallelized automatically
Common Operation Mappings Operation Pandas Polars Select column df["col"]df.select("col")Filter df[df["col"] > 10]df.filter(pl.col("col") > 10)Add column df.assign(x=...)df.with_columns(x=...)Group by df.groupby("col").agg(...)df.group_by("col").agg(...)Window df.groupby("col").transform(...)df.with_columns(...).over("col")
Key Syntax Patterns Pandas sequential (slow):
df.assign(
col_a=lambda df_: df_.value * 10 ,
col_b=lambda df_: df_.value * 100
)
df.with_columns(
col_a=pl.col("value" ) * 10 ,
col_b=pl.col("value" ) * 100 ,
)
For comprehensive migration guide, load references/pandas_migration.md.
Best Practices
Performance Optimization
Use lazy evaluation for large datasets:
lf = pl.scan_csv("large.csv" )
result = lf.filter (...).select(...).collect()
Avoid Python functions in hot paths:
Stay within expression API for parallelization
Use .map_elements() only when necessary
Prefer native Polars operations
Use streaming for very large data:
lf.collect(streaming=True )
Select only needed columns early:
lf.select("col1" , "col2" ).filter (...)
lf.filter (...).select("col1" , "col2" )
Use appropriate data types:
Categorical for low-cardinality strings
Appropriate integer sizes (i32 vs i64)
Date types for temporal data
Expression Patterns pl.when(condition).then(value).otherwise(other_value)
Column operations across multiple columns:
df.select(pl.col("^.*_value$" ) * 2 )
pl.col("x" ).fill_null(0 )
pl.col("x" ).is_null()
pl.col("x" ).drop_nulls()
For additional best practices and patterns, load references/best_practices.md.
Resources This skill includes comprehensive reference documentation:
references/
core_concepts.md - Detailed explanations of expressions, lazy evaluation, and type systemoperations.md - Comprehensive guide to all common operations with examplespandas_migration.md - Complete migration guide from pandas to Polarsio_guide.md - Data I/O operations for all supported formatstransformations.md - Joins, concatenation, pivots, and reshaping operationsbest_practices.md - Performance optimization tips and common patterns Load these references as needed when users require detailed information about specific topics.
