Skip to main content Skills Marketplace Descubra e explore skills de IA criadas pela comunidade.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Mostrar detalhes do prompt Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
npx skills add https://github.com/vamseeachanta/workspace-hub --skill polarsO comando permanece em uma só linha. Role horizontalmente para revisá-lo antes de copiar.
Prefere uma cópia local? Baixe os arquivos disponíveis atualmente no SkillsMP.
Baixar Zip Baixando... Mais deste repositório Write outbound email and external messages in Vamsee Achanta's voice — a subtle offer to help, never bold or rash claims. Load before drafting ANY email, LinkedIn/Collide reply, proposal note, or outreach sent under his name.
Save/publish analysis or computation results from ANY ecosystem repo to Hugging Face as a queryable, viewer-renderable dataset. Use when the user wants to "save results to hugging face", "publish dataset to HF", "hugging face data saving", "save analysis results", "hf dataset", "make results queryable", or "render via datasets-server API". Reshapes nested results into flat parquet tables, writes a dataset card with a viewer `configs:` block and provenance, applies license/public-vs-private routing, enforces a domain data-quality gate (faithful-to-source != correct), publishes to `aceengineer/<repo>-<projection>`, and verifies via the datasets-server API.
Clone, create, fork, configure, and manage GitHub repositories. Manage remotes, secrets, releases, and workflows. Works with gh CLI or falls back to git + GitHub REST API via curl.
Ocupações relacionadas SOC
Baseado na classificação ocupacional SOC
Explorador de arquivos
2 arquivos name polars version 1.0.0 description High-performance DataFrame library for fast data processing with lazy evaluation, parallel execution, and memory efficiency author workspace-hub category data-analysis capabilities ["Lazy evaluation and query optimization","Parallel processing on all CPU cores","Memory-efficient operations for large datasets","Expression-based API for complex transformations","Streaming processing for out-of-memory datasets","Zero-copy data sharing with Arrow"] tools ["polars","pyarrow","connectorx"] tags ["polars","dataframe","performance","parallel","lazy-evaluation","arrow","rust","data-processing"] platforms ["python","rust"] related_skills ["pandas-data-processing","numpy-numerical-analysis","streamlit","dash"]
Polars High-Performance DataFrame Skill
Master Polars for blazing-fast data processing with lazy evaluation, parallel execution, and memory-efficient operations on datasets of any size.
When to Use This Skill
USE Polars when:
Large datasets - Working with data too large for pandas (10GB+)Performance critical - Need maximum speed for data transformationsMemory constrained - Limited RAM requires efficient memory usageParallel processing - Want to utilize all CPU cores automaticallyComplex aggregations - Group by, window functions, rolling calculationsLazy evaluation - Query optimization before execution mattersETL pipelines - Building production data pipelinesStreaming data - Processing data larger than memory
DON'T USE Polars when:
Pandas ecosystem required - Need specific pandas-only librariesSmall datasets - Under 100MB where pandas is sufficientLegacy code - Extensive existing pandas codebaseMatplotlib/Seaborn direct integration - These work better with pandasTime series with specialized needs - Some pandas time series features are more mature
Prerequisites
pip install polars
pip install 'polars[all]'
pip install 'polars[numpy,pandas,pyarrow,fsspec,connectorx,xlsx2csv,deltalake,timezone]'
uv pip install polars pyarrow connectorx
Core Capabilities
1. DataFrame Creation and I/O
Creating DataFrames:
import polars as pl
import numpy as np
from datetime import datetime, date
df = pl.DataFrame({
"id" : [1
2
3
4
5
"name"
"Alice"
"Bob"
"Charlie"
"Diana"
"Eve"
"value"
100.5
200.3
150.7
300.2
250.8
"category"
"A"
"B"
"A"
"C"
"B"
"timestamp"
2025
1
1
10
0
2025
1
2
11
30
2025
1
3
9
15
2025
1
4
14
45
2025
1
5
16
0
print
print
f"Shape: {df.shape} "
print
f"Schema: {df.schema} "
1000
5
"col_a"
"col_b"
"col_c"
"col_d"
"col_e"
"x"
1
"y"
"a"
"x"
2
"y"
"b"
"x"
3
"y"
"c"
"integers"
1
2
3
"floats"
1.0
2.0
3.0
"strings"
"a"
"b"
"c"
"integers"
"floats"
"strings"
df = pl.read_csv("data.csv" )
df = pl.read_csv(
"data.csv" ,
separator="," ,
has_header=True ,
skip_rows=0 ,
n_rows=10000 ,
columns=["col1" , "col2" , "col3" ],
dtypes={"id" : pl.Int64, "value" : pl.Float32},
null_values=["NA" , "N/A" , "" ],
ignore_errors=True ,
try_parse_dates=True ,
encoding="utf8"
)
df = pl.read_parquet("data.parquet" )
df = pl.read_parquet("data/*.parquet" )
df = pl.read_parquet(
"large_data.parquet" ,
columns=["id" , "value" , "date" ],
n_rows=100000 ,
row_count_name="row_nr"
)
df = pl.read_json("data.json" )
df = pl.read_ndjson("data.jsonl" )
df = pl.read_excel("data.xlsx" , sheet_name="Sheet1" )
df = pl.read_delta("delta_table/" )
df = pl.read_database(
query="SELECT * FROM sales WHERE date > '2025-01-01'" ,
connection="postgresql://user:pass@localhost/db"
)
df = pl.read_csv("https://example.com/data.csv" )
df.write_csv("output.csv" )
df.write_parquet(
"output.parquet" ,
compression="zstd" ,
compression_level=3 ,
statistics=True ,
row_group_size=100000
)
df.write_json("output.json" , row_oriented=True )
df.write_ndjson("output.jsonl" )
df.write_delta("delta_table/" , mode="overwrite" )
df.write_ipc("output.arrow" )
2. Lazy Evaluation and Query Optimization import polars as pl
lf = pl.scan_csv("large_data.csv" )
df = pl.DataFrame({"x" : [1 , 2 , 3 ]})
lf = df.lazy()
result_lf = (
lf
.filter (pl.col("date" ) >= "2025-01-01" )
.with_columns([
(pl.col("revenue" ) - pl.col("cost" )).alias("profit" ),
pl.col("category" ).cast(pl.Categorical)
])
.group_by("category" )
.agg([
pl.col("profit" ).sum ().alias("total_profit" ),
pl.col("profit" ).mean().alias("avg_profit" ),
pl.count().alias("count" )
])
.sort("total_profit" , descending=True )
)
print (result_lf.explain())
result_df = result_lf.collect()
result_df = result_lf.collect(streaming=True )
sample = result_lf.fetch(1000 )
Query Optimization Benefits:
lf = (
pl.scan_parquet("data/*.parquet" )
.filter (pl.col("country" ) == "USA" )
.select(["id" , "name" , "revenue" ])
.filter (pl.col("revenue" ) > 1000 )
)
print ("Naive plan:" )
print (lf.explain(optimized=False ))
print ("\nOptimized plan:" )
print (lf.explain(optimized=True ))
def process_large_file (input_path: str , output_path: str ):
"""Process file that doesn't fit in memory."""
result = (
pl.scan_csv(input_path)
.filter (pl.col("status" ) == "active" )
.group_by("region" )
.agg([
pl.col("sales" ).sum (),
pl.col("customers" ).n_unique()
])
.collect(streaming=True )
)
result.write_parquet(output_path)
return result
(
pl.scan_csv("huge_file.csv" )
.filter (pl.col("value" ) > 0 )
.sink_parquet("filtered_output.parquet" )
)
3. Expression API import polars as pl
df = pl.DataFrame({
"a" : [1 , 2 , 3 , 4 , 5 ],
"b" : [10 , 20 , 30 , 40 , 50 ],
"c" : ["x" , "y" , "x" , "y" , "x" ],
"d" : [1.5 , 2.5 , 3.5 , 4.5 , 5.5 ]
})
df.select(pl.col("a" ))
df.select(pl.col("a" , "b" , "c" ))
df.select(pl.col("^a.*$" ))
df.select(pl.all ())
df.select(pl.exclude("c" ))
df.select([
pl.col("a" ),
(pl.col("a" ) + pl.col("b" )).alias("sum" ),
(pl.col("a" ) * pl.col("d" )).alias("product" ),
(pl.col("b" ) / pl.col("a" )).alias("ratio" ),
(pl.col("a" ) ** 2 ).alias("squared" ),
(pl.col("a" ) % 2 ).alias("modulo" )
])
df.select([
pl.col("a" ),
pl.when(pl.col("a" ) > 3 )
.then(pl.lit("high" ))
.otherwise(pl.lit("low" ))
.alias("category" ),
pl.when(pl.col("a" ) < 2 )
.then(pl.lit("low" ))
.when(pl.col("a" ) < 4 )
.then(pl.lit("medium" ))
.otherwise(pl.lit("high" ))
.alias("tier" )
])
df_str = pl.DataFrame({
"text" : ["Hello World" , "Polars is Fast" , "Data Analysis" ]
})
df_str.select([
pl.col("text" ),
pl.col("text" ).str .to_lowercase().alias("lower" ),
pl.col("text" ).str .to_uppercase().alias("upper" ),
pl.col("text" ).str .len_chars().alias("length" ),
pl.col("text" ).str .split(" " ).alias("words" ),
pl.col("text" ).str .contains("a" ).alias("has_a" ),
pl.col("text" ).str .replace("a" , "X" ).alias("replaced" )
])
df_list = pl.DataFrame({
"values" : [[1 , 2 , 3 ], [4 , 5 ], [6 , 7 , 8 , 9 ]]
})
df_list.select([
pl.col("values" ),
pl.col("values" ).list .len ().alias("count" ),
pl.col("values" ).list .sum ().alias("sum" ),
pl.col("values" ).list .mean().alias("mean" ),
pl.col("values" ).list .first().alias("first" ),
pl.col("values" ).list .last().alias("last" ),
pl.col("values" ).list .get(0 ).alias("index_0" ),
pl.col("values" ).list .contains(5 ).alias("has_5" )
])
df_struct = pl.DataFrame({
"point" : [{"x" : 1 , "y" : 2 }, {"x" : 3 , "y" : 4 }]
})
df_struct.select([
pl.col("point" ),
pl.col("point" ).struct.field("x" ).alias("x" ),
pl.col("point" ).struct.field("y" ).alias("y" )
])
df_dt = pl.DataFrame({
"timestamp" : pl.date_range(
datetime(2025 , 1 , 1 ),
datetime(2025 , 12 , 31 ),
"1d" ,
eager=True
)
})
df_dt.select([
pl.col("timestamp" ),
pl.col("timestamp" ).dt.year().alias("year" ),
pl.col("timestamp" ).dt.month().alias("month" ),
pl.col("timestamp" ).dt.day().alias("day" ),
pl.col("timestamp" ).dt.weekday().alias("weekday" ),
pl.col("timestamp" ).dt.week().alias("week" ),
pl.col("timestamp" ).dt.quarter().alias("quarter" ),
pl.col("timestamp" ).dt.strftime("%Y-%m-%d" ).alias("formatted" )
])
4. GroupBy and Aggregations import polars as pl
df = pl.DataFrame({
"category" : ["A" , "B" , "A" , "B" , "A" , "C" ],
"subcategory" : ["x" , "y" , "x" , "y" , "z" , "x" ],
"value" : [100 , 200 , 150 , 250 , 175 , 300 ],
"quantity" : [10 , 20 , 15 , 25 , 12 , 30 ]
})
result = df.group_by("category" ).agg([
pl.col("value" ).sum ().alias("total_value" ),
pl.col("value" ).mean().alias("avg_value" ),
pl.col("value" ).min ().alias("min_value" ),
pl.col("value" ).max ().alias("max_value" ),
pl.col("value" ).std().alias("std_value" ),
pl.col("quantity" ).sum ().alias("total_quantity" ),
pl.count().alias("count" )
])
print (result)
result = df.group_by(["category" , "subcategory" ]).agg([
pl.col("value" ).sum (),
pl.count()
])
result = df.group_by("category" , maintain_order=True ).agg(
pl.col("value" ).sum ()
)
agg_exprs = [
pl.col(c).mean().alias(f"{c} _mean" )
for c in ["value" , "quantity" ]
]
result = df.group_by("category" ).agg(agg_exprs)
result = df.group_by("category" ).agg([
pl.col("value" ).sum ().alias("sum" ),
pl.col("value" ).mean().alias("mean" ),
pl.col("value" ).median().alias("median" ),
pl.col("value" ).quantile(0.25 ).alias("q25" ),
pl.col("value" ).quantile(0.75 ).alias("q75" ),
pl.col("value" ).var().alias("variance" ),
pl.col("value" ).skew().alias("skewness" )
])
result = df.group_by("category" ).agg([
pl.col("value" ).filter (pl.col("quantity" ) > 15 ).sum ().alias("high_qty_value" ),
pl.col("value" ).filter (pl.col("quantity" ) <= 15 ).sum ().alias("low_qty_value" )
])
result = df.group_by("category" ).agg([
pl.col("value" ).first().alias("first_value" ),
pl.col("value" ).last().alias("last_value" ),
pl.col("value" ).head(3 ).alias("top_3" ),
pl.col("value" ).tail(2 ).alias("bottom_2" )
])
result = df.group_by("category" ).agg([
pl.col("subcategory" ).n_unique().alias("unique_subcats" ),
pl.col("subcategory" ).unique().alias("subcategories" )
])
result = df.group_by("category" ).agg([
pl.col("value" ).map_elements(
lambda s: s.to_numpy().std(ddof=1 ),
return_dtype=pl.Float64
).alias("custom_std" )
])
5. Window Functions import polars as pl
df = pl.DataFrame({
"date" : pl.date_range(date(2025 , 1 , 1 ), date(2025 , 1 , 10 ), eager=True ),
"category" : ["A" , "B" ] * 5 ,
"value" : [100 , 110 , 105 , 115 , 108 , 120 , 112 , 125 , 118 , 130 ]
})
df.with_columns([
pl.col("value" ).rank().over("category" ).alias("rank" ),
pl.col("value" ).rank(descending=True ).over("category" ).alias("rank_desc" )
])
df.with_columns([
pl.col("value" ).cum_sum().over("category" ).alias("cumsum" ),
pl.col("value" ).cum_max().over("category" ).alias("cummax" ),
pl.col("value" ).cum_min().over("category" ).alias("cummin" ),
pl.col("value" ).cum_count().over("category" ).alias("cumcount" )
])
df.with_columns([
pl.col("value" ).shift(1 ).over("category" ).alias("lag_1" ),
pl.col("value" ).shift(-1 ).over("category" ).alias("lead_1" ),
pl.col("value" ).shift(2 ).over("category" ).alias("lag_2" ),
(pl.col("value" ) - pl.col("value" ).shift(1 ).over("category" )).alias("diff" )
])
df.with_columns([
pl.col("value" ).pct_change().over("category" ).alias("pct_change" )
])
df.with_columns([
pl.col("value" ).rolling_mean(window_size=3 ).over("category" ).alias("rolling_mean_3" ),
pl.col("value" ).rolling_sum(window_size=3 ).over("category" ).alias("rolling_sum_3" ),
pl.col("value" ).rolling_std(window_size=3 ).over("category" ).alias("rolling_std_3" ),
pl.col("value" ).rolling_min(window_size=3 ).over("category" ).alias("rolling_min_3" ),
pl.col("value" ).rolling_max(window_size=3 ).over("category" ).alias("rolling_max_3" )
])
df_ts = pl.DataFrame({
"timestamp" : pl.datetime_range(
datetime(2025 , 1 , 1 ),
datetime(2025 , 1 , 10 ),
"1h" ,
eager=True
),
"value" : range (217 )
})
df_ts.with_columns([
pl.col("value" ).rolling_mean_by(
by="timestamp" ,
window_size="6h"
).alias("rolling_mean_6h" ),
pl.col("value" ).rolling_sum_by(
by="timestamp" ,
window_size="1d"
).alias("rolling_sum_1d" )
])
df.with_columns([
pl.col("value" ).ewm_mean(span=3 ).alias("ewm_mean" ),
pl.col("value" ).ewm_std(span=3 ).alias("ewm_std" )
])
6. Joins and Concatenation import polars as pl
orders = pl.DataFrame({
"order_id" : [1 , 2 , 3 , 4 , 5 ],
"customer_id" : [101 , 102 , 101 , 103 , 104 ],
"amount" : [250.0 , 150.0 , 300.0 , 200.0 , 175.0 ]
})
customers = pl.DataFrame({
"customer_id" : [101 , 102 , 103 , 105 ],
"name" : ["Alice" , "Bob" , "Charlie" , "Diana" ],
"region" : ["East" , "West" , "East" , "North" ]
})
result = orders.join(customers, on="customer_id" , how="inner" )
result = orders.join(customers, on="customer_id" , how="left" )
result = orders.join(customers, on="customer_id" , how="right" )
result = orders.join(customers, on="customer_id" , how="full" )
result = orders.join(customers, how="cross" )
result = orders.join(customers, on="customer_id" , how="semi" )
result = orders.join(customers, on="customer_id" , how="anti" )
df1 = pl.DataFrame({"a" : [1 , 2 ], "b" : ["x" , "y" ], "val1" : [10 , 20 ]})
df2 = pl.DataFrame({"a" : [1 , 2 ], "b" : ["x" , "z" ], "val2" : [100 , 200 ]})
result = df1.join(df2, on=["a" , "b" ], how="inner" )
result = orders.join(
customers.rename({"customer_id" : "cust_id" }),
left_on="customer_id" ,
right_on="cust_id"
)
df1 = pl.DataFrame({"id" : [1 , 2 ], "value" : [10 , 20 ]})
df2 = pl.DataFrame({"id" : [1 , 2 ], "value" : [100 , 200 ]})
result = df1.join(df2, on="id" , suffix="_right" )
df1 = pl.DataFrame({"a" : [1 , 2 ], "b" : [3 , 4 ]})
df2 = pl.DataFrame({"a" : [5 , 6 ], "b" : [7 , 8 ]})
df3 = pl.DataFrame({"a" : [9 , 10 ], "b" : [11 , 12 ]})
combined = pl.concat([df1, df2, df3])
df1 = pl.DataFrame({"a" : [1 , 2 , 3 ]})
df2 = pl.DataFrame({"b" : [4 , 5 , 6 ]})
combined = pl.concat([df1, df2], how="horizontal" )
df1 = pl.DataFrame({"a" : [1 , 2 ], "b" : [3 , 4 ]})
df2 = pl.DataFrame({"b" : [5 , 6 ], "c" : [7 , 8 ]})
combined = pl.concat([df1, df2], how="diagonal" )
df1 = pl.DataFrame({"a" : [1 , 2 ], "b" : [3 , 4 ]})
df2 = pl.DataFrame({"a" : [5 , 6 ], "c" : [7 , 8 ]})
combined = pl.concat([df1, df2], how="diagonal_relaxed" )
trades = pl.DataFrame({
"time" : pl.datetime_range(datetime(2025 , 1 , 1 , 9 , 0 ), datetime(2025 , 1 , 1 , 9 , 10 ), "1m" , eager=True ),
"price" : [100.0 , 101.0 , 100.5 , 102.0 , 101.5 , 103.0 , 102.5 , 104.0 , 103.5 , 105.0 , 104.5 ]
})
quotes = pl.DataFrame({
"time" : pl.datetime_range(datetime(2025 , 1 , 1 , 9 , 0 ), datetime(2025 , 1 , 1 , 9 , 10 ), "2m" , eager=True ),
"bid" : [99.5 , 100.5 , 101.5 , 102.5 , 103.5 , 104.5 ]
})
result = trades.join_asof(
quotes,
on="time" ,
strategy="backward"
)
Complete Examples
Example 1: ETL Pipeline for Sales Data import polars as pl
from pathlib import Path
from datetime import datetime
def etl_sales_pipeline (
input_dir: Path,
output_dir: Path,
min_date: str = "2025-01-01"
) -> dict :
"""
Complete ETL pipeline for sales data processing.
Demonstrates: lazy evaluation, joins, aggregations, window functions.
"""
output_dir.mkdir(parents=True , exist_ok=True )
sales = pl.scan_parquet(input_dir / "sales/*.parquet" )
products = pl.scan_parquet(input_dir / "products.parquet" )
customers = pl.scan_parquet(input_dir / "customers.parquet" )
sales_clean = (
sales
.filter (
(pl.col("date" ) >= min_date) &
(pl.col("quantity" ) > 0 ) &
(pl.col("unit_price" ) > 0 )
)
.with_columns([
(pl.col("quantity" ) * pl.col("unit_price" )).alias("revenue" ),
pl.col("date" ).str .strptime(pl.Date, "%Y-%m-%d" ).alias("date_parsed" )
])
.with_columns([
pl.col("date_parsed" ).dt.year().alias("year" ),
pl.col("date_parsed" ).dt.month().alias("month" ),
pl.col("date_parsed" ).dt.weekday().alias("weekday" )
])
)
enriched = (
sales_clean
.join(products, on="product_id" , how="left" )
.join(customers, on="customer_id" , how="left" )
)
with_metrics = (
enriched
.with_columns([
pl.col("revenue" )
.cum_sum()
.over(["customer_id" ])
.alias("customer_cumulative_revenue" ),
pl.col("revenue" )
.rank(descending=True )
.over(["category" , "month" ])
.alias("category_month_rank" ),
pl.col("revenue" )
.rolling_mean(window_size=7 )
.over(["product_id" ])
.alias("rolling_7d_avg" )
])
)
daily_summary = (
with_metrics
.group_by(["date_parsed" , "category" ])
.agg([
pl.col("revenue" ).sum ().alias("total_revenue" ),
pl.col("quantity" ).sum ().alias("total_quantity" ),
pl.col("order_id" ).n_unique().alias("order_count" ),
pl.col("customer_id" ).n_unique().alias("unique_customers" ),
pl.col("revenue" ).mean().alias("avg_order_value" )
])
.sort(["date_parsed" , "category" ])
)
monthly_summary = (
with_metrics
.group_by(["year" , "month" , "category" ])
.agg([
pl.col("revenue" ).sum ().alias("total_revenue" ),
pl.col("revenue" ).mean().alias("avg_revenue" ),
pl.col("revenue" ).std().alias("std_revenue" ),
pl.count().alias("transaction_count" )
])
.sort(["year" , "month" , "category" ])
)
customer_summary = (
with_metrics
.group_by("customer_id" )
.agg([
pl.col("revenue" ).sum ().alias("lifetime_value" ),
pl.col("order_id" ).n_unique().alias("order_count" ),
pl.col("date_parsed" ).min ().alias("first_purchase" ),
pl.col("date_parsed" ).max ().alias("last_purchase" ),
pl.col("category" ).n_unique().alias("categories_purchased" )
])
.with_columns([
(pl.col("last_purchase" ) - pl.col("first_purchase" ))
.dt.total_days()
.alias("customer_tenure_days" ),
(pl.col("lifetime_value" ) / pl.col("order_count" ))
.alias("avg_order_value" )
])
.sort("lifetime_value" , descending=True )
)
results = {
"daily" : daily_summary.collect(),
"monthly" : monthly_summary.collect(),
"customer" : customer_summary.collect(),
"detailed" : with_metrics.collect(streaming=True )
}
results["daily" ].write_parquet(output_dir / "daily_summary.parquet" )
results["monthly" ].write_parquet(output_dir / "monthly_summary.parquet" )
results["customer" ].write_parquet(output_dir / "customer_summary.parquet" )
results["detailed" ].write_parquet(
output_dir / "detailed_sales.parquet" ,
compression="zstd"
)
return {
"total_revenue" : results["daily" ]["total_revenue" ].sum (),
"total_orders" : results["daily" ]["order_count" ].sum (),
"unique_customers" : results["customer" ].height,
"date_range" : f"{results['daily' ]['date_parsed' ].min ()} to {results['daily' ]['date_parsed' ].max ()} "
}
summary = etl_sales_pipeline(
input_dir=Path("data/raw" ),
output_dir=Path("data/processed" ),
min_date="2025-01-01"
)
print (f"Pipeline complete: {summary} " )
Example 2: Time Series Analysis import polars as pl
import numpy as np
from datetime import datetime, timedelta
def analyze_time_series (
df: pl.DataFrame,
value_column: str ,
time_column: str ,
group_column: str = None
) -> dict :
"""
Comprehensive time series analysis with Polars.
Returns statistics, trends, seasonality indicators.
"""
df = df.with_columns([
pl.col(time_column).cast(pl.Datetime).alias(time_column)
])
df = df.with_columns([
pl.col(time_column).dt.year().alias("year" ),
pl.col(time_column).dt.month().alias("month" ),
pl.col(time_column).dt.day().alias("day" ),
pl.col(time_column).dt.weekday().alias("weekday" ),
pl.col(time_column).dt.hour().alias("hour" ),
pl.col(time_column).dt.week().alias("week_of_year" )
])
stats = df.select([
pl.col(value_column).mean().alias("mean" ),
pl.col(value_column).std().alias("std" ),
pl.col(value_column).min ().alias("min" ),
pl.col(value_column).max ().alias("max" ),
pl.col(value_column).median().alias("median" ),
pl.col(value_column).quantile(0.25 ).alias("q25" ),
pl.col(value_column).quantile(0.75 ).alias("q75" ),
pl.col(value_column).skew().alias("skewness" ),
pl.col(value_column).kurtosis().alias("kurtosis" )
]).to_dicts()[0 ]
df_trend = df.with_columns([
pl.col(value_column).rolling_mean(window_size=7 ).alias("ma_7" ),
pl.col(value_column).rolling_mean(window_size=30 ).alias("ma_30" ),
pl.col(value_column).rolling_mean(window_size=90 ).alias("ma_90" ),
pl.col(value_column).ewm_mean(span=7 ).alias("ema_7" ),
pl.col(value_column).ewm_mean(span=30 ).alias("ema_30" ),
pl.col(value_column).pct_change().alias("pct_change_1" ),
pl.col(value_column).pct_change(n=7 ).alias("pct_change_7" ),
(pl.col(value_column) - pl.col(value_column).shift(1 )).alias("diff_1" ),
((pl.col(value_column) - pl.col(value_column).mean()) /
pl.col(value_column).std()).alias("z_score" )
])
monthly_pattern = (
df
.group_by("month" )
.agg([
pl.col(value_column).mean().alias("avg" ),
pl.col(value_column).std().alias("std" ),
pl.count().alias("count" )
])
.sort("month" )
)
weekday_pattern = (
df
.group_by("weekday" )
.agg([
pl.col(value_column).mean().alias("avg" ),
pl.col(value_column).std().alias("std" ),
pl.count().alias("count" )
])
.sort("weekday" )
)
hourly_pattern = (
df
.group_by("hour" )
.agg([
pl.col(value_column).mean().alias("avg" ),
pl.col(value_column).std().alias("std" ),
pl.count().alias("count" )
])
.sort("hour" )
)
mean_val = df[value_column].mean()
std_val = df[value_column].std()
anomalies = df.filter (
(pl.col(value_column) > mean_val + 3 * std_val) |
(pl.col(value_column) < mean_val - 3 * std_val)
)
return {
"statistics" : stats,
"trend_data" : df_trend,
"monthly_pattern" : monthly_pattern,
"weekday_pattern" : weekday_pattern,
"hourly_pattern" : hourly_pattern,
"anomalies" : anomalies,
"anomaly_count" : anomalies.height
}
np.random.seed(42 )
n_points = 10000
base = np.sin(np.linspace(0 , 20 * np.pi, n_points)) * 100
trend = np.linspace(0 , 50 , n_points)
noise = np.random.randn(n_points) * 10
sample_df = pl.DataFrame({
"timestamp" : pl.datetime_range(
datetime(2024 , 1 , 1 ),
datetime(2024 , 1 , 1 ) + timedelta(hours=n_points),
"1h" ,
eager=True
)[:n_points],
"value" : base + trend + noise
})
results = analyze_time_series(
sample_df,
value_column="value" ,
time_column="timestamp"
)
print ("Statistics:" , results["statistics" ])
print (f"Anomalies detected: {results['anomaly_count' ]} " )
print ("\nMonthly pattern:" )
print (results["monthly_pattern" ])
Example 3: Large-Scale Data Processing with Streaming import polars as pl
from pathlib import Path
import time
def process_large_dataset (
input_pattern: str ,
output_path: str ,
chunk_report_every: int = 1_000_000
) -> dict :
"""
Process very large datasets using streaming.
This approach handles datasets larger than available RAM.
"""
start_time = time.time()
lf = pl.scan_parquet(input_pattern)
processed = (
lf
.filter (
(pl.col("status" ) == "completed" ) &
(pl.col("amount" ) > 0 )
)
.with_columns([
(pl.col("amount" ) * pl.col("quantity" )).alias("total" ),
pl.col("timestamp" ).str .strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S" ),
pl.col("category" ).cast(pl.Categorical)
])
.with_columns([
pl.col("timestamp" ).dt.date().alias("date" ),
pl.col("timestamp" ).dt.hour().alias("hour" )
])
.group_by(["date" , "category" ])
.agg([
pl.col("total" ).sum ().alias("daily_total" ),
pl.col("total" ).mean().alias("daily_avg" ),
pl.col("total" ).count().alias("transaction_count" ),
pl.col("user_id" ).n_unique().alias("unique_users" )
])
.sort(["date" , "category" ])
)
print ("Query plan:" )
print (processed.explain())
result = processed.collect(streaming=True )
result.write_parquet(
output_path,
compression="zstd" ,
compression_level=3
)
elapsed = time.time() - start_time
return {
"rows_output" : result.height,
"columns" : result.columns,
"elapsed_seconds" : elapsed,
"output_path" : output_path
}
def sink_large_dataset (input_pattern: str , output_path: str ):
"""
Process and write directly to file without collecting in memory.
"""
(
pl.scan_parquet(input_pattern)
.filter (pl.col("status" ) == "completed" )
.with_columns([
(pl.col("amount" ) * pl.col("quantity" )).alias("total" )
])
.group_by("category" )
.agg([
pl.col("total" ).sum (),
pl.count()
])
.sink_parquet(output_path)
)
print (f"Results written to {output_path} " )
Integration Examples
Polars with Plotly Visualization import polars as pl
import plotly.express as px
import plotly.graph_objects as go
def create_dashboard_data (df: pl.DataFrame ) -> dict :
"""Prepare data for Plotly dashboard."""
daily_trend = (
df
.group_by("date" )
.agg([
pl.col("revenue" ).sum ().alias("revenue" ),
pl.col("orders" ).sum ().alias("orders" )
])
.sort("date" )
.to_pandas()
)
category_breakdown = (
df
.group_by("category" )
.agg(pl.col("revenue" ).sum ())
.sort("revenue" , descending=True )
.to_pandas()
)
regional = (
df
.group_by("region" )
.agg([
pl.col("revenue" ).sum (),
pl.col("orders" ).count()
])
.to_pandas()
)
return {
"daily_trend" : daily_trend,
"category_breakdown" : category_breakdown,
"regional" : regional
}
def plot_time_series (df_pandas, x_col, y_col, title ):
"""Create interactive time series plot."""
fig = px.line(
df_pandas,
x=x_col,
y=y_col,
title=title
)
fig.update_layout(hovermode="x unified" )
return fig
Polars with Pandas Interop import polars as pl
import pandas as pd
polars_df = pl.DataFrame({"a" : [1 , 2 , 3 ], "b" : [4 , 5 , 6 ]})
pandas_df = polars_df.to_pandas()
pandas_df = pd.DataFrame({"x" : [1 , 2 , 3 ], "y" : ["a" , "b" , "c" ]})
polars_df = pl.from_pandas(pandas_df)
polars_df = pl.from_pandas(pandas_df, nan_to_null=True )
def hybrid_pipeline (input_path: str ):
"""Use Polars for processing, Pandas for visualization libraries."""
processed = (
pl.scan_parquet(input_path)
.filter (pl.col("value" ) > 0 )
.group_by("category" )
.agg([
pl.col("value" ).sum (),
pl.col("value" ).mean().alias("avg_value" )
])
.collect()
)
import seaborn as sns
pandas_df = processed.to_pandas()
sns.barplot(data=pandas_df, x="category" , y="value" )
return processed
Best Practices
1. Use Lazy Evaluation by Default
lf = pl.scan_parquet("data.parquet" )
result = (
lf
.filter (pl.col("x" ) > 0 )
.select(["x" , "y" ])
.collect()
)
df = pl.read_parquet("data.parquet" )
df = df.filter (pl.col("x" ) > 0 )
df = df.select(["x" , "y" ])
2. Chain Operations
result = (
df
.filter (pl.col("status" ) == "active" )
.with_columns([
(pl.col("a" ) + pl.col("b" )).alias("sum" ),
pl.col("date" ).dt.year().alias("year" )
])
.group_by("year" )
.agg(pl.col("sum" ).mean())
)
df = df.filter (pl.col("status" ) == "active" )
df = df.with_columns((pl.col("a" ) + pl.col("b" )).alias("sum" ))
df = df.with_columns(pl.col("date" ).dt.year().alias("year" ))
result = df.group_by("year" ).agg(pl.col("sum" ).mean())
3. Use Appropriate Data Types
df = df.with_columns([
pl.col("small_int" ).cast(pl.Int16),
pl.col("category" ).cast(pl.Categorical),
pl.col("flag" ).cast(pl.Boolean),
pl.col("precise_float" ).cast(pl.Float32)
])
print (df.estimated_size("mb" ))
4. Filter Early
result = (
pl.scan_parquet("data.parquet" )
.filter (pl.col("date" ) >= "2025-01-01" )
.group_by("category" )
.agg(pl.col("value" ).sum ())
.collect()
)
result = (
pl.scan_parquet("data.parquet" )
.group_by("category" )
.agg(pl.col("value" ).sum ())
.filter (...)
.collect()
)
5. Use Expressions Over Apply
df.with_columns([
pl.when(pl.col("x" ) > 0 ).then(pl.col("x" )).otherwise(0 ).alias("positive_x" )
])
df.with_columns([
pl.col("x" ).map_elements(lambda v: v if v > 0 else 0 ).alias("positive_x" )
])
Troubleshooting
Common Issues
result = lf.collect(streaming=True )
lf.sink_parquet("output.parquet" )
for chunk in pl.read_csv_batched("large.csv" , batch_size=100000 ):
process(chunk)
print (lf.explain(optimized=True ))
result = lf.profile()
print (result[1 ])
Issue: Type Mismatch in Join
df1 = df1.with_columns(pl.col("id" ).cast(pl.Int64))
df2 = df2.with_columns(pl.col("id" ).cast(pl.Int64))
result = df1.join(df2, on="id" )
Issue: Date Parsing Errors
df = df.with_columns([
pl.col("date_str" ).str .strptime(pl.Date, "%Y-%m-%d" ),
pl.col("datetime_str" ).str .strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S" )
])
Version History
1.0.0 (2026-01-17): Initial release with comprehensive Polars coverage
Core DataFrame operations
Lazy evaluation patterns
Expression API reference
GroupBy and window functions
Join operations
ETL pipeline examples
Time series analysis
Streaming for large datasets
Integration examples
Best practices and troubleshooting
Resources
Use Polars for maximum performance on large datasets with intuitive, expressive data transformations!
