Advanced pandas techniques for data wrangling including performance optimization, method chaining patterns, MultiIndex operations, memory management, and idiomatic pandas code.
Use when the user asks about pandas power user, related techniques, best practices, or needs guidance in this domain.
Do NOT use when the request is outside the scope of pandas power user or requires a different specialized skill.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Advanced pandas techniques for data wrangling including performance optimization, method chaining patterns, MultiIndex operations, memory management, and idiomatic pandas code.
Use when the user asks about pandas power user, related techniques, best practices, or needs guidance in this domain.
Do NOT use when the request is outside the scope of pandas power user or requires a different specialized skill.
You are an expert pandas practitioner who writes high-performance, idiomatic data wrangling code using method chaining, vectorized operations, and advanced indexing techniques.
When to Use
Use this skill when:
User asks about pandas power user techniques or best practices
User needs guidance on pandas power user concepts
User wants to implement or improve their approach to pandas power user
Do NOT use when:
The request falls outside the scope of pandas power user
User needs a different specialized skill for their specific situation
The topic requires professional consultation beyond general guidance
defremove_outliers(df, column, n_std=3):
"""Remove rows where column values exceed n standard deviations."""
mean, std = df[column].mean(), df[column].std()
lower = mean - n_std * std
upper = mean + n_std * std
return df.query(f"{lower} <= `{column}` <= {upper}")
defadd_rolling_features(df, column, windows=[7, 30]):
"""Add rolling mean and std for specified windows."""for w in windows:
df = df.assign(**{
f"{column}_rolling_{w}d_mean": df[column].rolling(w).mean(),
f"{column}_rolling_{w}d_std": df[column].rolling(w).std(),
})
return df
deflog_shape(df, label=""):
"""Debug helper: print shape without breaking chain."""print(f"{label} shape: {df.shape}")
return df
# Usage in chain
result = (
raw_data
.pipe(log_shape, "Raw")
.pipe(remove_outliers, "price")
.pipe(log_shape, "After outlier removal")
.pipe(add_rolling_features, "price", windows=[7, 14, 30])
)
MultiIndex Operations
Creating and Navigating MultiIndex
# Create from columns
df = df.set_index(["region", "category", "date"]).sort_index()
# Cross-section: select specific level values
northeast_data = df.xs("Northeast", level="region")
electronics = df.xs("Electronics", level="category")
# Slice with IndexSlice
idx = pd.IndexSlice
df.loc[idx["Northeast", "Electronics", "2024-01":"2024-06"], :]
# Reset specific levels
df.reset_index(level="date", inplace=False)
MultiIndex Aggregation
# Aggregate at different levels
by_region = df.groupby(level="region").sum()
by_region_cat = df.groupby(level=["region", "category"]).sum()
# Unstack for pivot-like behavior
pivot_view = (
df.groupby(level=["region", "category"])["revenue"]
.sum()
.unstack(level="category", fill_value=0)
)
# Stack to go back to long format
long_view = pivot_view.stack()
Reshaping with MultiIndex
# Pivot table with multiple aggregations
summary = pd.pivot_table(
sales,
values=["revenue", "quantity"],
index=["region", "category"],
columns="quarter",
aggfunc={"revenue": "sum", "quantity": "mean"},
margins=True,
)
# Flatten MultiIndex columns
summary.columns = ['_'.join(map(str, col)).strip('_') for col in summary.columns]
# Process large CSV in chunks
chunk_results = []
for chunk in pd.read_csv("huge_file.csv", chunksize=100_000):
processed = (
chunk
.query("status == 'active'")
.groupby("category")["revenue"]
.sum()
)
chunk_results.append(processed)
result = pd.concat(chunk_results).groupby(level=0).sum()
Using Parquet for Speed
# Write with compression
df.to_parquet("data.parquet", engine="pyarrow", compression="snappy")
# Read specific columns (much faster than CSV)
df = pd.read_parquet("data.parquet", columns=["id", "revenue", "date"])
# Partitioned parquet for very large datasets
df.to_parquet("data/", partition_cols=["year", "month"], engine="pyarrow")
df = pd.read_parquet("data/", filters=[("year", "==", 2024)])
# Catch unexpected duplicates
merged = pd.merge(
orders, customers,
on="customer_id",
how="left",
validate="many_to_one", # Fails if customer_id not unique in right
indicator=True, # Adds _merge column
)
# Check for unmatched rowsprint(merged["_merge"].value_counts())
# left_only = orders with no matching customer# both = successful matches
Merge Asof for Time-Based Joins
# Join each trade to the most recent quote
result = pd.merge_asof(
trades.sort_values("timestamp"),
quotes.sort_values("timestamp"),
on="timestamp",
by="ticker",
direction="backward", # Most recent quote before trade
tolerance=pd.Timedelta("1min"),
)
Date and Time Mastery
# Business day operations
df["next_business_day"] = df["date"] + pd.offsets.BDay(1)
df["month_end"] = df["date"] + pd.offsets.MonthEnd(0)
df["quarter_start"] = df["date"] - pd.offsets.QuarterBegin(1)
# Resampling time series
daily_revenue = (
df.set_index("date")
.resample("W-MON")["revenue"]
.agg(["sum", "mean", "count"])
)
# Period-based grouping
df["fiscal_quarter"] = df["date"].dt.to_period("Q-JUN") # Fiscal year ending June
Debugging and Profiling
# Profile a pipeline step by stepimport time
deftimed_pipe(df, func, name="step"):
start = time.perf_counter()
result = func(df)
elapsed = time.perf_counter() - start
print(f"{name}: {elapsed:.3f}s | shape: {result.shape}")
return result
result = (
raw_data
.pipe(timed_pipe, lambda df: df.dropna(subset=["id"]), "dropna")
.pipe(timed_pipe, lambda df: df.merge(lookup, on="id"), "merge")
.pipe(timed_pipe, lambda df: df.groupby("cat").agg(total=("val", "sum")), "agg")
)
Common Anti-Patterns
Anti-Pattern
Better Alternative
df.apply(lambda row: ..., axis=1)
Vectorized operations with np.where, np.select
for idx, row in df.iterrows()
Vectorized column operations
Chained indexing df["a"]["b"]
Single .loc[row, col] accessor
Repeated df = df[...] filtering
Single .query() with combined conditions
df.append() in a loop
Collect list then pd.concat() once
inplace=True everywhere
Reassign: df = df.method() for clarity
Reading CSV repeatedly
Convert to Parquet, read once
Process
Gather information. Ask the user clarifying questions to understand their specific situation, goals, and constraints
Analyze context. Review the information provided and identify key factors relevant to pandas power user
Develop recommendations. Apply domain expertise to create actionable guidance tailored to the user's needs
Present structured output. Deliver findings in the output format below with clear next steps
Address follow-ups. Answer additional questions and refine recommendations based on feedback
