Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Advanced sub-skill for pandas focused on memory optimization, execution speed, and handling large-scale datasets (10M+ rows). Covers low-level dtypes, efficient indexing, and vectorization of complex logic.
version
2.2
license
BSD-3-Clause
pandas - Performance & Memory Management
Standard pandas code is often memory-hungry and slow. This sub-skill provides the techniques to make pandas 10x faster and use 5x less RAM by understanding its internal architecture (BlockManager and Arrow backend).
When to Use
Your DataFrame is larger than 1GB and causes RAM pressure.
pd.read_csv is taking too long to load data.
Row-wise operations (apply, iterrows) are creating bottlenecks.
You need to perform complex joins or lookups on millions of rows.
Pandas usually creates copies of data during operations. To handle large data, you must minimize copies and use the most efficient bit-width for your data types.
If you need to map values from a dictionary/other table millions of times:
# ❌ SLOW: df.merge() or df['id'].map(large_dict)# ✅ FAST: Use a Series as a lookup table with index
lookup_table = pd.Series(data=values, index=keys)
result = lookup_table.reindex(df['target_ids']).values
Efficient I/O
1. Parquet with Filtering (Predicate Pushdown)
Never use CSV for large data storage. Use Parquet.
# Save as partitioned parquet
df.to_parquet('data_dir', partition_cols=['year', 'month'])
# Load only specific columns and rows (Fast)
df_subset = pd.read_parquet('data_dir', columns=['price', 'id'],
filters=[('year', '==', 2023)])
2. Chunking for Memory-Limited Systems
If the file is 50GB and you have 16GB RAM:
# Process in chunks of 100k rows
chunk_size = 100_000for chunk in pd.read_csv("massive.csv", chunksize=chunk_size):
# Perform aggregation
summary = chunk.groupby('id')['value'].sum()
# Save or update a running total
Critical Rules for Performance
✅ DO
Use In-place operations sparingly - Contrary to myth, inplace=True often creates internal copies anyway. Focus on dtypes instead.
Sort Index for Slicing - If you slice a large DataFrame by index, ensure it is sorted: df.sort_index(inplace=True). This turns an O(N) operation into O(log N).
Use pd.to_datetime with format - Specifying the format (%Y-%m-%d) is much faster than automatic parsing.
Leverage .eval() - For complex arithmetic like (A + B) / (C * D), df.eval() is faster and more memory-efficient as it uses numexpr.
❌ DON'T
Never iterate with iterrows() - It converts each row into a Series object, which is incredibly slow.
Avoid object dtypes - Any column with object dtype (usually strings) is a pointer to a Python object, which is memory-intensive. Use category or string[pyarrow].
Don't use append() in a loop - It creates a full copy of the DataFrame every time. Collect data in a list and use pd.concat().
Anti-Patterns (NEVER)
# ❌ BAD: Growing a DataFrame row by row
df = pd.DataFrame()
for data in large_source:
df = pd.concat([df, pd.DataFrame([data])]) # ❌ Disaster for performance!# ✅ GOOD: List of dicts to DataFrame
data_list = []
for data in large_source:
data_list.append(data)
df = pd.DataFrame(data_list)
# ❌ BAD: Manual string formatting# df['name'].apply(lambda x: f"USER_{x}")# ✅ GOOD: Vectorized string accessor
df['name'] = "USER_" + df['name'].astype(str)
Practical Workflows
1. Identifying Memory Hogs
# Get detailed memory breakdown (including object overhead)print(df.memory_usage(deep=True))
# Identify columns with too many unique strings (bad for 'category')for col in df.select_dtypes(include=['object']):
print(f"{col}: {df[col].nunique() / len(df):.2%}")
2. Fast Deduplication of 10M+ Rows
# Using sorting + shift is often faster than drop_duplicates
df = df.sort_values(['id', 'timestamp'])
mask = (df['id'] != df['id'].shift())
df_unique = df[mask]
3. Merging with Multi-Index
# If you join on multiple columns, setting them as an index # and using join() can be 5x faster than merge()
df1.set_index(['key1', 'key2'], inplace=True)
df2.set_index(['key1', 'key2'], inplace=True)
result = df1.join(df2, how='inner')
This sub-skill turns pandas from a prototyping tool into a high-performance engine capable of handling industrial-scale scientific data.
