Out-of-core DataFrame for billion-row data via lazy evaluation and memory-mapped files. Use when data exceeds RAM (10 GB–TB) for fast aggregation, filtering, virtual columns, and visualization without loading. Supports HDF5, Arrow, Parquet, CSV with cloud (S3, GCS, Azure). Built-in ML transformers (scaling, PCA, K-means). In-memory: polars; distributed: dask.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Out-of-core DataFrame for billion-row data via lazy evaluation and memory-mapped files. Use when data exceeds RAM (10 GB–TB) for fast aggregation, filtering, virtual columns, and visualization without loading. Supports HDF5, Arrow, Parquet, CSV with cloud (S3, GCS, Azure). Built-in ML transformers (scaling, PCA, K-means). In-memory: polars; distributed: dask.
license
MIT
Vaex DataFrames
Overview
Vaex is a high-performance Python library for lazy, out-of-core DataFrame operations on datasets too large to fit in RAM. It processes over a billion rows per second using memory-mapped files and lazy evaluation, enabling interactive exploration and analysis without loading data into memory.
When to Use
Processing tabular datasets larger than available RAM (10 GB to terabytes)
Fast statistical aggregations on massive datasets (mean, std, quantiles at billion-row scale)
Creating visualizations (heatmaps, histograms) of large datasets without sampling
Building ML preprocessing pipelines (scaling, encoding, PCA) on big data
Converting between data formats (CSV to HDF5/Arrow for fast repeated access)
Feature engineering with virtual columns that consume zero additional memory
Working with astronomical catalogs, financial time series, or large scientific datasets
For in-memory speed on data that fits in RAM, use polars instead
For distributed multi-node computing, use dask instead
vaex.ml provides transformers for preprocessing, dimensionality reduction, clustering, and scikit-learn model wrapping. All transformers create virtual columns (zero memory overhead).
Vaex operations build an expression graph without executing computation. Evaluation is triggered only when a result is accessed (printing a value, calling .values, exporting).
Rule of thumb: Keep columns virtual unless the same complex expression is used in 3+ aggregations.
Memory-Mapped File Architecture
HDF5 and Apache Arrow files are memory-mapped: the OS maps file pages to virtual memory on demand, so opening a 100 GB file is instant and uses minimal RAM. Data pages are read from disk only when accessed.
# Opens instantly regardless of file size
df = vaex.open('100gb_dataset.hdf5') # ~0.001s, minimal RAM
mean = df.column.mean() # Streams through data, ~RSS stays low
Format Comparison
Feature
HDF5
Arrow/Feather
Parquet
CSV
Load speed
Instant
Instant
Fast
Slow
Memory-mapped
Yes
Yes
No
No
Compression
Optional (gzip, lzf, blosc)
No
Default (snappy, gzip, brotli)
No
Columnar
Yes
Yes
Yes
No
Portability
Good
Excellent
Excellent
Excellent
Best for
Local Vaex workflows
Cross-language interop
Distributed systems
Data exchange
Recommendation: Convert CSV to HDF5 once (vaex.from_csv('data.csv', convert='data.hdf5')), then use HDF5 for all future loads.
import vaex, vaex.ml
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
np.random.seed(42)
n = 50_000
df = vaex.from_arrays(
age=np.random.randint(18, 70, n).astype(float),
income=np.random.uniform(20000, 150000, n),
region=np.random.choice(['East', 'West', 'Central'], n),
target=np.random.randint(0, 2, n),
)
train, test = df[:40_000], df[40_000:]
# Preprocessing pipeline
train = vaex.ml.LabelEncoder(features=['region']).fit_transform(train)
train = vaex.ml.StandardScaler(features=['age', 'income']).fit_transform(train)
# Train model
features = ['standard_scaled_age', 'standard_scaled_income', 'label_encoded_region']
model = vaex.ml.sklearn.Predictor(
features=features, target='target',
model=GradientBoostingClassifier(n_estimators=50, random_state=42),
prediction_name='prediction',
)
model.fit(train)
# Save pipeline state (encoding + scaling + model in one file)
train.state_write('ml_pipeline.json')
# Deploy: apply saved state to new data
test.state_load('ml_pipeline.json')
accuracy = (test.prediction == test.target).mean()
print(f"Test accuracy: {accuracy:.3f}") # ~0.50 (random data)# Production: prod_df = vaex.open('new_batch.hdf5'); prod_df.state_load('ml_pipeline.json')
Key Parameters
Parameter
Module
Default
Range/Options
Effect
shape
plot, plot1d
64 (1D), (256,256) (2D)
32-2048
Histogram bin count / heatmap resolution
limits
plot, plot1d
'minmax'
'99%', '99.7%', [min,max]
Axis ranges; percentile-based for outlier handling
f
plot
'identity'
'log', 'log10', 'sqrt'
Color scale transform for density plots
delay
aggregations
False
True/False
Batch multiple aggregations into single pass
convert
from_csv
None
file path string
Auto-convert CSV to HDF5 during load
chunk_size
from_csv
5,000,000
100K-50M
Rows per chunk for CSV processing
n_components
PCA
2
1-n_features
Number of principal components
n_clusters
KMeans
8
2-100+
Number of clusters
features
all ML transformers
required
list of column names
Columns to transform
compression
export_hdf5
None
'gzip', 'lzf', 'blosc'
Trade file size for I/O speed
Best Practices
Always convert CSV to HDF5 or Arrow for repeated access. One-time conversion pays for itself on the first reload: vaex.from_csv('data.csv', convert='data.hdf5').
Keep columns virtual until you must materialize. Virtual columns have zero memory cost. Materialize only when a complex expression is reused in 3+ aggregations.
Batch aggregations with delay=True. Each separate aggregation call scans the entire dataset. Batching with vaex.execute([df.x.mean(delay=True), df.x.std(delay=True)]) reduces N passes to 1.
Use selections instead of creating filtered DataFrames when computing statistics on multiple subsets. df.select(df.age > 30, name='senior') then df.salary.mean(selection='senior') is more efficient than creating df_senior = df[df.age > 30].
Avoid .values and .to_pandas_df() on large data. These load data into RAM, defeating Vaex's purpose. Use only on small subsets or samples.
Save pipeline state for reproducibility.df.state_write('state.json') captures virtual columns, selections, and ML transformers for deployment.
Anti-pattern -- row iteration. Never iterate rows in Vaex. Use vectorized expressions and aggregations instead.
