Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that do not fit in memory.
Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that do not fit in memory.
license
MIT license
metadata
{"skill-author":"K-Dense Inc."}
Vaex
Overview
Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.
When to Use This Skill
Use Vaex when:
Processing tabular datasets larger than available RAM (gigabytes to terabytes)
Performing fast statistical aggregations on massive datasets
Creating visualizations and heatmaps of large datasets
Building machine learning pipelines on big data
Converting between data formats (CSV, HDF5, Arrow, Parquet)
Needing lazy evaluation and virtual columns to avoid memory overhead
Working with astronomical data, financial time series, or other large-scale scientific datasets
Core Capabilities
Vaex provides six primary capability areas, each documented in detail in the references directory:
1. DataFrames and Data Loading
Load and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference references/core_dataframes.md for:
Opening large files efficiently
Converting from pandas/NumPy/Arrow
Working with example datasets
Understanding DataFrame structure
2. Data Processing and Manipulation
Perform filtering, create virtual columns, use expressions, and aggregate data without loading everything into memory. Reference references/data_processing.md for:
Create interactive visualizations of large datasets including heatmaps, histograms, and scatter plots. Reference references/visualization.md for:
Creating 1D and 2D plots
Heatmap visualizations
Working with selections
Customizing plots and subplots
5. Machine Learning Integration
Build ML pipelines with transformers, encoders, and integration with scikit-learn, XGBoost, and other frameworks. Reference references/machine_learning.md for:
Feature scaling and encoding
PCA and dimensionality reduction
K-means clustering
Integration with scikit-learn/XGBoost/CatBoost
Model serialization and deployment
6. I/O Operations
Efficiently read and write data in various formats with optimal performance. Reference references/io_operations.md for:
File format recommendations
Export strategies
Working with Apache Arrow
CSV handling for large files
Server and remote data access
Quick Start Pattern
For most Vaex tasks, follow this pattern:
import vaex
# 1. Open or create DataFrame
df = vaex.open('large_file.hdf5') # or .csv, .arrow, .parquet# OR
df = vaex.from_pandas(pandas_df)
# 2. Explore the dataprint(df) # Shows first/last rows and column info
df.describe() # Statistical summary# 3. Create virtual columns (no memory overhead)
df['new_column'] = df.x ** 2 + df.y
# 4. Filter with selections
df_filtered = df[df.age > 25]
# 5. Compute statistics (fast, lazy evaluation)
mean_val = df.x.mean()
stats = df.groupby('category').agg({'value': 'sum'})
# 6. Visualize
df.plot1d(df.x, limits=[0, 100])
df.plot(df.x, df.y, limits='99.7%')
# 7. Export if needed
df.export_hdf5('output.hdf5')
Working with References
The reference files contain detailed information about each capability area. Load references into context based on the specific task:
Basic operations: Start with references/core_dataframes.md and references/data_processing.md
Visualization tasks: Use references/visualization.md
ML pipelines: Reference references/machine_learning.md
File I/O: Consult references/io_operations.md
Best Practices
Use HDF5 or Apache Arrow formats for optimal performance with large datasets
Leverage virtual columns instead of materializing data to save memory
Batch operations using delay=True when performing multiple calculations
Export to efficient formats rather than keeping data in CSV
Use expressions for complex calculations without intermediate storage
Profile with df.stat() to understand memory usage and optimize operations
Common Patterns
Pattern: Converting Large CSV to HDF5
import vaex
# Open large CSV (processes in chunks automatically)
df = vaex.from_csv('large_file.csv')
# Export to HDF5 for faster future access
df.export_hdf5('large_file.hdf5')
# Future loads are instant
df = vaex.open('large_file.hdf5')
Pattern: Efficient Aggregations
# Use delay=True to batch multiple operations
mean_x = df.x.mean(delay=True)
std_y = df.y.std(delay=True)
sum_z = df.z.sum(delay=True)
# Execute all at once
results = vaex.execute([mean_x, std_y, sum_z])
Pattern: Virtual Columns for Feature Engineering
# No memory overhead - computed on the fly
df['age_squared'] = df.age ** 2
df['full_name'] = df.first_name + ' ' + df.last_name
df['is_adult'] = df.age >= 18
Resources
This skill includes reference documentation in the references/ directory:
core_dataframes.md - DataFrame creation, loading, and basic structure
data_processing.md - Filtering, expressions, aggregations, and transformations
performance.md - Optimization strategies and lazy evaluation
visualization.md - Plotting and interactive visualizations
machine_learning.md - ML pipelines and model integration
io_operations.md - File formats and data import/export
