data-analysis

name	data-analysis
description	Expert guidance for data analysis, data science, and machine learning projects. Covers Python data tools, SQL/databases, visualization, statistics, ML/AI, data engineering, and MLOps. Use when working on data analysis tasks, choosing data tools, building data pipelines, creating visualizations, performing statistical analysis, or building ML models.

Data Analysis

Expert guidance for data analysis and data science workflows. Provides tool selection, workflow patterns, and curated resources across the full data stack.

Tool Selection Quick Reference

Data Manipulation

Need	Recommended Tool	Alternative
General tabular data	Pandas	Polars (faster, multithreaded)
Large datasets (out-of-core)	Polars, Dask	Vaex, Modin
GPU-accelerated DataFrames	cuDF (RAPIDS)	CuPy (NumPy on GPU)
Parallel pandas operations	Pandarallel	Modin
Cross-engine portability	Fugue (Pandas/Spark/Dask)	-
Fuzzy string matching	TheFuzz	-
Date/time handling	Pendulum, Arrow	DateUtil

Automated EDA

Need	Recommended Tool
One-line auto visualization	AutoViz
EDA with dataset comparison	Sweetviz
Data quality profiling	YData Profiling
Missing data patterns	Missingno
Interactive GUI analysis	D-Tale, PyGWalker
Conversational analysis (LLM)	PandasAI

Data Quality & Validation

Need	Recommended Tool
Outlier/anomaly detection	PyOD, Alibi Detect
Schema validation (DataFrames)	Pandera
Schema validation (general)	Pydantic, Cerberus
Data pipeline testing	Great Expectations

Feature Engineering

Need	Recommended Tool
Automated feature engineering	FeatureTools
Scikit-learn compatible FE	Feature Engine
Dimensionality reduction (PCA/MCA)	Prince
Categorical encoding	Category Encoders
Imbalanced datasets	Imbalanced Learn
Distribution fitting	Fitter

Visualization

Need	Recommended Tool	Alternative
Static plots	Matplotlib	Plotnine (ggplot2-style)
Statistical visualization	Seaborn	-
Interactive plots	Plotly	Bokeh, Altair
Geographic/maps	Folium, GeoPandas	QGIS, OSMnx
Large dataset rendering	Datashader	Deck.gl
3D/scientific	VisPy	Glumpy
No-code visualization	Flourish	-
Chart selection guide	From Data to Viz	Data Viz Catalogue

Dashboards & BI

Need	Recommended Tool
Quick data apps	Streamlit
Production dashboards	Dash (Plotly)
ML demo interfaces	Gradio
Notebook to web app	Voila
Full-stack Python web	Reflex, Taipy
Enterprise BI	Tableau, Power BI
Open-source BI	Apache Superset, Metabase
Monitoring dashboards	Grafana

SQL & Databases

Need	Recommended Tool
In-process analytics	DuckDB
Python ORM	SQLAlchemy
PostgreSQL adapter	Psycopg2
SQL linting/formatting	SQLFluff
SQL transpilation	SQLGlot
Natural language to SQL	Vanna.AI
Time-series database	TimescaleDB, TDengine
Database GUI	DBeaver, Beekeeper Studio

Statistics & Probability

Need	Recommended Tool
General scientific computing	SciPy
Statistical modeling	Statsmodels
Bayesian modeling	PyMC, NumPyro
User-friendly stats	Pingouin
Survival analysis	Lifelines, scikit-survival
Causal inference	DoWhy, CausalImpact
Bayesian visualization	ArviZ
GAMs	PyGAM

Time Series

Need	Recommended Tool
General forecasting	Facebook Prophet
Bayesian forecasting	Uber Orbit
ML time series framework	sktime
Deep learning forecasting	PyTorch Forecasting, GluonTS
Zero-shot forecasting	TimesFM (Google)
Feature extraction	TSFresh
ARIMA modeling	pmdarima

Machine Learning

Need	Recommended Tool
Classical ML	Scikit-learn
Gradient boosting	XGBoost, LightGBM, CatBoost
AutoML	H2O
GPU-accelerated ML	cuML (RAPIDS)
Model explainability	SHAP, InterpretML
Hyperparameter tuning	Optuna

Deep Learning

Need	Recommended Tool
Research & production	PyTorch
High-level API	Keras
Fast prototyping	Fast.ai
Transformers/NLP	HuggingFace Transformers
Object detection	Ultralytics (YOLOv8)
Model interoperability	ONNX
Efficient fine-tuning	PEFT, Unsloth
Graph neural networks	PyTorch Geometric

Data Engineering

Need	Recommended Tool
SQL transformations	dbt-core
Workflow orchestration	Apache Airflow, Dagster, Prefect
Distributed processing	Apache Spark
Event streaming	Apache Kafka
Table format (lakehouse)	Apache Iceberg, Delta Lake
Distributed SQL queries	Trino
Data lineage	OpenLineage, DataHub
Reproducible pipelines	Kedro

MLOps

Need	Recommended Tool
Experiment tracking	MLflow, Wandb
Data/model versioning	DVC
Model/data drift monitoring	Evidently
Model serving	BentoML, KServe
LLM inference (high-throughput)	vLLM
Unified LLM API	LiteLLM
ML on Kubernetes	Kubeflow
Feature store	Feast

Web Scraping

Need	Recommended Tool
Simple HTTP requests	Requests, HTTPX
HTML parsing	BeautifulSoup
Browser automation	Playwright, Selenium
Full crawling framework	Scrapy
AI-powered scraping	ScrapeGraph AI, Crawl4AI
Auto-detect patterns	AutoScraper
Text extraction from web	Trafilatura

NLP

Need	Recommended Tool
Production NLP pipeline	SpaCy
Text preprocessing	NLTK, TextBlob
Sentence embeddings	SentenceTransformers
Topic modeling	Gensim
Transformer models	HuggingFace Transformers
Conversational AI	Rasa
Adversarial NLP testing	TextAttack

Data Analysis Workflow

1. Data Acquisition

• Files: OpenPyXL (Excel), PyPDF2/Camelot (PDF), CleverCSV (messy CSV)
• Web: Requests/HTTPX + BeautifulSoup, or Scrapy/Crawl4AI for large-scale
• Databases: SQLAlchemy + DuckDB for analytics, Psycopg2 for PostgreSQL
• APIs: HTTPX (async), Requests-cache (with caching)
• Synthetic: Faker, Mimesis for test data generation

2. Exploration & Profiling

• Run YData Profiling or Sweetviz for automated EDA
• Use Missingno to visualize missing data patterns
• Use D-Tale or PyGWalker for interactive exploration
• For quick auto-visualization: AutoViz

3. Cleaning & Validation

• Pandera for DataFrame schema validation
• Great Expectations for data pipeline testing
• PyOD for outlier detection
• Pandas DQ for automatic type correction and cleaning

4. Feature Engineering

• FeatureTools for automated feature engineering
• Category Encoders for categorical variables
• Prince for dimensionality reduction (PCA/MCA)
• Imbalanced Learn for handling class imbalance

5. Analysis & Modeling

• Statistical: SciPy, Statsmodels, Pingouin
• Classical ML: Scikit-learn, XGBoost/LightGBM/CatBoost
• Deep Learning: PyTorch + relevant framework
• Time Series: Prophet, sktime, pmdarima
• NLP: SpaCy, HuggingFace Transformers
• Hyperparameter tuning: Optuna

6. Visualization & Reporting

• Matplotlib/Seaborn for static publication plots
• Plotly/Altair for interactive visualizations
• Streamlit/Dash for data apps and dashboards
• Gradio for ML model demos

7. Deployment & Monitoring

• MLflow/Wandb for experiment tracking
• DVC for data versioning
• BentoML/KServe for model serving
• Evidently for drift monitoring

Reference Files

Read these for detailed tool lists and learning resources:

File	Contents	Read when...
references/python-ecosystem.md	Python tools: data processing, EDA, validation, feature engineering, specialized tools	Working with Python data libraries
references/sql-databases.md	SQL resources, database tools, drivers, learning materials	Working with SQL or databases
references/visualization-dashboards.md	Visualization tools, dashboard frameworks, BI software	Building charts, dashboards, or data apps
references/statistics-ml.md	Statistics, ML, deep learning, NLP, time series, AI tools	Doing statistical analysis or building models
references/data-engineering.md	Data pipelines, MLOps, cloud infrastructure, web scraping	Building data pipelines or deploying models
references/learning-resources.md	Courses, datasets, cheatsheets, interview prep, career	Looking for learning materials or datasets

File Processing Quick Reference

Format	Library
Excel (.xlsx)	OpenPyXL, Xlwings
CSV (messy)	CleverCSV
PDF (text)	PyPDF2, PyMuPDF
PDF (tables)	Camelot
Word (.docx)	Python-docx
HTML to Markdown	Python-markdownify, MarkItDown
XML	Xmltodict
JSON querying	JmesPath, jq (CLI)
YAML	yq (CLI)
Tabular export	Tablib (XLSX/JSON/CSV)

Command-Line Data Tools

Tool	Purpose
jq	JSON processor
yq	YAML/XML processor
q	Run SQL on CSV/TSV files
VisiData	Interactive tabular data explorer
csvkit	CSV manipulation suite
Miller	Multi-format data processor (CSV/JSON/etc.)
DuckDB CLI	SQL analytics on files
hyperfine	Benchmarking
termgraph	Terminal-based graphs