data-analysis

name	data-analysis
description	Comprehensive data analysis agent skill for loading, cleaning, exploring, visualizing, and reporting on structured datasets. Supports CSV, JSON, Excel, and SQL data sources. Produces statistical summaries, correlation matrices, time series analysis, regression models, hypothesis tests, and publication-quality visualizations.
version	1.0.0
author	Skill Foundry
platforms	["claude-code","codex","cursor","gemini-cli","openclaw","copilot"]
tags	["data-analysis","visualization","statistics","pandas","matplotlib","seaborn","plotly","reporting","csv","json","excel","sql"]

Data Analysis Agent Skill

Overview

A production-grade agent skill for end-to-end data analysis workflows. Use this skill when you need to load datasets, inspect structure, clean messy data, perform statistical analyses, create visualizations, or generate structured reports from tabular data.

When to Trigger

Activate this skill when the user asks to:

• Analyze data: "Analyze this CSV", "What patterns do you see in sales.csv?", "Explore this dataset", "Summarize the data in users.json"
• Create charts/graphs: "Make a bar chart of…", "Plot revenue over time", "Visualize the correlation matrix", "Show me a heatmap of…"
• Find patterns: "Find trends in this data", "Is there a correlation between X and Y?", "Cluster customers from this data", "Detect anomalies in…"
• Generate reports: "Create a report from survey_results.xlsx", "Summarize quarterly metrics", "Build a dashboard from sales data"
• Clean data: "Clean this messy dataset", "Fix missing values in…", "Normalize these columns", "Deduplicate this CSV"
• Statistical testing: "Run a t-test on group A vs B", "Check if this distribution is normal", "Perform regression analysis", "Calculate confidence intervals"

Near-Miss Negatives — Do NOT Trigger

• Questions about database schema design without actual data (e.g., "What columns should my users table have?")
• Questions about spreadsheet software UI (e.g., "How do I freeze a row in Google Sheets?")
• General math / statistics theory questions without a dataset context (e.g., "Explain the central limit theorem")
• Pure SQL query writing without a data-analysis intent (e.g., "Write a query to join three tables" — use a SQL skill instead)
• Questions about ETL pipeline architecture or data engineering (e.g., "Design a data ingestion pipeline")

Step-by-Step Workflow

Follow these phases in order. Skip phases that don't apply (e.g., if data is already clean) but always state that you're skipping and why.

Phase 1: Load

Determine the data source and load it into a DataFrame.

CSV    → pd.read_csv(filepath, ...)
JSON   → pd.read_json(filepath, ...)
Excel  → pd.read_excel(filepath, sheet_name=...)
SQL    → pd.read_sql_query(query, connection)

Checklist:

• Identify encoding (try UTF-8, Latin-1, detect automatically)
• For CSV: inspect delimiter (comma, tab, semicolon), quote character
• For Excel: list available sheets, load the right one
• For JSON: handle nested structures with pd.json_normalize() if needed
• For SQL: confirm read-only access, never run destructive queries
• Load a sample first if the dataset is large (>100k rows)

Phase 2: Inspect

Understand what you're working with before touching anything.

df.shape          # rows × columns
df.info()         # dtypes, non-null counts, memory
df.head(10)       # first rows
df.tail(5)        # last rows
df.describe()     # numeric summary stats
df.describe(include='object')  # categorical summary
df.dtypes         # column types
df.columns.tolist()  # column names

Checklist:

• Report shape: rows × columns
• List all columns with their dtypes
• Show summary statistics for numeric columns
• Show value counts for low-cardinality categorical columns
• Flag potential issues: wrong dtypes, placeholder values, suspicious zeros

Phase 3: Clean

Address data quality issues. Never modify the source file without prompting the user first. Work on a copy.

Common operations:

Issue	Approach
Missing values	df.isnull().sum() → decide drop vs impute
Wrong dtypes	pd.to_numeric(), pd.to_datetime(), astype()
Outliers	IQR method, Z-score, domain-specific thresholds
Duplicates	df.duplicated().sum() → df.drop_duplicates()
Inconsistent strings	.str.strip(), .str.lower(), .str.replace()
Date parsing	pd.to_datetime() with format or infer
Normalization	Min-max scaling, Z-score standardization
Categorical encoding	One-hot, label encoding for ML prep

Rules:

1. Always work on df_clean = df.copy(), never mutate the original in-place without explicit user consent.
2. Report every change: "Dropped 47 duplicate rows (2.3% of data)", "Imputed missing age values with median (142 cells)".
3. Flag suspicious patterns even if you don't fix them: "Column 'salary' has 340 zero values — verify if these are legitimate."
4. If a cleaning decision is irreversible, ask first.

Phase 4: Analyze

Apply appropriate analytical methods based on the question.

Exploratory Data Analysis (EDA):

• Univariate: histograms, box plots, value counts per column
• Bivariate: scatter plots, correlation coefficients, grouped means
• Multivariate: pair plots, correlation matrix heatmap, PCA

Statistical Methods (see references/statistical-methods.md):

Goal	Method
Compare two groups	Independent t-test, Mann-Whitney U
Compare 3+ groups	One-way ANOVA, Kruskal-Wallis
Relationship between two continuous vars	Pearson/Spearman correlation
Predict continuous outcome	Linear regression, polynomial regression
Predict categorical outcome	Logistic regression
Check normality	Shapiro-Wilk test, Q-Q plot
Detect time trends	Moving averages, decomposition, stationarity tests
Find clusters	K-means, hierarchical clustering, DBSCAN
Reduce dimensions	PCA, t-SNE (visualization only)

Time Series specifics:

• Set datetime index: df.set_index('date', inplace=True)
• Resample: df.resample('M').mean()
• Rolling windows: df['value'].rolling(7).mean()
• Decomposition: trend, seasonal, residual

Phase 5: Visualize

Choose the right chart for the data and question. See references/visualization-patterns.md for the full guide.

Library selection:

• Static, publication-quality: matplotlib + seaborn
• Interactive, exploratory: plotly
• Statistical plots: seaborn (box, violin, pair, joint, heatmap)

Quick reference:

Data Type	Question	Chart
Categorical × Numeric	Compare amounts	Bar chart, box plot
Numeric × Numeric	Relationship	Scatter plot, line chart
Time × Numeric	Trend over time	Line chart, area chart
Categorical × Categorical	Cross-tabulation	Heatmap, stacked bar
Distribution	Shape of data	Histogram, KDE, violin
Part-to-whole	Proportions	Pie chart* (≤5 categories), treemap
Correlation matrix	Relationships	Heatmap
Rankings	Order	Horizontal bar chart

*Pie charts: use only when ≤5 categories and values sum to a meaningful whole. Prefer bar charts otherwise.

Best practices:

• Always label axes and add a title
• Use accessible color palettes (avoid red-green for colorblind users)
• Sort bar charts by value unless categories have a natural order
• Add data source and date to chart footnotes
• For interactive charts, include hover tooltips

Phase 6: Report

Synthesize findings into a structured report.

Report structure:

1. Executive Summary — 2–3 sentences with the key finding
2. Data Overview — source, shape, date range, columns
3. Data Quality — issues found, actions taken
4. Key Findings — bullet points with numbers, ranked by importance
5. Visualizations — inline charts with captions
6. Statistical Results — test statistics, p-values, effect sizes
7. Limitations & Caveats — data gaps, assumptions, edge cases
8. Recommendations — actionable next steps or further analysis

Output formats:

• Quick answer: plain text summary in chat with key numbers
• Detailed report: Markdown document with embedded charts
• Dashboard: interactive HTML with plotly (offer if >5 charts)
• Export: offer to save cleaned data and charts as files

Tool-Aware Implementation

Python Libraries

This skill assumes Python 3.9+ with the following libraries available. Check availability before use; install missing packages as needed.

# Core
import pandas as pd
import numpy as np

# Visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Interactive
import plotly.express as px
import plotly.graph_objects as go

# Statistics
from scipy import stats
from scipy.stats import norm, ttest_ind, f_oneway, pearsonr, spearmanr

# Optional: machine learning
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression, LogisticRegression

Matplotlib setup for non-interactive environments:

import matplotlib
matplotlib.use('Agg')  # headless rendering

Plotly in notebooks vs scripts:

# In Jupyter/notebook environments:
import plotly.io as pio
pio.renderers.default = 'notebook'

# For saving to files:
fig.write_html('chart.html')
fig.write_image('chart.png')

Platform-Specific Notes

Platform	Matplotlib backend	File output	Notes
Claude Code	Agg	write to files	Save charts as PNG/HTML, display from disk
Codex	Agg	write to files	Same approach
Cursor	Agg or interactive	write to files	Can open HTML in preview
Gemini CLI	Agg	write to files	Save charts, display paths
OpenClaw	Agg	write to files	Use canvas for HTML output
Copilot	Agg	write to files	Standard file-based approach

Safety & Guardrails

1. Never modify source data in-place — always create a copy or backup before transformations. Offer to save cleaned data as a new file.
2. Flag data quality issues — don't silently fix problems. Report missing values, outliers, and type inconsistencies before and after cleaning.
3. Statistical honesty — report p-values and effect sizes, not just "significant" or "not significant". Don't p-hack by running multiple tests without correction. Mention when sample sizes are too small for reliable inference.
4. Privacy awareness — if a dataset appears to contain PII (emails, phone numbers, names), warn the user and suggest anonymization before analysis.
5. Large file handling — for files >100MB, use chunked reading (chunksize parameter) or sample before full analysis. Warn about memory constraints.
6. SQL safety — use read-only connections. Never run INSERT, UPDATE, DELETE, DROP, or ALTER. Use transactions or connection strings that enforce read-only mode.
7. Deterministic results — set random seeds for reproducible analysis: np.random.seed(42).

Scripts

scripts/validate_dataset.py

PEP 723 compliant data quality validation script. Run with:

python scripts/validate_dataset.py path/to/dataset.csv
# or
python scripts/validate_dataset.py path/to/dataset.json
# or
python scripts/validate_dataset.py path/to/dataset.xlsx --sheet "Sheet1"

Produces a structured quality report covering missing values, outliers, type consistency, duplicates, and basic statistics. See script docstring for details.

References

• data-cleaning-guide.md — Handling nulls, outliers, type coercion, deduplication, and normalization patterns.
• visualization-patterns.md — Chart type selection guide, color best practices, accessibility considerations.
• statistical-methods.md — Descriptive statistics, hypothesis testing, regression, correlation, confidence intervals, and when to use each method.