| name | data-analysis |
| description | Analyze structured data and produce evidence backed reports with Python. Use when the user asks to analyze CSV, Excel, SQL results, KPIs, business metrics, or other tabular data, including requests like "look at this data", "what trends do you see", "make a chart", "summarize these numbers", or "write a report from this spreadsheet". Also trigger for research synthesis from multiple sources and for database table analysis involving systems like ODPS, BigQuery, or MySQL. This skill covers data requirements, sufficiency checks, Python analysis, charts, and a polished Markdown report with limits and sources. |
Data Analysis
Overview
This skill turns raw materials — tables, metrics, research notes, search results — into professional, evidence-backed reports. It follows a deliberate workflow: define what's needed, verify what's actually available, analyze with Python, and produce a structured Markdown report.
The goal is credible analysis, not maximal confidence. When evidence is incomplete, say what's known, what's uncertain, and what would resolve it.
Default to the user's conversation language. Default to Python for any structured data work.
Core principle: verify before you act
Never generate SQL, analysis code, or field references based on assumptions. If you haven't seen the actual table schema, column names, data types, or sample rows, you don't know what's there — and guessing leads to broken queries and wasted effort.
The right sequence is always: learn what exists → plan based on reality → execute.
Workflow
Follow these steps in order. Skip to a later step only if the user explicitly asks for a rough first pass.
1. Define the analytic task
Clarify the question, audience, time range, granularity, key dimensions, and desired output. If underspecified, make minimum reasonable assumptions and state them.
2. Understand available data
This step has two goals: (a) figure out what data is needed to answer the question credibly, and (b) figure out what data is actually available.
When data files are already provided (CSV, Excel, etc.):
- Read the file first. Check columns, types, row count, sample values, missing data.
- Only then decide if it's sufficient for the analysis.
When the user mentions database tables (ODPS, BigQuery, MySQL, etc.):
- Do NOT generate analysis SQL yet. You don't know the real schema.
- If a database query tool/skill is available, use it under the safe-query contract below. If no query tool is available, ask the user to provide the schema or a sample.
- Safe-query contract: read-only
SELECT/metadata inspection only; no DDL/DML; always include partition/time filters when the table is partitioned or time-based; always include LIMIT for previews; estimate or cap returned rows before extraction; avoid full-table scans.
- Inspect safely: get available partitions/date range, partition maturity, row counts by partition, real fields/types/comments, and a small filtered sample.
- Only after you've seen the real fields, types, and sample data should you proceed to plan the analysis.
When no data is provided yet:
- Define what data would be needed: metrics, dimensions, time range, grain, mandatory fields, optional fields.
- Return a concrete Data Requirements section the user can act on.
Proactively identify what data would strengthen the analysis: Based on the analytic task, think about what data is needed — including data the user may not have thought of. For example, if the task is retention analysis and the user only mentioned an events table, you should recognize that acquisition channel data would enable cohort-by-channel breakdown, or that a user profile table would allow segmentation by user attributes. Then ask the user whether that data is available: "To do a stronger analysis, I'd also want [X] because [reason] — can you get that?" This turns a basic analysis into a high-value one.
3. Check sufficiency and trustworthiness
For structured data: check row/column scope, field meanings, duplicates, missing values, outliers, time coverage, metric definition mismatches, and sample size.
Also check data freshness and grain before trusting any number:
- Partition maturity: verify the partitions you need are fully populated (check date range and per-partition row counts). For T+N outcome metrics, confirm the outcome window has actually matured; a newly created or recently-populated table bounds your usable window.
- Row grain: confirm what one row represents. A row that looks atomic may be pre-aggregated (a count/size column is a hint) — mistaking "entity × day" for "one action" conflates distinct concepts.
- Missing-value rule: decide and state whether missing rows are dropped or counted as zero — this alone can shift results by points; quantify the impact once.
For research materials: check source identity, quality, recency, cross-source consistency, unsupported assertions, and missing primary evidence.
If materials are too weak, say so and narrow claim strength. Read references/source-trustworthiness.md when source quality matters. Read references/common-pitfalls.md before drawing conclusions about effects, comparisons, or trends.
4. Choose the analysis mode
Structured Data Analysis — for CSV, Excel, SQL results, KPI tables, tabular exports. Use Python. Focus on descriptive statistics, comparisons, trends, distributions, anomalies, segmentation, and candidate explanations with evidence.
Research Material Analysis — for articles, notes, search summaries, multi-source briefings. Use Python when it adds value (summary tables, timelines, charts). Focus on source grouping, consensus/disagreement, evidence strength, key themes, and decision-relevant takeaways.
Database-Assisted Analysis — when data lives in a database and a query tool is available. Use the available database tool/skill under the safe-query contract; if unavailable, ask for schema/sample data. The workflow is:
- Use the appropriate query tool under the safe-query contract above to inspect schemas, partitions, row counts, and sample data
- Based on verified schema, define extraction requirements: metrics, dimensions, filters, time range, grain, partition filters, expected row count, and row cap
- Generate or request SQL only after the contract is satisfied; review it for read-only operations, partition/time filters, row limits, and grain before execution
- Receive results, then proceed as Structured Data Analysis
If multiple query rounds are needed (e.g., exploratory queries reveal the need for joins with other tables), iterate: explore → learn → refine → execute.
5. Python environment (automatic)
The Python environment (pandas, matplotlib, seaborn) is checked and set up automatically via a PreToolUse hook — no manual action needed. When the first Bash command runs, the hook will:
- Confirm the environment is ready, or
- Auto-install missing packages into a local
.venv and report the path, or
- Warn if installation failed
Read the additionalContext injected by the hook to know the environment status. If a .venv was created, use the PYTHON_EXEC path from the context for subsequent analysis code.
Read references/python-analysis-conventions.md before writing analysis code.
6. Analyze
Separate facts (directly from materials), inferences (drawn from facts), and recommendations (based on inferences). Use explicit phrasing: "the data shows", "this suggests", "a plausible explanation is". Keep analysis reproducible — preserve enough code detail to explain how results were produced.
Guard against the common traps that flip conclusions (full list in references/common-pitfalls.md):
- Analysis base first: every report needs a denominator trail: source population, extracted rows/entities, cleaned rows/entities, each major filter/match, final analysis N, and coverage.
- Correlation ≠ causation / selection effect: the absolute metric of a treated group is not the treatment's effect — good subjects are often selected into treatment. To attribute an effect, compare against a baseline (same entities before/after, a control, or the segment average) and report lift = treated − baseline, not the level alone.
- Comparable windows for before/after effects: pre/post windows must be equally observable. Exclude or separately label entities whose pre-window cannot exist or is truncated (new items, partial-day starts, immature outcomes).
- Experiment/effect gate: for AB tests or causal claims, check experiment unit, randomization/SRM, exposure/trigger definition, contamination, concurrent experiments, confidence intervals, guardrails, and multiple comparisons. If this gate is not satisfied, write descriptive/directional language only.
- Entity ID drift: if the subject can change identity across events (relist, merge, migration), translate keys via the mapping table before joining, or you silently lose data.
- Survivorship bias: a "must exist in both periods" pairing filter can drop exactly the class you're studying (e.g., entities dormant before, active after). Check what the filter removes; prefer full-population baselines for activation questions.
- Lock one scope: keep window, source table, grain, and definitions consistent across all sub-analyses; call out any necessary exception.
- Matched baselines need coverage labels: when a matched/weighted baseline covers only a subset, report matched N and coverage rate next to the first conclusion that uses it.
- Segmentation fields are not interchangeable: verify that category, region, seller, or attribute fields match the business taxonomy. Large
Other/unknown buckets must be reconciled before becoming a business conclusion.
- Contribution metrics need names: if using
sum(delta) or n × avg_delta, call it "sample contribution" or similar; do not let readers interpret it as causal or full-population business impact.
- Verify surprises: a result that contradicts strong domain intuition is usually a data/scope bug — re-derive with a cleaner scope and confirm the mechanism before reporting.
- Check stability: break key findings down by day/window; a pattern that reverses across days is composition-driven, not a law.
7. Visualize intentionally
Use charts only when they improve comprehension. Prefer line charts for trends, bar charts for comparisons, histograms for distributions, tables for exact values. Avoid decorative charts and pie charts.
Read references/visualization-guidelines.md when choosing visuals.
8. Write the report
Default to Markdown with this structure: Title → Executive Summary → Objective and Scope → Analysis Base → Data Overview → Key Findings → Detailed Analysis → Conclusions and Recommendations or Next Validation → Limitations and Risks → Appendix (sources, SQL, method notes).
Read references/report-structure.md before writing the final report.
Always include a Limitations section covering sample size, definition inconsistencies, weak sources, missing history, selection bias, or correlation-vs-causation limits.
Always include a traceability appendix with data sources, methodology notes, assumptions, and any SQL used for data extraction (if applicable). Even for quick answers, retain a compact sources appendix unless the user says not to.
Before final delivery, run the delivery portion of references/analysis-checklist.md. For effect, experiment, or comparison reports, also read references/common-pitfalls.md.
Self-consistency pass before delivery: after edits, verify the summary, tables, and one-line conclusion agree on every number; cross-references ("see section X") resolve; and terminology is consistent. A clean-context review (fresh reader or sub-agent) catches drift that accumulates over many edit rounds. For reporting audiences, state only the final correct conclusion — drop "I previously said X, now Y" self-corrections, "will rerun later" placeholders, and internal version archaeology unless the discarded hypothesis is itself instructive. If a cross-cut is needed for the conclusion, run it before finalizing or narrow the claim.
Scope boundary
This skill handles the analysis layer: define data needs, assess sufficiency, analyze inputs, generate charts and reports. It can coordinate with query tools (ODPS, BigQuery, etc.) for data retrieval, but does not perform web scraping or search.
References
Load these only when relevant:
