data-profile

name	data-profile
description	"Use when onboarding a dataset or asking 'what\'s in this data?' — per-column quality report with join-compatibility checks; flags possible PII/sensitive columns"

data-profile : know the data before you trust it

DS-STAR's single biggest correctness lever is analyzing every file before planning — its ablation shows hard-task accuracy collapses 45.24 → 26.98 without it (see ../ds-star-plus/references/evidence.md). This skill makes that step a first-class output: a structured data-quality report you can read, share, and act on, independent of any one question.

When this applies

Onboarding a new dataset; deciding whether data is fit to answer a question; hunting the silent issues (hidden nulls, sentinels, duplicate fan-out, type surprises, join mismatches) that wreck analyses later. Run it standalone, or as the pre-step before ds-clarify / ds-star-plus.

The cardinal rule

Print real content; never assume structure. The same discipline DS-STAR's analyzer uses: load and show — actual column names, actual value spellings, actual dtypes — instead of guessing from the file name. Surfaced reality is the whole value.

How to run it

Stage 0 — PII / sensitive-data scan

PII / sensitive-data scan: Run scripts/detect_pii.py's scan_columns over the loaded column names and a sample of values. If any column is flagged, surface it prominently (⚠️ possible PII in columns: ...) before any data is summarized or leaves the sandbox. Recommend masking, exclusion, or confirming with the data owner before proceeding. Note: this is a heuristic (false positives and negatives are possible).

Stage 1 — Inventory

List every file in scope with size and type. For a folder, group by extension. Do not load gigabytes blindly — sample large files.

Stage 2 — Per-file profile

Run ../ds-star-plus/scripts/analyze_file.py <path> (the ready-made describer) as the base, then extend it with the data-quality checks in references/checks.md. For each file emit:

• shape, columns + dtypes, sheet/table names (Excel/SQLite — list ALL, data often hides off-sheet);
• missingness per column (count + %), and suspected sentinels masquerading as values (-, ?, NEW, 0, 9999, empty string);
• cardinality per column; flag candidate keys (unique, non-null) and constant columns;
• duplicate-row count and the key that would dedup them;
• for numeric columns: min/max/mean/median + simple outlier flags (IQR or z-score);
• for low-cardinality object columns: value counts;
• encoding / parse anomalies (mixed types in a column, stray whitespace, inconsistent casing).

Stage 3 — Cross-file (folder mode)

For columns that share a name/role across files, check join compatibility: dtype match, value-overlap, and whether a join would fan out (one-to-many) or drop rows (non-overlap). This is where multi-file analyses silently go wrong; flag it before anyone joins.

The per-file digest this stage produces — column names + dtypes, sheet/table names, low-cardinality value lists, and candidate keys — is exactly what the column-level retrieval protocol scores against (../ds-star-plus/references/retrieval.md Stage 3). In data-lake mode (N > ~100 files), this profile is the input that lets retrieval find on-topic files by structure rather than filename guesswork.

Stage 4 — Report

Emit a single report (markdown by default; the structure is in references/checks.md), ending with a "Watch-outs" list — the specific issues that would bite an analysis (e.g. "amount has 3% nulls and a - sentinel; order_id is NOT unique — fan-out risk on join"). State assumptions you made (e.g. sampling) plainly.

Output

A readable data-quality report plus a short watch-outs list. If the user is heading into a specific analysis, hand the watch-outs to ds-clarify so they become spec decisions (null policy, dedup key, units) rather than silent surprises.

Quick reference

Data-quality checks + report structure: references/checks.md. Base describer (reused): ../ds-star-plus/scripts/analyze_file.py.