aidp-profiling-tables

# `aidp-profiling-tables` — single-table profile Produce a column-level profile of an AIDP table via Spark SQL. Self-contained: control-plane lookups use `oci raw-request`; profiling SQL runs through the bundled `scripts/aidp_sql.py` helper. No aidp MCP server is required. ## When to use - "Profile <table>", "what does <table> look like", "column stats / data quality snapshot". ## Workflow 1. Resolve the table (`aidp-catalog-explore` / `.aidp/catalog.md`) → fully-qualified `catalog.schema.table` and its columns/types. Without a cache, list via `oci raw-request`: `GET /tables?catalogKey=<cat>&schemaKey=<cat.schema>` and filter for the table client-side (see [references/no-mcp-rest-map.md](../../references/no-mcp-rest-map.md)). Use the column types to pick the right per-column profiling SQL. 2. Run bounded profiling SQL via the helper (one cell per call; the scratch notebook + kernel are managed for you): ```bash python "$PLUGIN_DIR/scripts/aidp_sql.py" --region <r> --datalake <ocid> --workspace <ws> --cluster <key> \ --code "spark.sql('''<profiling SQL>''').show(50, truncate=False)" ``` - **Overview:** `SELECT COUNT(*) FROM t` (flag if LARGE; sample for the rest). - **Numeric cols:** `MIN`, `MAX`, `AVG`, `COUNT`, null %, approx distinct (`approx_count_distinct`). - **String/categorical:** null %, `approx_count_distinct`, top-K via `GROUP BY … ORDER BY count DESC LIMIT k`. - **Date/timestamp:** `MIN`/`MAX` range, null %. Use `TABLESAMPLE`/`LIMIT` on large tables to stay cheap; say when you sampled. The helper returns JSON (`status`, `outputs`, `spark_job_ids`) — parse `outputs` for the result rows. 3. Present a per-column table: type, null %, distinct, min/max/mean (numeric), top values (categorical). 4. Offer to feed findings into `.aidp/catalog.md` value dictionaries (`aidp-catalog-init`) and to add data-quality rules (`aidp-data-quality`). ## Reliability rules - Profile from real query output, not assumptions; note sampling. - For very large tables, profile a sample and label it clearly. - The helper mints a UPST from the api_key DEFAULT profile and auto-creates a scratch notebook; pass `--session-profile AIDP_SESSION` only if your tenancy is session-token-only. On a kernel/auth error, refresh (`oci session refresh --profile AIDP_SESSION`) and retry. ## References - [references/oci-raw-request.md](../../references/oci-raw-request.md) · [references/no-mcp-rest-map.md](../../references/no-mcp-rest-map.md) · pairs with `aidp-data-quality`, `aidp-catalog-init`