audit-skill-md

<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> # Audit `skills/datafusion_python/SKILL.md` You are auditing the user-facing skill at [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) against the current state of the Python API. The skill is the source of truth for how AI coding assistants are taught to write `datafusion-python` code, so it must match what the project actually ships. This skill identifies gaps caused by upstream syncs, refactors, or renames, and (if asked) applies the edits directly to `SKILL.md`. The skill is most usefully run **after** the `check-upstream` step of an upstream sync (see `dev/release/upstream-sync.md`) — once any new APIs are exposed, this skill makes sure they get documented. ## What the skill covers The user-facing `SKILL.md` documents these public surfaces. This list is not exhaustive — if a new top-level area is added (e.g., a new `Catalog` API exposed at the package root), include it. | Surface | Module | Sections in SKILL.md | |---|---|---| | `SessionContext` | `python/datafusion/context.py` | "Data Loading" | | `DataFrame` | `python/datafusion/dataframe.py` | "DataFrame Operations Quick Reference", "Executing and Collecting Results", "Idiomatic Patterns" | | `Expr` | `python/datafusion/expr.py` | "Expression Building", "Common Pitfalls" | | `functions` | `python/datafusion/functions/__init__.py` | "Available Functions (Categorized)", scattered uses throughout | | `functions.spark` | `python/datafusion/functions/spark.py` | "Available Functions (Categorized)" → "Spark-Compatible Functions" subsection | | Top-level helpers (`col`, `lit`, `WindowFrame`, ...) | `python/datafusion/__init__.py` | "Import Conventions", "Core Abstractions" | ## Scope argument The user may specify a scope via `$ARGUMENTS` to limit the audit. If no scope is given or `all` is specified, audit every area. | Scope | Audit target | |---|---| | `session-context` | `SessionContext` methods and the "Data Loading" section | | `dataframe` | `DataFrame` methods and the operations / executing / patterns sections | | `expr` | `Expr` methods/operators and the "Expression Building" section | | `functions` | `functions/__init__.py` `__all__` and the "Available Functions (Categorized)" section | | `spark-functions` | `functions/spark.py` `__all__`, the "Spark-Compatible Functions" subsection, and the divergent-semantics table | | `patterns` | "Idiomatic Patterns" section — confirm patterns still match recommended style | | `pitfalls` | "Common Pitfalls" — confirm each pitfall still reproduces, drop ones fixed upstream | | `version-notes` | Cross-check version annotations (see below) | | `all` | Everything above | ## Inputs to read Before producing the report: 1. `skills/datafusion_python/SKILL.md` — the document being audited. 2. The relevant Python module(s) for the chosen scope. Public surface is the `__all__` list (where defined) plus `class` and `def` symbols not prefixed with `_`. 3. `Cargo.toml` (root) for the current `datafusion-python` version — read the `version` field under `[workspace.package]` (format `NN.0.0`). The major version always matches the upstream `datafusion` crate, so a single `datafusion-python` version expresses both. `python/datafusion/__init__.py`'s `__version__` is the same value exposed at runtime. 4. Recent commits touching the relevant module(s) for context on what changed since the last sync: ```bash git log --oneline -- python/datafusion/dataframe.py | head -20 ``` ## What to look for Walk through each scoped area and flag four kinds of issues. ### 1. New APIs not mentioned For each public symbol in the module's `__all__` (or each public class method), check whether it appears anywhere in `SKILL.md`. A symbol is "covered" if it shows up in: - A code block (the strongest signal — it's demonstrated). - The "Available Functions (Categorized)" list. - The SQL-to-DataFrame Reference table. **Decide whether each missing symbol deserves an entry.** Not every public symbol belongs in `SKILL.md` — the skill is curated for the patterns users hit daily, not exhaustive API reference. Use these heuristics: - **Add it** if it replaces or supersedes something already in the skill (e.g., a new operation that is the idiomatic alternative to a documented workaround). - **Add it** if it fits a category already present (a new aggregate function goes in the aggregate list; a new join type goes in the joining section). - **Add it** if it changes how a documented pattern should be written. - **Skip it** if it is genuinely niche / advanced / experimental. - **Skip it** if it is internal plumbing exposed for FFI but not user-facing. When you flag a missing symbol, include a one-line proposed insertion point (which section / which table row) so a reviewer can decide quickly. ### 2. Stale mentions For each function name, method name, or import shown in `SKILL.md`, verify it still exists in the current API: - Function names mentioned in prose or in the categorized list should appear in `python/datafusion/functions/__init__.py`'s `__all__`. - Spark function names mentioned in the "Spark-Compatible Functions" subsection should appear in `python/datafusion/functions/spark.py`'s `__all__`. Also confirm the divergent-semantics table still matches the current spark vs. main signatures. - Method calls in code blocks should resolve against the current class. - Imports (`from datafusion import ...`) should succeed against the current `__init__.py`. A quick way to check imports without running them: ```bash python -c "from datafusion import SessionContext, col, lit; from datafusion import functions as F; print('ok')" ``` For each stale mention, propose either: - a rename to the current name, or - removal if the API is gone with no replacement. ### 3. Examples that drifted from idiomatic style The skill teaches a Pythonic style: prefer plain strings to `col(...)` when a column reference is all you need; prefer raw Python values to `lit(...)` where auto-wrapping applies. Recent refactors (see the `make-pythonic` skill) keep moving more functions toward accepting native types. For each code example in `SKILL.md`, check: - Does it use `lit(value)` where a raw value would work? Comparison RHS, arithmetic with a column, etc. all auto-wrap. (Reserve `lit()` for the cases listed in pitfall #2.) - Does it use `col("name")` where a plain string would work? `select(...)`, `aggregate([keys], ...)`, `sort(...)`, `sort_by(...)` all accept plain name strings. - Do `functions.py` calls match the current pythonic signature for that function? If `make-pythonic` recently changed a signature (e.g., `repeat(string, n: Expr | int)`), the example should pass `3` rather than `lit(3)`. - Does any example use a deprecated or removed parameter name? For drift, propose the updated snippet. If the change is purely stylistic and the older form still works, mark the suggestion as **non-blocking**. ### 4. Missing or stale version notes When an API depends on a specific version, the skill should say so — otherwise an agent referencing the skill in an older project will write code that fails at import or at runtime. `datafusion-python` shares its major version number with the upstream `datafusion` crate (e.g., `datafusion-python 53.x` tracks upstream `datafusion 53`). Always express version requirements in terms of `datafusion-python` only — there is no need to call out upstream and package versions separately. Add a version note when: - A method or function shown in the skill was added in a specific release (e.g., a new `DataFrame` method that didn't exist before 53). - A breaking change altered behavior in a specific release (signature change, default-value change, new required argument). - A pitfall was fixed in a specific release. Either annotate the pitfall block with "fixed in datafusion-python NN, kept here for users on older versions" or remove it once the supported floor moves past that version. Format for version notes (inline, italicized): ```markdown *Requires datafusion-python 53 or newer.* ``` For each missing/stale version note, propose the exact line and where it belongs. ## How to discover changes since the last audit If the user supplies a previous version or commit SHA where the audit was last run, diff against it: ```bash # Public-API-relevant changes since SHA <prev> git log --oneline <prev>..HEAD -- python/datafusion/ # Whose signatures actually moved git diff <prev>..HEAD -- python/datafusion/functions.py | grep '^[+-]def ' ``` If no prior audit point is given, fall back to "since the last upstream sync" by inspecting commits that touch `Cargo.toml`'s `datafusion` pin: ```bash git log --oneline -- Cargo.toml | grep -i datafusion | head -5 ``` ## Output Format Produce a report grouped by scope. Each finding is one bullet with a proposed action, so a maintainer can review the list quickly and apply edits in order. ``` ## SKILL.md Audit (scope: <scope>) Audited against: - skills/datafusion_python/SKILL.md @ <git SHA / "working tree"> - datafusion-python <version> ### New APIs to cover - `DataFrame.foo()` — added in datafusion-python 53. Insert in "DataFrame Operations Quick Reference" under <subsection>. Proposed snippet: ```python df.foo(...) ``` ### Stale mentions - "old_function_name" referenced in the categorized list (line N) — renamed to "new_function_name". Replace. ### Drifted examples - "Filtering" section, `df.filter(col("a") > lit(10))` — drop `lit(10)`, auto-wrap applies. (non-blocking) - "Aggregation" section, `df.aggregate([col("region")], ...)` — pass `"region"` as a plain string per "Projection" guidance. ### Version notes - `DataFrame.foo()` block needs *Requires datafusion-python 53 or newer.* - "Common Pitfalls" #N — fixed in datafusion-python 53; remove the pitfall and update the SQL-to-DataFrame row to no longer flag the workaround. ### No-change confirmed - `SessionContext` data-loading section — all entries match current API. ``` If asked to apply the changes, edit `skills/datafusion_python/SKILL.md` directly with `Edit` tool calls, one finding at a time, and re-run the relevant doctest sanity check at the end: ```bash pytest --doctest-modules python/datafusion -q ``` ## What NOT to flag - **Internal helpers / underscored names.** Private symbols are not part of the user-facing surface. - **Functions intentionally omitted.** Niche / advanced APIs (custom catalogs, raw FFI plumbing, low-level execution plan accessors) live in the API reference, not the skill. If an omission was deliberate and a comment / commit explains why, leave it out. - **Style nits inside explanatory prose.** The skill mixes example code and prose; only enforce the pythonic style on actual code blocks. - **Function-by-function coverage of every `functions.py` symbol.** The "Available Functions (Categorized)" list is curated by category, not exhaustive. Adding a single new aggregate to the aggregate list is enough — the user follows the pointer to the API reference for the rest. ## Coordination with other skills - Run `/check-upstream` first to expose any missing upstream APIs into the Python layer. Without that, this skill cannot recommend documenting something that is not yet exposed. - Run `/make-pythonic` before this skill if a Pythonic-signature pass is planned for a release — that way this skill can update examples to the final signature in one shot rather than churning them twice. - The order during an upstream sync (PR 3 of `dev/release/upstream-sync.md`) is therefore: `/check-upstream` → `/make-pythonic` (optional) → `/audit-skill-md`.