菜单
Check if upstream Apache DataFusion features (functions, DataFrame ops, SessionContext methods, FFI types) are exposed in this Python project. Use when adding missing functions, auditing API coverage, or ensuring parity with upstream.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/apache/datafusion-python --skill check-upstream命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Audit the user-facing skill at skills/datafusion_python/SKILL.md against the current public Python API. Find new APIs that should be documented, stale mentions of removed/renamed APIs, examples that drifted from current idiomatic style, and places that need a "requires datafusion-python NN or newer" note. Run after upstream syncs and before each release.
TRIGGER — read before adding, changing, or reviewing any __datafusion_*__ capsule getter, any FFI_* export that asks for a TaskContextProvider or an extension codec, or any code that calls FFI_QueryPlanner::new / FFI_TableProvider::new / FFI_{Logical,Physical}ExtensionCodec::new. These methods are one protocol with a settled convention. Do not design it fresh; do not construct a SessionContext inside an extension library.
Audit and improve datafusion-python functions to accept native Python types (int, float, str, bool) instead of requiring explicit lit() or col() wrapping. Analyzes function signatures, checks upstream Rust implementations for type constraints, and applies the appropriate coercion pattern.
正在显示 SKILL.md
| name | check-upstream |
| description | Check if upstream Apache DataFusion features (functions, DataFrame ops, SessionContext methods, FFI types) are exposed in this Python project. Use when adding missing functions, auditing API coverage, or ensuring parity with upstream. |
You are auditing the datafusion-python project to find features from the upstream Apache DataFusion Rust library that are not yet exposed in this Python binding project. Your goal is to identify gaps and, if asked, implement the missing bindings.
IMPORTANT: The Python API is the source of truth for coverage. A function or method is considered "exposed" if it exists in the Python API (e.g., python/datafusion/functions.py), even if there is no corresponding entry in the Rust bindings. Many upstream functions are aliases of other functions — the Python layer can expose these aliases by calling a different underlying Rust binding. Do NOT report a function as missing if it appears in the Python __all__ list and has a working implementation, regardless of whether a matching #[pyfunction] exists in Rust.
IMPORTANT: audit the total upstream surface, not the delta since the last pin. Gaps accumulate across syncs. A patch-release bump with a "bug fixes only" changelog does not mean there is nothing to find — pre-existing gaps from earlier majors still need to be surfaced. Always run the full comparison.
If a recent upstream bump required any of the following while fixing
compile errors in crates/core/ or the FFI example, treat that as a
hard signal that user-facing surface area grew and run this skill
before considering the bump done. Each pattern corresponds to a class of
gap that frequently shows up in the audit:
| Signal during PR 1 compile fix | Likely gap to check |
|---|---|
New Expr::* variant added to a non-exhaustive match (HigherOrderFunction, Lambda, LambdaVariable, …) | New lambda / higher-order scalar functions (any_match, array_transform, list_transform, …) |
New ScalarValue::* variant (ListView, LargeListView, …) | New scalar / array functions that consume or produce the type |
New required trait method on ExecutionPlan / TableProvider / *UDFImpl (apply_expressions, …) | Corresponding capability on the Python wrapper class |
Renamed or restructured struct field (e.g. Cast.data_type → Cast.field: FieldRef) | Any Python accessor / SKILL.md doc that read the old field |
Newly deprecated trait method with a _with_args / _with_options replacement | The *_with_options variant frequently warrants a separate Python entry point |
PR 1 of dev/release/upstream-sync.md asks you to log these signals as
they appear. When you run this skill, use that log as a checklist: every
entry must either show up in the audit output or be explicitly skipped
with a reason.
The user may specify an area via $ARGUMENTS. If no area is specified or "all" is given, check all areas.
Upstream source of truth:
Where they are exposed in this project:
python/datafusion/functions.py — each function wraps a call to datafusion._internal.functionscrates/core/src/functions.rs — #[pyfunction] definitions registered via init_module()Evaluated and not requiring separate Python exposure:
get_field_path — already covered by get_field(expr, *names), which takes a
variadic field path and dispatches to the same underlying
functions::core::get_field UDF as the upstream get_field_path helper.How to check:
python/datafusion/functions.py (check the __all__ list and function definitions)#[pyfunction]. Many functions are aliases that reuse another function's Rust binding.__all__ list / function definitionsUpstream source of truth:
Where they are exposed in this project:
python/datafusion/functions.py (aggregate functions are mixed in with scalar functions)crates/core/src/functions.rsEvaluated and not requiring separate Python exposure:
count_distinct — covered by count(expr, distinct=True). Both forms call
count_udaf with distinct: bool = true and produce the same logical plan.sum_distinct — covered by sum(expr, distinct=True).avg_distinct — covered by avg(expr, distinct=True).How to check:
python/datafusion/functions.py (check __all__ list and function definitions)Upstream source of truth:
Where they are exposed in this project:
python/datafusion/functions.py (window functions like rank, dense_rank, lag, lead, etc.)crates/core/src/functions.rsHow to check:
python/datafusion/functions.py (check __all__ list and function definitions)Upstream source of truth:
Where they are exposed in this project:
python/datafusion/functions.py and python/datafusion/user_defined.py (TableFunction/udtf)crates/core/src/functions.rs and crates/core/src/udtf.rsHow to check:
Upstream source of truth:
Where they are exposed in this project:
python/datafusion/dataframe.py — the DataFrame classcrates/core/src/dataframe.rs — PyDataFrame with #[pymethods]Evaluated and not requiring separate Python exposure:
show_limit — already covered by DataFrame.show(), which provides the same functionality with a simpler APIwith_param_values — already covered by the param_values argument on SessionContext.sql(), which accomplishes the same thing more robustlyunion_by_name_distinct — already covered by DataFrame.union_by_name(distinct=True), which provides a more Pythonic APIHow to check:
python/datafusion/dataframe.py — this is the source of truth for coveragecrates/core/src/dataframe.rs) may be consulted for context, but a method is covered if it exists in the Python APIUpstream source of truth:
Where they are exposed in this project:
python/datafusion/context.py — the SessionContext classcrates/core/src/context.rs — PySessionContext with #[pymethods]How to check:
python/datafusion/context.py — this is the source of truth for coveragecrates/core/src/context.rs) may be consulted for context, but a method is covered if it exists in the Python APIUpstream source of truth:
Where they are exposed in this project:
crates/core/src/ and crates/util/src/examples/datafusion-ffi-example/src/Cargo.toml and crates/core/Cargo.tomlDiscovering currently supported FFI types:
Grep for use datafusion_ffi:: in crates/core/src/ and crates/util/src/ to find all FFI types currently imported and used.
Evaluated and not requiring direct Python exposure: These upstream FFI types have been reviewed and do not need to be independently exposed to end users:
FFI_ExecutionPlan — already used indirectly through table providers; no need for direct exposureFFI_PhysicalExpr / FFI_PhysicalSortExpr — internal physical planning types not expected to be needed by end usersFFI_RecordBatchStream — one level deeper than FFI_ExecutionPlan, used internally when execution plans stream resultsFFI_SessionRef / ForeignSession — session sharing across FFI; Python manages sessions natively via SessionContextFFI_SessionConfig — Python can configure sessions natively without FFIFFI_ConfigOptions / FFI_TableOptions — internal configuration plumbingFFI_PlanProperties / FFI_Boundedness / FFI_EmissionType — read from existing plans, not user-facingFFI_Partitioning — supporting type for physical planningFFI_Option, FFI_Result, WrappedSchema, WrappedArray, FFI_ColumnarValue, FFI_Volatility, FFI_InsertOp, FFI_AccumulatorArgs, FFI_Accumulator, FFI_GroupsAccumulator, FFI_EmitTo, FFI_AggregateOrderSensitivity, FFI_PartitionEvaluator, FFI_PartitionEvaluatorArgs, FFI_Range, FFI_SortOptions, FFI_Distribution, FFI_ExprProperties, FFI_SortProperties, FFI_Interval, FFI_TableProviderFilterPushDown, FFI_TableType) — used as building blocks within the types above, not independently exposedHow to check:
use datafusion_ffi:: in crates/core/src/ and crates/util/src/, then compare against the upstream datafusion-ffi crate's lib.rs exportsfrom_pycapsule() methodScalarUDFExportable) for FFI objectsinit_module() and Python __init__.pyexamples/datafusion-ffi-example/datafusion-spark crate)Upstream source of truth:
Where they are exposed in this project:
python/datafusion/functions/spark.py — each function wraps
a call to datafusion._internal.functions.spark; the public surface is
the module's __all__ list.crates/core/src/spark_functions.rs — #[pyfunction]
definitions registered via init_module() and re-exported under
datafusion._internal.functions.spark.Coverage policy: The spark namespace mirrors
pyspark.sql.functions parameter names and shapes exactly so pyspark
callers can paste code unchanged. Extras over pyspark are permitted as
long as positional pyspark calls still work — for example, the spark
avg / try_sum / collect_list / collect_set retain the
distinct/filter/order_by/null_treatment kwargs from the main
namespace while pyspark's single-positional form continues to work.
How to check:
datafusion-spark function list from the crate
source under datafusion/spark/src/function/ (each subdirectory is a
category: string/, math/, datetime/, etc.). The crate's
function.rs collects all ScalarUDF factories.pyspark.sql.functions for the public-facing
shape — pyspark is the contract this namespace is matching.python/datafusion/functions/spark.py's __all__. A function is
covered if it exists in the Python spark namespace, even if it
aliases another function's Rust binding.__all__ Hygiene (functions.py and functions/spark.py)Independent of upstream parity, also flag public def symbols in
python/datafusion/functions.py and python/datafusion/functions/spark.py
that are missing from that file's __all__. These are functions a user
can call but that do not show up in
from datafusion.functions import *, in tab-completion against the
namespace, or in generated API docs — typically an oversight rather than
an intentional omission.
How to check:
^def ([a-z_][a-z0-9_]*)\( in each file to enumerate every
public function definition.__all__ list at the top of the same file._).A historical example: instr and position shipped as public defs but
were absent from __all__ until the gap was caught here.
For each finding, propose adding the name to __all__ in alphabetical
position with the existing entries.
After identifying missing APIs, search the open issues at https://github.com/apache/datafusion-python/issues for each gap to see if an issue already exists requesting that API be exposed. Search using the function or method name as the query.
For each area checked, produce a report like:
## [Area Name] Coverage Report
### Currently Exposed (X functions/methods)
- list of what's already available
### Missing from Upstream (Y functions/methods)
- function_name — brief description of what it does (existing issue: #123)
- function_name — brief description of what it does (no existing issue)
### Notes
- Any relevant observations about partial implementations, naming differences, etc.
If the user asks you to implement missing features, follow these patterns:
Step 1: Rust binding in crates/core/src/functions.rs:
#[pyfunction]
#[pyo3(signature = (arg1, arg2))]
fn new_function_name(arg1: PyExpr, arg2: PyExpr) -> PyResult<PyExpr> {
Ok(datafusion::functions::module::expr_fn::new_function_name(arg1.expr, arg2.expr).into())
}
Then register in init_module():
m.add_wrapped(wrap_pyfunction!(new_function_name))?;
Step 2: Python wrapper in python/datafusion/functions.py:
def new_function_name(arg1: Expr, arg2: Expr) -> Expr:
"""Description of what the function does.
Args:
arg1: Description of first argument.
arg2: Description of second argument.
Returns:
Description of return value.
"""
return Expr(f.new_function_name(arg1.expr, arg2.expr))
Add to __all__ list.
Step 1: Rust binding in crates/core/src/dataframe.rs:
#[pymethods]
impl PyDataFrame {
fn new_method(&self, py: Python, param: PyExpr) -> PyDataFusionResult<Self> {
let df = self.df.as_ref().clone().new_method(param.into())?;
Ok(Self::new(df))
}
}
Step 2: Python wrapper in python/datafusion/dataframe.py:
def new_method(self, param: Expr) -> DataFrame:
"""Description of the method."""
return DataFrame(self.df.new_method(param.expr))
Step 1: Rust binding in crates/core/src/context.rs:
#[pymethods]
impl PySessionContext {
pub fn new_method(&self, py: Python, param: String) -> PyDataFusionResult<PyDataFrame> {
let df = wait_for_future(py, self.ctx.new_method(¶m))?;
Ok(PyDataFrame::new(df))
}
}
Step 2: Python wrapper in python/datafusion/context.py:
def new_method(self, param: str) -> DataFrame:
"""Description of the method."""
return DataFrame(self.ctx.new_method(param))
FFI types require a full pipeline from C struct through to a typed Python wrapper. Each layer must be present.
Step 1: Rust PyO3 wrapper class in a new or existing file under crates/core/src/:
use datafusion_ffi::new_type::FFI_NewType;
#[pyclass(from_py_object, frozen, name = "RawNewType", module = "datafusion.module_name", subclass)]
pub struct PyNewType {
pub inner: Arc<dyn NewTypeTrait>,
}
#[pymethods]
impl PyNewType {
#[staticmethod]
fn from_pycapsule(obj: &Bound<'_, PyAny>) -> PyDataFusionResult<Self> {
let capsule = obj
.getattr("__datafusion_new_type__")?
.call0()?
.downcast::<PyCapsule>()?;
let ffi_ptr = unsafe { capsule.reference::<FFI_NewType>() };
let provider: Arc<dyn NewTypeTrait> = ffi_ptr.into();
Ok(Self { inner: provider })
}
fn some_method(&self) -> PyResult<...> {
// wrap inner trait method
}
}
Register in the appropriate init_module():
m.add_class::<PyNewType>()?;
Step 2: Python Protocol type in the appropriate Python module (e.g., python/datafusion/catalog.py):
class NewTypeExportable(Protocol):
"""Type hint for objects providing a __datafusion_new_type__ PyCapsule."""
def __datafusion_new_type__(self) -> object: ...
Step 3: Python wrapper class in the same module:
class NewType:
"""Description of the type.
This class wraps a DataFusion NewType, which can be created from a native
Python implementation or imported from an FFI-compatible library.
"""
def __init__(
self,
new_type: df_internal.module_name.RawNewType | NewTypeExportable,
) -> None:
if isinstance(new_type, df_internal.module_name.RawNewType):
self._raw = new_type
else:
self._raw = df_internal.module_name.RawNewType.from_pycapsule(new_type)
def some_method(self) -> ReturnType:
"""Description of the method."""
return self._raw.some_method()
Step 4: ABC base class (if users should be able to subclass and provide custom implementations in Python):
from abc import ABC, abstractmethod
class NewTypeProvider(ABC):
"""Abstract base class for implementing a custom NewType in Python."""
@abstractmethod
def some_method(self) -> ReturnType:
"""Description of the method."""
...
Step 5: Module exports — add to the appropriate __init__.py:
NewType) to python/datafusion/__init__.pyNewTypeProvider) if applicableNewTypeExportable) if it should be publicStep 6: FFI example — add an example implementation under examples/datafusion-ffi-example/src/:
// examples/datafusion-ffi-example/src/new_type.rs
use datafusion_ffi::new_type::FFI_NewType;
// ... example showing how an external Rust library exposes this type via PyCapsule
Checklist for each FFI type:
from_pycapsule() methodNewTypeExportable) for FFI objectsinit_module() and Python __init__.pyexamples/datafusion-ffi-example/Table | TableProviderExportable)crates/core/Cargo.toml — check the datafusion dependency version to ensure you're comparing against the right upstream version.array_append / list_append) should both be exposed if upstream supports them.__all__ list in functions.py to see what's publicly exported vs just defined.