sundials-devs-python-callbacks

# sundials4py User-Supplied Function Wrapper Skill This skill guides you through creating the nanobind C++ wrapper code needed to expose a SUNDIALS C callback function to Python. These wrappers cannot be autogenerated by litgen because they accept function pointers, so they must be hand-coded following the patterns described below. ## Why these wrappers exist nanobind cannot directly bind C functions that accept raw function pointers. The solution is a multi-layer indirection pattern: 0. **`python` member on the C struct** - a `void*` field on the SUNDIALS object that holds a pointer to the function table. Some objects (e.g. `SUNStepper`) already have this; others may need it added first. 1. **Function table** - a struct of `nb::object` members, one per callback, stored either in the integrator's `user_data` or in the object's `python` member. 2. **Wrapper function** - a C-linkage function matching the original C typedef signature that retrieves the Python callable from the function table and invokes it. 3. **nanobind `m.def`** - a lambda registered with nanobind that accepts a `std::function`, stores it in the function table, and passes the wrapper function to the underlying C "Set" API. Every new user-supplied function requires additions to layers 1-3. If the target object does not yet have a `python` member, layer 0 must be addressed first. ## Step-by-step process > **Note on examples**: The examples below use ARKODE and CVODES to illustrate > the patterns, but the same structure applies identically to all packages > (ARKODE, CVODES, IDAS, KINSOL). Each has its own function table struct, > wrapper functions, convenience macro, and `get_*_fn_table` helper. When > working in a different package, substitute the corresponding names > (e.g. `ida_user_supplied_fn_table`, `get_ida_fn_table`, `BIND_IDA_CALLBACK`, > etc.). When the user asks you to wrap a new callback, gather these inputs first: - **Module**: which submodule (arkode, cvodes, idas, kinsol, or a core module like sundials). - **C "Set" function name**: e.g. `ARKodeSetPostprocessStepFn`. - **C function-pointer typedef**: e.g. `ARKPostProcessFn`. - **Parameter list of the typedef**: needed to detect "special" parameters (see below). - **Storage mechanism**: whether the function table is accessed via `user_data` (like ARKODE/CVODE/IDA/KINSOL) or via a `python` member on the struct (standalone objects like `SUNStepper`, `SUNAdaptController`, etc.). Then produce code fragments as described in the following sections. If the target object does not yet have a `python` member, start with Step 0. Otherwise, skip to Step 1. ### 0. Add a `python` member to the C struct (if it does not already have one) Some SUNDIALS objects (e.g. `SUNStepper`) already carry a `void* python` member for the function table. Others may not. If the struct you are targeting has no `python` member, you must add one before any binding work can proceed. This involves changes to three C-side files. Use `SUNStepper` as the reference implementation for the pattern. All of the relevant code can be found in `include/sundials/sundials_stepper.h`, `src/sundials/sundials_stepper_impl.h`, and `src/sundials/sundials_stepper.c`. #### 0a. Add the field to the private implementation struct Open the `*_impl.h` header that defines the struct internals (e.g. `src/sundials/sundials_logger_impl.h`). Add a `void* python;` field. Place it near the `content` field if one exists, since they serve analogous purposes (`content` is for user-supplied C data, `python` is for the binding-layer function table). **Before** (SUNLogger example): ```c struct SUNLogger_ { /* ... existing fields ... */ void* content; SUNLoggerQueueMsgFn queue_msg; SUNLoggerFlushMsgFn flush_msg; }; ``` **After**: ```c struct SUNLogger_ { /* ... existing fields ... */ void* content; /* Python binding function table (managed by sundials4py) */ void* python; SUNLoggerQueueMsgFn queue_msg; SUNLoggerFlushMsgFn flush_msg; }; ``` #### 0b. Initialize `python` to NULL in the Create function Open the `.c` source file containing the object's `Create` function. Initialize the new field to `NULL` alongside the other field initializations. Also add a forward declaration for the destroy helper, guarded by `#if defined(SUNDIALS_ENABLE_PYTHON)`. This function will be defined on the C++ binding side (see Step 0c) and is called during object destruction to prevent leaks. **Example** - modifying `SUNLogger_Create` in `src/sundials/sundials_logger.c`: ```c /* Forward declaration of function used to destroy any data allocated for Python */ #if defined(SUNDIALS_ENABLE_PYTHON) void SUNLoggerFunctionTable_Destroy(void* ptr); #endif SUNErrCode SUNLogger_Create(SUNComm comm, int output_rank, SUNLogger* logger_ptr) { /* ... existing allocation code ... */ logger->content = NULL; logger->python = NULL; /* <-- add this */ /* ... rest of initialization ... */ } ``` The naming convention for the forward-declared destroy function is `<ObjectName>FunctionTable_Destroy`. Follow what `SUNStepper` uses: `SUNStepperFunctionTable_Destroy`. #### 0c. Clean up `python` in the Destroy function In the same `.c` file, find the object's `Destroy` function. Add a guarded block that calls the destroy helper and nulls out the pointer. This must happen **before** the `free()` of the object itself, but **after** any user-level destroy ops have run (so the user callback is not called after its function table is freed). **Example** - modifying `SUNLogger_Destroy` in `src/sundials/sundials_logger.c`: ```c SUNErrCode SUNLogger_Destroy(SUNLogger* logger_ptr) { /* ... existing checks and cleanup ... */ SUNLogger logger = *logger_ptr; if (logger) { #if SUNDIALS_LOGGING_LEVEL > 0 if (sunLoggerIsOutputRank(logger, NULL)) { SUNHashMap_Destroy(&logger->filenames); } #endif #if defined(SUNDIALS_ENABLE_PYTHON) SUNLoggerFunctionTable_Destroy(logger->python); #endif logger->python = NULL; #if SUNDIALS_MPI_ENABLED if (logger->comm != SUN_COMM_NULL) { MPI_Comm_free(&logger->comm); } #endif free(logger); *logger_ptr = NULL; } return retval; } ``` #### 0d. Define the FunctionTable_Destroy function on the binding side The forward-declared `*FunctionTable_Destroy` function must be defined in a C++ source file in the bindings. Typically this goes in the same `.cpp` file where the nanobind `m.def` registrations live, or in a dedicated helpers file for that module. It casts the `void*` to the function table struct and deletes it. **Example** - in `bindings/sundials4py/sundials/sundials_logger.cpp`: ```cpp // Must be extern "C" since the forward declaration in the C source is in a C file extern "C" void SUNLoggerFunctionTable_Destroy(void* ptr) { delete static_cast<SUNLoggerFunctionTable*>(ptr); } ``` For reference, `SUNStepper` follows this exact pattern - `SUNStepperFunctionTable_Destroy` is forward-declared in `sundials_stepper.c` and defined in the binding-side C++ code. ### 1. Add a member to the function table struct Open the appropriate `*_usersupplied.hpp` header (e.g. `bindings/sundials4py/arkode/arkode_usersupplied.hpp`). Add an `nb::object` member to the function table struct. The member name should be a lowercase, underscore-separated version of the callback's short name. **File**: `bindings/sundials4py/<module>/<module>_usersupplied.hpp` ```cpp struct <module>_user_supplied_fn_table { // ... existing members ... nb::object <member_name>; // <-- add this }; ``` **Example** - adding `postprocessstepfn` to ARKODE: ```cpp struct arkode_user_supplied_fn_table { // ... existing members ... nb::object postprocessstepfn; }; ``` ### 2. Write the wrapper function Immediately below the struct (in the same `*_usersupplied.hpp` file), add a wrapper function template. The wrapper delegates to `sundials4py::user_supplied_fn_caller`. #### Standard callbacks (no special parameters) For callbacks whose parameters map 1:1 between C and C++, use the generic template form: ```cpp template<typename... Args> inline <return_type> <module>_<member_name>_wrapper(Args... args) { return sundials4py::user_supplied_fn_caller< std::remove_pointer_t<CTypeDef>, // the C function-pointer typedef, de-pointered <FunctionTableStruct>, // the function table struct type <MemStruct>, // integrator memory struct (e.g., ARKodeMem) or object type (e.g., SUNStepper) <user_data_offset> // (optional) positional offset of user_data arg; omit or set to 0 for `python`-member objects >(&<FunctionTableStruct>::<member_name>, args...); } ``` **Key parameters of `user_supplied_fn_caller`**: | Template param | Meaning | |----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1st | The C typedef with pointer removed (`std::remove_pointer_t<...>`) | | 2nd | The function table struct type | | 3rd | The integrator memory struct type or SUNDIALS object type | | 4th (optional) | Integer indicating which argument (0-indexed from the end) is the `user_data` pointer. Only needed for integrator callbacks; omit for `python`-member objects. Check the C typedef to find the position. | **Example** - standard ARKODE callback via `user_data`: ```cpp template<typename... Args> inline int arkode_postprocessstepfn_wrapper(Args... args) { return sundials4py::user_supplied_fn_caller< std::remove_pointer_t<ARKPostProcessFn>, arkode_user_supplied_fn_table, ARKodeMem, 1>(&arkode_user_supplied_fn_table::postprocessstepfn, args...); } ``` **Example** - SUNStepper callback via `python` member (no offset needed): ```cpp template<typename... Args> inline SUNErrCode sunstepper_evolve_wrapper(Args... args) { return sundials4py::user_supplied_fn_caller< std::remove_pointer_t<SUNStepperEvolveFn>, SUNStepperFunctionTable, SUNStepper>(&SUNStepperFunctionTable::evolve, std::forward<Args>(args)...); } ``` #### Special callbacks (array or output-pointer parameters) Some C typedefs have parameters that need translation for Python: | C parameter pattern | C++ / Python analog | |-----------------------------------------------------------------|------------------------------------------------------------| | `N_Vector*` used as a 1D array of N_Vectors | `std::vector<N_Vector>` | | Output pointer (e.g., `sunbooleantype*` used to return a value) | Pack into `std::tuple` return | | `sunrealtype*` used as an array | `std::vector<sunrealtype>` or pointer-to-first with length | When any of these appear, you cannot use the generic variadic template. Instead: 1. **Define a custom `std::function` type alias** that mirrors the C typedef but replaces the special parameters with their C++ analogs. Output parameters become additional elements in a `std::tuple` return type. 2. **Write a fully-explicit wrapper function** (not a variadic template) that: - Casts `user_data` / accesses the `python` member to get the function table. - Extracts the Python callable and casts it to the custom `std::function` type. - Converts C arrays to `std::vector` before calling. - Unpacks `std::tuple` return values back into the output pointers after calling. **Example** - CVODES `CVLsPrecSetupFnBS` with an N_Vector array and an output pointer: ```cpp // Custom std::function type: replaces N_Vector* array with std::vector, // and moves the jcurPtrB output pointer into a tuple return value. using CVLsPrecSetupStdFnBS = std::tuple<int, sunbooleantype>( sunrealtype t, N_Vector y, std::vector<N_Vector> yS_1d, N_Vector yB, N_Vector fyB, sunbooleantype jokB, sunrealtype gammaB, void* user_dataB); inline int cvode_lsprecsetupfnBS_wrapper(sunrealtype t, N_Vector y, N_Vector* yS_1d, N_Vector yB,