python-c

name	python-c
description	Accelerating Python with native code — choosing an interface layer (C-API, cffi, Cython, nanobind, Rust), modern CPython C-API hygiene, free-threading, the stable ABI, and (the hard part) building/shipping compiled extensions under a hatchling project. Use when adding, reviving, or packaging a C/C++/Rust acceleration module in a Python library.

Python + native acceleration

Purpose

Guide the decision to drop to native code and the mechanics of doing it well in a modern (2026) Python project: pick the right interface, keep the C-API portable across 3.12–3.14 and the free-threaded build, and — the part most projects get wrong — actually compile and ship the extension when the build backend is hatchling. Assumes the conventions in the python-backend skill (hatchling, uv pip install ., no editable installs).

Decide first: do you even need native code?

Native code is a standing tax — a compiler in the build, a platform-specific wheel matrix, a C-API that shifts under you every release. Pay it only when profiling says so.

• Profile before porting. Measure with pytest-benchmark / perf / py-spy. Know the hot function and its call frequency.
• Call frequency dominates. A pure-Python function at ~200 ns/call (≈5 M/sec) that runs once per input gains nothing measurable from a ~50 ns C version — the win is real only in a tight inner loop over large data. (This is exactly why Amara's isxml was correctly revived as pure Python, not C: it runs at most once per input source. See the cxmlstring-revival note.)
• Cheaper wins first. Algorithmic fixes, functools/itertools, vectorising with NumPy, or PyPy often beat hand-written C without any of the tax.
• If it's I/O-bound, native code won't help — that's what asyncio is for.

Write down the benchmark and the decision. "We chose C because X was N% of runtime at M calls/sec" is a comment worth its lines.

Choosing the interface layer

Rough guide, fastest-to-adopt last:

Approach	Use when	Notes
ctypes	Calling an existing shared library, no build step	Stdlib; per-call overhead is high — fine for coarse calls, bad for hot loops
cffi	Binding an existing C library/ABI from Python	Pure wrapper, no new language; ABI or API mode; low friction
Cython	Writing new accelerated logic that's mostly Python-shaped	A compiler + language, not just a binder; great for incremental typing of existing .py; needs a C toolchain
nanobind	New C++ bindings	The modern successor to pybind11 — ~4× faster compiles, ~5–10× smaller/lower-overhead, and can target the stable ABI. Prefer over pybind11 for new work
pybind11	Existing C++ binding codebases	Still fine; nanobind is the greenfield choice
Raw CPython C-API (#include <Python.h>)	A small, self-contained hot kernel with no C++	Maximum control, maximum maintenance; you own every refcount and every API deprecation
Rust + PyO3 / maturin	New native module, greenfield, memory-safety matters	Not C, but often the better answer — maturin is a first-class build backend and sidesteps most C-API footguns

Heuristic: binding existing C → cffi; existing C++ → nanobind; new hot kernel → Cython (Python-shaped) or Rust/PyO3 (systems-shaped); raw C-API only for a tiny, stable kernel you're willing to hand-maintain.

Always ship a visible pure-Python fallback

An optional accelerator must degrade observably, never silently. The anti-pattern that killed Amara's isxml:

try:
    from amara.cmodules.cxmlstring import isxml  # never built under hatchling
except ImportError:
    pass  # <-- symbol now undefined; failure swallowed; dead for years

Do this instead — the pure version is the always-present baseline, the C version merely shadows it when present:

try:
    from amara.cmodules.cxmlstring import isxml as isxml   # optional C accelerator
except ImportError:
    from amara.uxml.xmlstring import isxml as isxml         # pure-Python fallback
    _log.debug('cxmlstring C accelerator unavailable; using pure-Python isxml')

Keep the two implementations behaviourally identical and test both against the same suite (parametrise over the import). The pure path is a feature (no-toolchain installs, PyPy, free-threading), not a consolation prize.

Modern C-API hygiene (3.12 → 3.14)

The C-API removes long-deprecated surface on a schedule; code that compiled on 3.9 may not on 3.12+.

• Legacy Unicode API is gone. PyUnicode_AS_UNICODE, PyUnicode_GET_SIZE, Py_UNICODE* buffers were removed in 3.12. Use PyUnicode_FromStringAndSize, PyUnicode_AsUTF8AndSize, PyUnicode_READ/PyUnicode_KIND, or convert to bytes and work on those. (This removal is precisely why Amara's real cxmlstring.c stopped compiling.)
• Prefer the high-level API over poking struct internals (ob_type, tp_*, ->ob_refcnt) — the internals move; the functions/macros (Py_TYPE, Py_SET_TYPE, Py_REFCNT) are stable.
• Refcounts and errors: honour borrowed-vs-new reference rules, Py_XDECREF on every error path, always return NULL with an exception set. Consider building with Py_DEBUG to catch leaks early.
• Use multi-phase init (PyModuleDef + PyModuleDef_Slot / Py_mod_exec) rather than a bare PyModule_Create where practical — it's required for clean subinterpreter and free-threading support.
• Guard version-specific code with #if PY_VERSION_HEX >= 0x030C0000.

Free-threaded (no-GIL) build

Free threading is experimental in 3.13 and officially supported in 3.14 (PEP 779) — no longer a curiosity. A C extension that doesn't opt in forces the GIL back on at import (with a warning), silently defeating the point.

• Declare support: for single-phase init call PyUnstable_Module_SetGIL(...); for multi-phase add the Py_mod_gil slot with Py_MOD_GIL_NOT_USED.
• Guard it — the symbol only exists on the free-threaded build:

  #ifdef Py_GIL_DISABLED
      /* Py_mod_gil slot / PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED) */
  #endif
  

• Opting in is a promise: without the GIL, your module's own shared/global state now needs real locking. Audit statics and caches before declaring NOT_USED.
• nanobind and PyO3 handle much of this for you; raw C-API means doing it by hand.

Rider — the 3.14 floor is for free-threading support, not for native code in general. requires-python = '>=3.14' does not give you free threading: it remains a separate build (python3.14t), not the default interpreter, so a plain 3.14 install is still a GIL build. Therefore:

• A native module that opts into and tests on the free-threaded build MUST set requires-python = '>=3.14', declare Py_mod_gil / Py_MOD_GIL_NOT_USED, and run the t build in CI — free threading is supported (not experimental) only from 3.14, and pinning it lets you drop the version guards and make the no-GIL promise a tested one.
• A native module that is merely a single-threaded accelerator SHOULD keep the project's normal floor (3.12) and, where feasible, ship an abi3 wheel — do not raise the floor just because the module is written in C. Native speed is orthogonal to the GIL build.

Stable ABI / Limited API — collapse the wheel matrix

By default you build one wheel per (Python minor × platform). Compiling against the Limited API (#define Py_LIMITED_API 0x030C0000) yields an abi3 wheel that works on that minor and every later one — turning cp312 cp313 cp314 into a single cp312-abi3.

• Worth it for widely-distributed libraries; the API subset is smaller, so not always possible for heavy C++/templated bindings.
• nanobind can target the stable ABI; Cython and raw C-API can with care. cffi's API mode also supports abi3.
• Free-threaded builds are a separate ABI axis (t-tagged) and are not covered by abi3 yet — you still ship a distinct free-threaded wheel.

Packaging under hatchling — the part that bites

Hatchling does not compile C extensions out of the box. Unlike setuptools, a bare [tool.hatch.build.targets.wheel] happily produces a pure py3-none-any wheel and silently ignores your .c/.pyx files. Combined with the except ImportError: pass anti-pattern, that's how an accelerator goes dead and nobody notices (Amara's setup.py→hatchling migration did exactly this).

You must add an explicit build hook. Options, simplest first:

1. Cython → hatch-cython. Note the extra build-system.requires — the hook imports Cython/setuptools at load time, before its own deps resolve:

[build-system]
requires = ['hatchling', 'hatch-cython', 'Cython', 'setuptools']
build-backend = 'hatchling.build'

[tool.hatch.build.targets.wheel.hooks.cython]
dependencies = ['hatch-cython']

[tool.hatch.build.targets.wheel.hooks.cython.options]
directives = { boundscheck = false, language_level = 3 }

2. Raw C-API / small kernel → custom hatch_build.py. Implement BuildHookInterface, drive compilation, and flag the wheel non-pure so it gets a platform tag:

# hatch_build.py
from hatchling.builders.hooks.plugin.interface import BuildHookInterface

class CustomBuildHook(BuildHookInterface):
    PLUGIN_NAME = 'custom'

    def initialize(self, version, build_data):
        build_data['pure_python'] = False   # emit a platform tag, not py3-none-any
        build_data['infer_tag'] = True
        # ... invoke the compiler (e.g. a setuptools/distutils Extension build) ...
        build_data['artifacts'].append('*.so')

[tool.hatch.build.targets.wheel.hooks.custom]

3. Anything non-trivial (CMake/Meson, C++/nanobind) → a dedicated backend. scikit-build-core (CMake) and meson-python (Meson) are the mature compiled backends; maturin for Rust. You can drive scikit-build-core from hatchling via [tool.hatch.build.targets.wheel.hooks.scikit-build], but for a heavy native component it's usually cleaner to make the compiled backend the primary one for that project.

Two more packaging musts

• The sdist must contain the native source. If only-include/sources omits clib/ (or your .pyx), even a wheel-from-sdist build fails. Make sure the C sources ship. Amara's only-include originally dropped clib/ — a latent build failure.
• Non-pure wheel ⇒ per-platform CI. Once the wheel is platform-specific, use cibuildwheel in CI to build/test the matrix (Linux manylinux/musllinux, macOS x86_64+arm64, Windows), plus one sdist. Publish the abi3 wheel(s) if you compiled against the Limited API. Test each built wheel in a clean env — a wheel that imports on the build host but not a fresh one is the classic missing-.so bug.

Testing native extensions

The unit/integration split for accelerators lives in the testing skill ("Native extensions"): behaviour tests are unit and parametrise over the compiled/pure import; the fallback is tested by forcing the C import to fail; the platform × Python matrix build is integration-only (cibuildwheel). Native-specific riders on top of that:

• The pure and compiled paths must stay behaviourally identical — a divergence surfaced by the shared assertions is the bug, not a test to relax.
• Build with Py_DEBUG in at least one CI leg to catch refcount/leak errors the pure path can't have.
• Smoke-import each built wheel in a clean env — the classic missing-.so/wrong-tag bug passes on the build host and fails everywhere else.

Checklist

• Profiled; the hot path and its call frequency justify native code (written down).
• Interface chosen deliberately (cffi/Cython/nanobind/PyO3/raw C-API) with a one-line rationale.
• Pure-Python fallback exists, is imported via try/except ImportError → pure, logs when it degrades, and is tested identically.
• C-API is 3.12+-clean (no removed Unicode/Py_UNICODE API; high-level API over struct internals; refcounts/errors correct).
• Free-threading opt-in decided and, if declared, shared state is actually locked (#ifdef Py_GIL_DISABLED). If the project supports the free-threaded build it sets requires-python = '>=3.14' and tests the t build; a single-threaded accelerator keeps the normal floor (3.12) — the C floor is not raised for its own sake.
• Wheel-matrix strategy chosen (abi3 via Limited API where feasible).
• hatchling build hook wired up (hatch-cython / custom hatch_build.py / scikit-build-core-meson-python-maturin) — the wheel is genuinely non-pure and contains the .so/.pyd.
• sdist ships the native source; cibuildwheel builds+tests the matrix in CI; built wheels import in a clean env.

References

• nanobind — repo · benchmarks · PyPI
• Behnel, "Cython, pybind11, cffi – which tool should you choose?"
• Hatch — build-hook plugin reference · hatch-cython
• scikit-build-core (+ hatchling plugin), [meson-python], [maturin]
• Free threading — Python HOWTO: C-API extension support · py-free-threading guide
• CPython Stable ABI / Limited API · cibuildwheel