build-compiled-extensions

name	build-compiled-extensions
description	How to build C/C++/Cython/Fortran extensions and frameworks like Caffe, OpenCV, protobuf-dependent projects from source.

Building Compiled Extensions & Frameworks

Pre-flight

pip install setuptools wheel cython 2>/dev/null
apt-get install -y build-essential python3-dev 2>/dev/null

Cython extension workflow

cython file.pyx   # Cythonize to .c first
python setup.py build_ext --inplace 2>&1
# Manual fallback:
gcc -shared -fPIC -O2 $(python3-config --includes) -o mod$(python3-config --extension-suffix) mod.c

Building large C++ frameworks (Caffe, etc.)

1. Check protobuf version: protoc --version and dpkg -l | grep protobuf
2. Fix API incompatibilities before building (see debug-and-fix skill)
3. Use make -j$(nproc) but capture errors: make 2>&1 | tee build.log
4. For Caffe specifically:
   • Set CPU_ONLY := 1 in Makefile.config for non-GPU environments
   • Fix protobuf SetTotalBytesLimit calls for newer protobuf
   • Use make all -j$(nproc) && make test && make runtest

When pip install times out

1. Install dependencies individually first: pip install numpy scipy
2. Build extensions only: python setup.py build_ext --inplace
3. Or: pip install --no-deps -e .

Common pitfalls

• Missing numpy headers: pip install numpy before building numpy-dependent C extensions
• Import from wrong dir: Test from /tmp, not the source directory
• Protobuf mismatch: Check version and fix API calls before compiling

Verification

find . -name "*.so" -newer setup.py
cd /tmp && python -c "import mypackage; print('OK')"