portfolio-optimization

name

portfolio-optimization

description

Guidance for implementing high-performance portfolio optimization using Python C extensions. This skill applies when tasks require optimizing financial computations (matrix operations, covariance calculations, portfolio risk metrics) by implementing C extensions for Python. Use when performance speedup requirements exist (e.g., 1.2x or greater) and the task involves numerical computations on large datasets (thousands of assets).

Portfolio Optimization

Overview

This skill provides guidance for implementing high-performance portfolio optimization algorithms using Python C extensions. It covers the workflow for creating C extensions that interface with NumPy arrays, proper verification strategies, and common pitfalls to avoid when optimizing numerical computations.

When to Apply This Skill

Apply this skill when:

• Implementing portfolio risk calculations (variance, volatility, Sharpe ratio)
• Optimizing matrix-vector operations for large asset portfolios
• Creating C extensions for Python numerical code
• Performance requirements specify speedup ratios (e.g., >= 1.2x)
• Working with covariance matrices and portfolio weights

Recommended Workflow

Phase 1: Codebase Understanding

Before writing any code:

1. Read all relevant source files completely - Understand the baseline implementation, data structures, and expected interfaces
2. Identify the mathematical operations - Common operations include:
   • Matrix-vector multiplication (covariance matrix times weights)
   • Dot products (weights times returns)
   • Square root operations (for volatility from variance)
3. Understand the test suite - Know what correctness tolerances are expected (e.g., 1e-10) and what performance benchmarks must be met
4. Document the input/output contracts - Array shapes, data types (typically float64), and return value specifications

Phase 2: Implementation Planning

Consider these factors before implementation:

1. Why C provides speedup:

   • Eliminates Python interpreter overhead
   • Enables direct memory access without bounds checking
   • Allows compiler optimizations (vectorization, loop unrolling)
   • Reduces temporary array allocations

2. Design decisions to make:

   • Whether to use NumPy C API for zero-copy array access
   • Memory layout assumptions (C-contiguous vs Fortran-contiguous)
   • Error handling strategy for type mismatches and dimension errors

3. Potential algorithmic optimizations:

   • Cache-friendly memory access patterns (row-major iteration for C arrays)
   • SIMD vectorization opportunities
   • Minimizing Python-to-C data conversion overhead

Phase 3: C Extension Implementation

When implementing the C extension:

1. Include proper headers:

   • Python.h (must be first)
   • numpy/arrayobject.h for NumPy array access

2. Initialize NumPy in the module init function:

   • Call import_array() to initialize NumPy C API

3. Use NumPy C API for array access:

   • PyArray_DATA() for getting data pointer
   • PyArray_DIM() for dimensions
   • PyArray_STRIDE() for memory strides
   • Check PyArray_IS_C_CONTIGUOUS() for memory layout

4. Implement robust error handling:

   • Validate array dimensions match expected shapes
   • Check data types (expect NPY_FLOAT64 for double precision)
   • Handle non-contiguous arrays (either reject or handle strides)
   • Set appropriate Python exceptions on error

Phase 4: Python Wrapper Implementation

Create a Python module that:

1. Imports the C extension module
2. Provides a clean interface matching the baseline API
3. Handles any necessary array preparation (ensuring contiguity)
4. Documents the interface clearly

Phase 5: Verification Strategy

Critical: Verify every change completely

1. After editing files, re-read them - Confirm edits were applied correctly, especially for multi-line changes

2. Test incrementally:

   • Build the C extension first and verify it compiles
   • Test individual functions before running full benchmarks
   • Use small test cases for correctness verification before scaling up

3. Correctness verification:

   • Compare outputs against baseline implementation
   • Use appropriate numerical tolerances (typically 1e-10 for double precision)
   • Test with known inputs where expected outputs can be calculated manually

4. Performance verification:

   • Run benchmarks with representative data sizes
   • Verify speedup meets requirements across different portfolio sizes
   • Test edge cases: small portfolios (n=1, n=10), large portfolios (n=5000+)

Edge Cases to Handle

Ensure the implementation addresses:

1. Empty portfolios (n=0) - Return appropriate default or error
2. Single-asset portfolios (n=1) - Degenerate case for covariance
3. Dimension mismatches - Weights vector length vs covariance matrix dimensions
4. Invalid inputs:
   • Non-square covariance matrices
   • NaN or infinity values in inputs
   • Negative variance (mathematically invalid)
5. Memory considerations:
   • Non-contiguous NumPy arrays
   • Memory allocation failures in C code
   • Large portfolios that may stress memory

Common Pitfalls to Avoid

Code Completeness

• Never truncate code in edit operations - always provide complete implementations
• Verify file contents after editing to confirm changes applied correctly
• Document all design choices explicitly

Testing Approach

• Avoid going directly from implementation to full benchmark testing
• Test each function individually before integration testing
• Do not rely solely on "tests pass" for validation - understand why they pass

C Extension Specific

• Always check NumPy array types before accessing data
• Handle reference counting properly to avoid memory leaks
• Initialize NumPy API with import_array() in module init
• Use PyErr_SetString() to set exceptions on errors

Performance Validation

• Verify speedup is consistent across different input sizes
• Profile if further optimizations might be needed
• Consider the overhead of Python-to-C transitions for small inputs

Build and Test Commands

Typical workflow commands:

# Build the C extension
python setup.py build_ext --inplace

# Run correctness tests
python -c "from portfolio_optimized import *; # test calls"

# Run benchmark
python benchmark.py

# Run full test suite
pytest test_portfolio.py -v

Verification Checklist

Before considering the task complete:

• All source files read and understood
• C extension compiles without warnings
• Individual functions tested for correctness
• Numerical results match baseline within tolerance
• Performance meets speedup requirements
• Edge cases explicitly tested or handled
• Error handling implemented for invalid inputs
• File contents verified after all edits
• No memory leaks in C code (proper reference counting)