python-performance

name	Python Performance
description	Master Python optimization techniques, profiling, memory management, and high-performance computing
version	2.1.0
sasmp_version	1.3.0
bonded_agent	07-best-practices
bond_type	PRIMARY_BOND
retry_strategy	exponential_backoff
observability	{"logging":true,"metrics":"execution_time_improvement"}

Python Performance Optimization

Overview

Master performance optimization in Python. Learn to profile code, identify bottlenecks, optimize algorithms, manage memory efficiently, and leverage high-performance libraries for compute-intensive tasks.

Learning Objectives

• Profile Python code to identify bottlenecks
• Optimize algorithms and data structures
• Manage memory efficiently
• Use compiled extensions (Cython, NumPy)
• Implement caching strategies
• Parallelize CPU-bound operations
• Benchmark and measure improvements

Core Topics

1. Profiling & Benchmarking

• timeit module for micro-benchmarks
• cProfile for function-level profiling
• line_profiler for line-by-line analysis
• memory_profiler for memory usage
• py-spy for production profiling
• Flame graphs and visualization

Code Example:

import timeit
import cProfile
import pstats

# 1. timeit for micro-benchmarks
def list_comprehension():
    return [x**2 for x in range(1000)]

def map_function():
    return list(map(lambda x: x**2, range(1000)))

# Compare performance
time_lc = timeit.timeit(list_comprehension, number=10000)
time_map = timeit.timeit(map_function, number=10000)
print(f"List comprehension: {time_lc:.4f}s")
print(f"Map function: {time_map:.4f}s")

# 2. cProfile for function profiling
def process_data():
    data = []
    for i in range(100000):
        data.append(i ** 2)
    return sum(data)

profiler = cProfile.Profile()
profiler.enable()
result = process_data()
profiler.disable()

stats = pstats.Stats(profiler)
stats.sort_stats('cumulative')
stats.print_stats(10)

# 3. Line profiling (requires line_profiler package)
# @profile decorator (add manually for line_profiler)
def slow_function():
    total = 
     i  ():
        total += i ** 
     total




 memory_profiler  profile


 ():
    large_list = [i  i  ()]
    large_dict = {i: i**  i  ()}
     (large_list) + (large_dict)

2. Algorithm & Data Structure Optimization

• Choosing efficient data structures
• Time complexity analysis
• Generator expressions vs lists
• Set operations for lookups
• Deque for queue operations
• Bisect for sorted lists

Code Example:

import bisect
from collections import deque, Counter, defaultdict
import time

# 1. List vs Set for membership testing
# Bad: O(n) lookup
def find_in_list(items, target):
    return target in items  # Linear search

# Good: O(1) lookup
def find_in_set(items, target):
    items_set = set(items)
    return target in items_set

items = list(range(100000))
# List: 0.001s, Set: 0.000001s (1000x faster!)

# 2. Generator expressions for memory efficiency
# Bad: Creates entire list in memory
squares_list = [x**2 for x in range(1000000)]  # ~4MB

# Good: Generates on-demand
squares_gen = (x**2 for x in range(1000000))   # ~128 bytes

# 3. Deque for efficient queue operations
# Bad: O(n) pop from beginning
queue_list = list(range(10000))
queue_list.pop(0)  # Slow

# Good: O(1) pop from both ends
queue_deque = deque(range(10000))
queue_deque.popleft()  



sorted_list = []
 i  [, , , , ]:
    sorted_list.append(i)
    sorted_list.sort()


sorted_list = []
 i  [, , , , ]:
    bisect.insort(sorted_list, i)



word_count = {}
 word  words:
     word  word_count:
        word_count[word] += 
    :
        word_count[word] = 


word_count = Counter(words)
most_common = word_count.most_common()

3. Memory Management

• Memory allocation and garbage collection
• Object pooling
• Slots for memory-efficient classes
• Reference counting
• Weak references
• Memory leaks detection

Code Example:

import gc
import sys
from weakref import WeakValueDictionary

# 1. __slots__ for memory-efficient classes
# Bad: Regular class (56 bytes per instance)
class RegularPoint:
    def __init__(self, x, y):
        self.x = x
        self.y = y

# Good: Slots class (32 bytes per instance - 43% smaller!)
class SlottedPoint:
    __slots__ = ['x', 'y']

    def __init__(self, x, y):
        self.x = x
        self.y = y

print(sys.getsizeof(RegularPoint(1, 2)))  # 56 bytes
print(sys.getsizeof(SlottedPoint(1, 2)))  # 32 bytes

# 2. Object pooling for expensive objects
class ObjectPool:
    def __init__(self, factory, max_size=10):
        self.factory = factory
        self.max_size = max_size
        self.pool = []

    def acquire(self):
        if self.pool:
            return .pool.pop()
         .factory()

     ():
         (.pool) < .max_size:
            .pool.append(obj)


db_pool = ObjectPool(: DatabaseConnection(), max_size=)
conn = db_pool.acquire()

db_pool.release(conn)


 :
     ():
        ._cache = WeakValueDictionary()

     ():
         ._cache.get(key)

     ():
        ._cache[key] = value


 ():
     batch  large_data:
        process_batch(batch)
        
        gc.collect()


 :
     ():
        .resource = allocate_resource()
         .resource

     ():
        .resource.cleanup()
         

4. High-Performance Computing

• NumPy vectorization
• Numba JIT compilation
• Cython for C extensions
• Multiprocessing for parallelism
• Concurrent.futures
• Performance comparison

Code Example:

import numpy as np
from numba import jit
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor

# 1. NumPy vectorization
# Bad: Python loops (slow)
def python_sum(n):
    total = 0
    for i in range(n):
        total += i ** 2
    return total

# Good: NumPy vectorization (100x faster!)
def numpy_sum(n):
    arr = np.arange(n)
    return np.sum(arr ** 2)

# Benchmark: python_sum(1000000) = 0.15s
#           numpy_sum(1000000)  = 0.002s

# 2. Numba JIT compilation
@jit(nopython=True)  # Compile to machine code
def fast_function(n):
    total = 0
    for i in range(n):
        total += i ** 2
    return total

# First call: compilation + execution
# Subsequent calls: 50x faster than pure Python!

# 3. Multiprocessing for CPU-bound tasks
def cpu_intensive_task(n):
    return (i * i  i  (n))


result = cpu_intensive_task()


 ProcessPoolExecutor(max_workers=)  executor:
    ranges = [, , , ]
    results = executor.(cpu_intensive_task, ranges)
    total = (results)




 functools  lru_cache


 ():
     n < :
         n
     fibonacci(n-) + fibonacci(n-)





 ():
    
    subset = data[:]

    
    view = (data)[:]

Hands-On Practice

Project 1: Performance Profiler

Build a comprehensive profiling tool.

Requirements:

• CPU profiling with cProfile
• Memory profiling
• Line-by-line analysis
• Visualization (flame graphs)
• HTML report generation
• Bottleneck identification

Key Skills: Profiling tools, visualization, analysis

Project 2: Data Processing Pipeline

Optimize data processing pipeline.

Requirements:

• Load large CSV files (1GB+)
• Transform and clean data
• Aggregate statistics
• Compare Python/NumPy/Pandas approaches
• Measure memory usage
• Optimize to <2GB RAM

Key Skills: NumPy, memory optimization, benchmarking

Project 3: Parallel Computing

Implement parallel algorithms.

Requirements:

• Matrix multiplication
• Image processing
• Monte Carlo simulation
• Compare threading/multiprocessing/asyncio
• Measure speedup
• Handle shared state

Key Skills: Parallelism, performance measurement

Assessment Criteria

• Profile code to identify bottlenecks
• Choose appropriate data structures
• Optimize algorithms for time complexity
• Manage memory efficiently
• Use vectorization where applicable
• Implement effective caching
• Parallelize CPU-bound operations

Resources

Official Documentation

• Python Performance Tips - Official tips
• NumPy Docs - NumPy documentation
• Numba Docs - JIT compilation

Learning Platforms

• High Performance Python - O'Reilly book
• Python Performance - Real Python guide
• Optimizing Python - PyCon talks

Tools

• cProfile - CPU profiling
• memory_profiler - Memory profiling
• py-spy - Sampling profiler
• Scalene - CPU/GPU/memory profiler

Next Steps

After mastering Python performance, explore:

• Cython - C extensions for Python
• PyPy - Alternative Python interpreter
• Dask - Parallel computing library
• CUDA - GPU programming with Python