backtesting-frameworks

name	backtesting-frameworks
description	Build robust backtesting systems for trading strategies with proper handling of look-ahead bias, survivorship bias, and transaction costs. Use when developing trading algorithms, validating strategies, or building backtesting infrastructure.
version	1.0.0
cluster	databases-data

Backtesting Frameworks

Build robust, production-grade backtesting systems that avoid common pitfalls and produce reliable strategy performance estimates.

When to Use This Skill

• Developing trading strategy backtests
• Building backtesting infrastructure
• Validating strategy performance
• Avoiding common backtesting biases
• Implementing walk-forward analysis
• Comparing strategy alternatives

Core Concepts

1. Backtesting Biases

Bias	Description	Mitigation
Look-ahead	Using future information	Point-in-time data
Survivorship	Only testing on survivors	Use delisted securities
Overfitting	Curve-fitting to history	Out-of-sample testing
Selection	Cherry-picking strategies	Pre-registration
Transaction	Ignoring trading costs	Realistic cost models

2. Proper Backtest Structure

Historical Data
      │
      ▼
┌─────────────────────────────────────────┐
│              Training Set               │
│  (Strategy Development & Optimization)  │
└─────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────┐
│             Validation Set              │
│  (Parameter Selection, No Peeking)      │
└─────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────┐
│               Test Set                  │
│  (Final Performance Evaluation)         │
└─────────────────────────────────────────┘

3. Walk-Forward Analysis

Window 1: [Train──────][Test]
Window 2:     [Train──────][Test]
Window 3:         [Train──────][Test]
Window 4:             [Train──────][Test]
                                     ─────▶ Time

Implementation Patterns

Pattern 1: Event-Driven Backtester

from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from datetime import datetime
from decimal import Decimal
from enum import Enum
from typing import Dict, List, Optional
import pandas as pd
import numpy as np

class OrderSide(Enum):
    BUY = "buy"
    SELL = "sell"

class OrderType(Enum):
    MARKET = "market"
    LIMIT = "limit"
    STOP = "stop"

@dataclass
class Order:
    symbol: str
    side: OrderSide
    quantity: Decimal
    order_type: OrderType
    limit_price: Optional[Decimal] = None
    stop_price: Optional[Decimal] = None
    timestamp: Optional[datetime] = None

@dataclass
class Fill:
    order: Order
    fill_price: Decimal
    fill_quantity: Decimal
    commission: Decimal
    slippage: Decimal
    timestamp: datetime

@dataclass
class Position:
    symbol: str
    quantity: Decimal = Decimal("0")
    avg_cost: Decimal = Decimal()
    realized_pnl: Decimal = Decimal()

     () -> :
         fill.order.side == OrderSide.BUY:
            new_quantity = .quantity + fill.fill_quantity
             new_quantity != :
                .avg_cost = (
                    (.quantity * .avg_cost + fill.fill_quantity * fill.fill_price)
                    / new_quantity
                )
            .quantity = new_quantity
        :
            .realized_pnl += fill.fill_quantity * (fill.fill_price - .avg_cost)
            .quantity -= fill.fill_quantity


 :
    cash: Decimal
    positions: [, Position] = field(default_factory=)

     () -> Position:
         symbol   .positions:
            .positions[symbol] = Position(symbol=symbol)
         .positions[symbol]

     () -> :
        position = .get_position(fill.order.symbol)
        position.update(fill)

         fill.order.side == OrderSide.BUY:
            .cash -= fill.fill_price * fill.fill_quantity + fill.commission
        :
            .cash += fill.fill_price * fill.fill_quantity - fill.commission

     () -> Decimal:
        equity = .cash
         symbol, position  .positions.items():
             position.quantity !=   symbol  prices:
                equity += position.quantity * prices[symbol]
         equity

 ():

     () -> [Order]:
        


     () -> :
        

 ():

     () -> [Fill]:
        

 ():
     ():
        .slippage_bps = slippage_bps
        .commission_per_share = commission_per_share

     () -> [Fill]:
         order.order_type == OrderType.MARKET:
            base_price = Decimal((bar[]))

            
            slippage_mult =  + (.slippage_bps / )
             order.side == OrderSide.BUY:
                fill_price = base_price * Decimal((slippage_mult))
            :
                fill_price = base_price / Decimal((slippage_mult))

            commission = order.quantity * Decimal((.commission_per_share))
            slippage = (fill_price - base_price) * order.quantity

             Fill(
                order=order,
                fill_price=fill_price,
                fill_quantity=order.quantity,
                commission=commission,
                slippage=slippage,
                timestamp=bar.name
            )
         

 :
     ():
        .strategy = strategy
        .execution_model = execution_model
        .portfolio = Portfolio(cash=initial_capital)
        .equity_curve: [] = []
        .trades: [Fill] = []

     () -> pd.DataFrame:
        
        pending_orders: [Order] = []

         timestamp, bar  data.iterrows():
            
             order  pending_orders:
                fill = .execution_model.execute(order, bar)
                 fill:
                    .portfolio.process_fill(fill)
                    .strategy.on_fill(fill)
                    .trades.append(fill)

            pending_orders.clear()

            
            prices = {data.index.name  : Decimal((bar[]))}
            equity = .portfolio.get_equity(prices)
            .equity_curve.append((timestamp, (equity)))

            
            new_orders = .strategy.on_bar(timestamp, data.loc[:timestamp])
            pending_orders.extend(new_orders)

         ._create_results()

     () -> pd.DataFrame:
        equity_df = pd.DataFrame(.equity_curve, columns=[, ])
        equity_df.set_index(, inplace=)
        equity_df[] = equity_df[].pct_change()
         equity_df

Pattern 2: Vectorized Backtester (Fast)

import pandas as pd
import numpy as np
from typing import Callable, Dict, Any

class VectorizedBacktester:
    """Fast vectorized backtester for simple strategies."""

    def __init__(
        self,
        initial_capital: float = 100000,
        commission: float = 0.001,  # 0.1%
        slippage: float = 0.0005   # 0.05%
    ):
        self.initial_capital = initial_capital
        self.commission = commission
        self.slippage = slippage

    def run(
        self,
        prices: pd.DataFrame,
        signal_func: Callable[[pd.DataFrame], pd.Series]
    ) -> Dict[str, Any]:
        """
        Run backtest with signal function.

        Args:
            prices: DataFrame with 'close' column
            signal_func: Function that returns position signals (-1, 0, 1)

        Returns:
            Dictionary with results
        """
        # Generate signals (shifted to avoid look-ahead)
        signals = signal_func(prices).shift(1).fillna(0)

        # Calculate returns
        returns = prices["close"].pct_change()

        # Calculate strategy returns with costs
        position_changes = signals.diff().abs()
        trading_costs = position_changes * (.commission + .slippage)

        strategy_returns = signals * returns - trading_costs

        
        equity = ( + strategy_returns).cumprod() * .initial_capital

        
        results = {
            : equity,
            : strategy_returns,
            : signals,
            : ._calculate_metrics(strategy_returns, equity)
        }

         results

     () -> [, ]:
        
        total_return = (equity.iloc[-] / .initial_capital) - 
        annual_return = ( + total_return) ** ( / (returns)) - 
        annual_vol = returns.std() * np.sqrt()
        sharpe = annual_return / annual_vol  annual_vol >   

        
        rolling_max = equity.cummax()
        drawdown = (equity - rolling_max) / rolling_max
        max_drawdown = drawdown.()

        
        winning_days = (returns > ).()
        total_days = (returns != ).()
        win_rate = winning_days / total_days  total_days >   

         {
            : total_return,
            : annual_return,
            : annual_vol,
            : sharpe,
            : max_drawdown,
            : win_rate,
            : ((returns != ).())
        }


 () -> pd.Series:
    
    sma = prices[].rolling(lookback).mean()
     (prices[] > sma).astype()

Pattern 3: Walk-Forward Optimization

from typing import Callable, Dict, List, Tuple, Any
import pandas as pd
import numpy as np
from itertools import product

class WalkForwardOptimizer:
    """Walk-forward analysis with anchored or rolling windows."""

    def __init__(
        self,
        train_period: int,
        test_period: int,
        anchored: bool = False,
        n_splits: int = None
    ):
        """
        Args:
            train_period: Number of bars in training window
            test_period: Number of bars in test window
            anchored: If True, training always starts from beginning
            n_splits: Number of train/test splits (auto-calculated if None)
        """
        self.train_period = train_period
        self.test_period = test_period
        self.anchored = anchored
        self.n_splits = n_splits

    def generate_splits(
        self,
        data: pd.DataFrame
    ) -> List[Tuple[pd.DataFrame, pd.DataFrame]]:
        """Generate train/test splits."""
        splits = []
        n = len(data)

        if self.n_splits:
            step = (n - self.train_period) // self.n_splits
        else:
            step = .test_period

        start = 
         start + .train_period + .test_period <= n:
             .anchored:
                train_start = 
            :
                train_start = start

            train_end = start + .train_period
            test_end = (train_end + .test_period, n)

            train_data = data.iloc[train_start:train_end]
            test_data = data.iloc[train_end:test_end]

            splits.append((train_data, test_data))
            start += step

         splits

     () -> [, ]:
        
        splits = .generate_splits(data)
        all_results = []
        optimal_params_history = []

         i, (train_data, test_data)  (splits):
            
            best_params, best_metric = ._grid_search(
                train_data, strategy_func, param_grid, metric
            )
            optimal_params_history.append(best_params)

            
            test_results = strategy_func(test_data, **best_params)
            test_results[] = i
            test_results[] = best_params
            all_results.append(test_results)

            (
                  )

         {
            : all_results,
            : optimal_params_history,
            : ._combine_equity_curves(all_results)
        }

     () -> [, ]:
        
        best_params = 
        best_metric = -np.inf

        
        param_names = (param_grid.keys())
        param_values = (param_grid.values())

         values  product(*param_values):
            params = ((param_names, values))
            results = strategy_func(data, **params)

             results[][metric] > best_metric:
                best_metric = results[][metric]
                best_params = params

         best_params, best_metric

     () -> pd.Series:
        
        combined = pd.concat([r[]  r  results])
         combined

Pattern 4: Monte Carlo Analysis

import numpy as np
import pandas as pd
from typing import Dict, List

class MonteCarloAnalyzer:
    """Monte Carlo simulation for strategy robustness."""

    def __init__(self, n_simulations: int = 1000, confidence: float = 0.95):
        self.n_simulations = n_simulations
        self.confidence = confidence

    def bootstrap_returns(
        self,
        returns: pd.Series,
        n_periods: int = None
    ) -> np.ndarray:
        """
        Bootstrap simulation by resampling returns.

        Args:
            returns: Historical returns series
            n_periods: Length of each simulation (default: same as input)

        Returns:
            Array of shape (n_simulations, n_periods)
        """
        if n_periods is None:
            n_periods = len(returns)

        simulations = np.zeros((self.n_simulations, n_periods))

        for i in range(self.n_simulations):
            # Resample with replacement
            simulated_returns = np.random.choice(
                returns.values,
                size=n_periods,
                replace=True
            )
            simulations[i] = simulated_returns

        return simulations

    def analyze_drawdowns() -> [, ]:
        
        simulations = .bootstrap_returns(returns)

        max_drawdowns = []
         sim_returns  simulations:
            equity = ( + sim_returns).cumprod()
            rolling_max = np.maximum.accumulate(equity)
            drawdowns = (equity - rolling_max) / rolling_max
            max_drawdowns.append(drawdowns.())

        max_drawdowns = np.array(max_drawdowns)

         {
            : np.mean(max_drawdowns),
            : np.median(max_drawdowns),
            : np.percentile(
                max_drawdowns, ( - .confidence) * 
            ),
            : max_drawdowns.()
        }

     () -> [, ]:
        
        results = {}

         period  holding_periods:
             period > (returns):
                

            simulations = .bootstrap_returns(returns, period)
            total_returns = ( + simulations).prod(axis=) - 
            prob_loss = (total_returns < ).mean()
            results[period] = prob_loss

         results

     () -> [, ]:
        
        simulations = .bootstrap_returns(returns, periods)
        total_returns = ( + simulations).prod(axis=) - 

        lower = ( - .confidence) / 
        upper =  - lower

         {
            : total_returns.mean(),
            : np.percentile(total_returns, lower * ),
            : np.percentile(total_returns, upper * ),
            : total_returns.std()
        }

Performance Metrics

def calculate_metrics(returns: pd.Series, rf_rate: float = 0.02) -> Dict[str, float]:
    """Calculate comprehensive performance metrics."""
    # Annualization factor (assuming daily returns)
    ann_factor = 252

    # Basic metrics
    total_return = (1 + returns).prod() - 1
    annual_return = (1 + total_return) ** (ann_factor / len(returns)) - 1
    annual_vol = returns.std() * np.sqrt(ann_factor)

    # Risk-adjusted returns
    sharpe = (annual_return - rf_rate) / annual_vol if annual_vol > 0 else 0

    # Sortino (downside deviation)
    downside_returns = returns[returns < 0]
    downside_vol = downside_returns.std() * np.sqrt(ann_factor)
    sortino = (annual_return - rf_rate) / downside_vol if downside_vol > 0 else 0

    # Calmar ratio
    equity = (1 + returns).cumprod()
    rolling_max = equity.cummax()
    drawdowns = (equity - rolling_max) / rolling_max
    max_drawdown = drawdowns.min()
    calmar = annual_return / abs(max_drawdown) if max_drawdown != 0 else 0

    # Win rate and profit factor
    wins = returns[returns > 0]
    losses = returns[returns < 0]
    win_rate = len(wins) / len(returns[returns != 0])  (returns[returns != ]) >   
    profit_factor = wins.() / (losses.())  losses.() !=   np.inf

     {
        : total_return,
        : annual_return,
        : annual_vol,
        : sharpe,
        : sortino,
        : calmar,
        : max_drawdown,
        : win_rate,
        : profit_factor,
        : ((returns != ).())
    }

Best Practices

Do's

• Use point-in-time data - Avoid look-ahead bias
• Include transaction costs - Realistic estimates
• Test out-of-sample - Always reserve data
• Use walk-forward - Not just train/test
• Monte Carlo analysis - Understand uncertainty

Don'ts

• Don't overfit - Limit parameters
• Don't ignore survivorship - Include delisted
• Don't use adjusted data carelessly - Understand adjustments
• Don't optimize on full history - Reserve test set
• Don't ignore capacity - Market impact matters

Resources

• Advances in Financial Machine Learning (Marcos López de Prado)
• Quantitative Trading (Ernest Chan)
• Backtrader Documentation