portfolio-optimization

> **Skill:** Portfolio Optimization | **Domain:** trading | **Category:** risk | **Level:** advanced > **Tags:** `trading`, `risk`, `portfolio`, `markowitz`, `hrp`, `risk-parity`, `optimization` --- ## Portfolio Optimization Skill ### Overview Complete modern portfolio construction toolkit. Implements five major allocation frameworks — Markowitz MVO, Equal Risk Contribution, Black-Litterman, Hierarchical Risk Parity, and Kelly Criterion — plus robust covariance estimation, tail risk measurement, and performance attribution. ### Python Module `xtrading/skills/portfolio_optimization.py` ### Stack - **scipy.optimize.minimize** — SLSQP constrained optimisation for MVO and ERC - **scipy.cluster.hierarchy** — Ward linkage clustering for HRP - **scipy.spatial.distance** — Condensed distance matrix for HRP - **numpy** — Matrix algebra, eigenvalue decomposition - **pandas** — Returns DataFrames, weight Series --- ## 1. Covariance Estimator — Robust Matrix Estimation ```python import pandas as pd from xtrading.skills.portfolio_optimization import CovarianceEstimator # returns: DataFrame of daily returns (T × N) returns = pd.DataFrame(...) # columns = asset names # Standard sample covariance (noisy for small T) cov_sample = CovarianceEstimator.sample(returns) # Ledoit-Wolf analytical shrinkage (recommended for N > 10 or T < 3N) cov_lw = CovarianceEstimator.ledoit_wolf(returns) # Exponentially weighted (upweights recent data, halflife = 60 days) cov_ewm = CovarianceEstimator.exponential(returns, halflife=60) # Constant correlation shrinkage target cov_cc = CovarianceEstimator.constant_correlation(returns) # Guarantee positive definiteness (clip negative eigenvalues) cov_pd = CovarianceEstimator.ensure_positive_definite(cov_lw, epsilon=1e-8) ``` ### When to Use Which Estimator | Method | Best For | Limitation | |--------|----------|------------| | `sample` | Large T (T >> 5N) | Noisy; singular if T < N | | `ledoit_wolf` | General use, N ≤ 50 | Shrinks toward identity | | `exponential` | Regime-aware, recent data | Requires tuning halflife | | `constant_correlation` | Stable correlation structure | Assumes constant ρ | --- ## 2. MeanVarianceOptimiser — Markowitz Efficient Frontier ```python from xtrading.skills.portfolio_optimization import MeanVarianceOptimiser opt = MeanVarianceOptimiser( returns=returns, # daily returns DataFrame cov_method="ledoit_wolf", # covariance estimator risk_free_rate=0.05, # annualised risk-free rate allow_short=False, # long-only (set True for long/short) ) # Maximum Sharpe Ratio (tangency) portfolio max_s = opt.max_sharpe() # max_s.method → "Max Sharpe" # max_s.weights → {"EURUSD": 0.32, "XAUUSD": 0.28, ...} # max_s.expected_return → 0.1842 (18.4% annualised) # max_s.expected_volatility → 0.0921 # max_s.sharpe_ratio → 1.457 # max_s.diversification_ratio → 1.23 # max_s.effective_n → 3.8 (1 / HHI) # Global Minimum Variance portfolio min_v = opt.min_variance() # Target a specific annual return (min variance for that return) port_10 = opt.target_return(target=0.10) # 10% annualised return # Maximum Diversification portfolio max_d = opt.max_diversification() # Full efficient frontier frontier = opt.efficient_frontier(n_points=50) # DataFrame: columns = ["return", "volatility", "sharpe"] # Rebalancing trades from current allocation current_w = {"EURUSD": 0.50, "XAUUSD": 0.30, "GBPUSD": 0.20} trades_df = max_s.rebalance_trades(current_w, portfolio_value=100_000) # asset current_weight target_weight delta_weight trade_value action # EURUSD 0.50 0.32 -0.18 -18000.0 sell # XAUUSD 0.30 0.28 -0.02 -2000.0 sell # GBPUSD 0.20 0.40 +0.20 +20000.0 buy ``` ### PortfolioWeights Fields | Field | Type | Description | |-------|------|-------------| | `method` | `str` | Optimisation method name | | `weights` | `dict[str, float]` | Asset → weight (sums to 1.0) | | `expected_return` | `float` | Annualised expected return | | `expected_volatility` | `float` | Annualised volatility | | `sharpe_ratio` | `float` | (Return − rf) / Volatility | | `diversification_ratio` | `float` | Weighted avg vol / portfolio vol | | `effective_n` | `float` | 1 / HHI (effective number of bets) | | `metadata` | `dict` | Method-specific extra data | --- ## 3. RiskParityOptimiser — Equal Risk Contribution ```python from xtrading.skills.portfolio_optimization import RiskParityOptimiser import numpy as np # Equal risk contribution (each asset = same % of portfolio variance) erc = RiskParityOptimiser( returns=returns, cov_method="ledoit_wolf", ) port = erc.optimise() # port.method → "Equal Risk Contribution (Risk Parity)" # port.metadata["risk_contributions"] → {"EURUSD": 0.25, "XAUUSD": 0.25, ...} # port.metadata["erc_convergence"] → True # Custom risk budgets (e.g. 60/40 risk allocation) budgets = np.array([0.60, 0.40]) erc_custom = RiskParityOptimiser(returns[["SPY", "TLT"]], risk_budgets=budgets) port_custom = erc_custom.optimise() ``` **Key Property**: Risk parity does NOT require return estimates. It only uses the covariance matrix, making it robust to estimation error in expected returns. --- ## 4. BlackLittermanModel — Bayesian Return Integration ```python import numpy as np import pandas as pd from xtrading.skills.portfolio_optimization import BlackLittermanModel # Market-cap weights (or any prior/benchmark weights) market_caps = pd.Series({ "EURUSD": 1_000_000, "XAUUSD": 500_000, "GBPUSD": 750_000, "USDJPY": 250_000, }) bl = BlackLittermanModel( market_caps=market_caps, returns=returns, risk_free=0.05, tau=0.05, # prior uncertainty (0.025–0.10) cov_method="ledoit_wolf", ) # Add investor views # View 1 (absolute): EURUSD will return 12% next year # View 2 (relative): XAUUSD will outperform GBPUSD by 5% P = np.array([ [1, 0, 0, 0], # View 1: long EURUSD [0, 1, -1, 0], # View 2: long XAUUSD, short GBPUSD ]) Q = np.array([0.12, 0.05]) # 12% and 5% view returns result = bl.add_views(P=P, Q=Q) # { # "prior_returns": {"EURUSD": 0.0821, "XAUUSD": 0.0654, ...}, # "posterior_returns": {"EURUSD": 0.0965, "XAUUSD": 0.0721, ...}, # "bl_weights": {"EURUSD": 0.3410, "XAUUSD": 0.2850, ...}, # "return_change": {"EURUSD": +0.0144, "XAUUSD": +0.0067, ...}, # "n_views": 2 # } ``` **Black-Litterman Formula**: ``` π = δ · Σ · w_mkt (equilibrium returns) Ω = τ · P · Σ · P' (view uncertainty, diagonal) posterior μ = [(τΣ)⁻¹ + P'Ω⁻¹P]⁻¹ · [(τΣ)⁻¹π + P'Ω⁻¹Q] ``` --- ## 5. HRPOptimiser — Hierarchical Risk Parity ```python from xtrading.skills.portfolio_optimization import HRPOptimiser hrp = HRPOptimiser(returns=returns) port = hrp.optimise() # port.method → "Hierarchical Risk Parity (HRP)" # port.weights → {"EURUSD": 0.2841, "XAUUSD": 0.3102, ...} # port.metadata["cluster_order"] → ["XAUUSD", "EURUSD", "USDJPY", "GBPUSD"] ``` ### HRP Algorithm Steps ``` Step 1: Distance matrix d_ij = √(0.5 × (1 − ρ_ij)) Step 2: Ward clustering Hierarchical linkage on correlation distance Step 3: Quasi-diagonalise Order assets so similar assets are adjacent Step 4: Recursive bisection Allocate weight proportionally to inverse variance: α = 1 − Var(left_cluster) / [Var(left) + Var(right)] left_weights *= α right_weights *= (1 − α) ``` **HRP Advantages over MVO**: - No matrix inversion → stable for large N - Respects correlation structure → less concentrated - No return estimates needed - Out-of-sample outperforms MVO (Lopez de Prado 2016) --- ## 6. KellyCriterion — Optimal Position Sizing ```python from xtrading.skills.portfolio_optimization import KellyCriterion # Discrete Kelly (binary bet / single trade) k = KellyCriterion.discrete( win_probability=0.60, win_payoff=1.5, # 1.5R on win loss_payoff=1.0, # 1.0R on loss ) # { # "full_kelly": 0.2667, # "half_kelly": 0.1333, # "quarter_kelly": 0.0667, # "kelly_pct": 26.67, # "edge": 0.4000, # "recommendation": "MODERATE BET" # } # Continuous Kelly (Gaussian strategy returns) k2 = KellyCriterion.continuous( mu=0.20, # 20% annualised expected return sigma=0.15, # 15% annualised volatility risk_free=0.05, ) # { # "full_kelly": 6.667, # leverage ratio (use fractional!) # "half_kelly": 3.333, # "growth_rate_full": 0.450, # "growth_rate_half": 0.431, # "sharpe_ratio": 1.00, # "recommendation": "AGGRESSIVE" # } # Multi-asset Kelly (full Kelly portfolio) import pandas as pd, numpy as np mu = pd.Series({"A": 0.12, "B": 0.08, "C": 0.15}) cov = pd.DataFrame([[0.04, 0.01, 0.02], [0.01, 0.02, 0.01], [0.02, 0.01, 0.06]], index=mu.index, columns=mu.index) k3 = KellyCriterion.multi_asset(mu, cov, risk_free=0.05) # { # "full_kelly_weights": {"A": 3.2, "B": 1.4, "C": 2.8}, # leveraged # "normalised_kelly_weights":{"A": 0.43, "B": 0.19, "C": 0.38}, # long-only # "leverage_ratio": 7.4 # } ``` ### Kelly Sizing Rules | Full Kelly | Half Kelly | Recommended Use | |------------|------------|-----------------| | < 0.02 | < 0.01 | No edge — skip trade | | 0.02–0.10 | 0.01–0.05 | Small bet (conservative) | | 0.10–0.20 | 0.05–0.10 | Moderate bet | | > 0.20 | > 0.10 | Strong edge — still use half-Kelly | **Rule**: Always trade **half-Kelly or less** in practice. Full Kelly maximises long-run growth but has high variance and frequent large drawdowns. --- ## 7. TailRiskEstimator — VaR and CVaR ```python import pandas as pd from xtrading.skills.portfolio_optimization import TailRiskEstimator # Historical simulation VaR (most conservative, data-driven) hist = TailRiskEstimator.historical_var( returns=portfolio_returns, # pd.Series of daily returns confidence=0.95, horizon_days=1, ) # { # "method": "historical", "confidence": 0.95, # "var": 0.0182, "var_pct": 1.82, # "cvar": 0.0251, "cvar_pct": 2.51, # "worst_return": -0.0487, "n_observations": 252 # } # Parametric (Gaussian) VaR — fast, assumes normality para = TailRiskEstimator.parametric_var( mu=0.0004, # daily mean return sigma=0.012, # daily volatility confidence=0.99, horizon_days=10, # 10-day regulatory horizon ) # Monte Carlo VaR (GBM paths, most flexible) mc = TailRiskEstimator.monte_carlo_var( mu=0.0004, sigma=0.012, confidence=0.95, horizon_days=1, n_paths=100_000, seed=42, ) # Cornish-Fisher VaR (adjusted for fat tails) cf = TailRiskEstimator.cornish_fisher_var( returns=portfolio_returns, confidence=0.95, ) # { # "method": "cornish_fisher", # "var": 0.0209, "var_pct": 2.09,