# Portfolio Optimization > Modern portfolio construction: Markowitz MVO, Risk Parity, Black-Litterman, Hierarchical Risk Parity (HRP), Kelly Criterion, VaR/CVaR tail risk, and portfolio analytics. USE FOR: portfolio optimization, Markowitz, efficient frontier, risk parity, Black-Litterman, HRP, Kelly criterion, VaR, CVaR, max Sharpe, minimum variance, covariance, portfolio weights, asset allocation, tail risk, diversification. - Skill: `mahmoud20138/portfolio-optimization` (Agent Skill) - Install (CLI): `npx skillmds@latest add mahmoud20138/portfolio-optimization` - Raw SKILL.md: https://api.skillmd.com/api/skills/mahmoud20138/portfolio-optimization/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Data & Analytics - Author: mahmoud20138 (https://skillmd.com/u/mahmoud20138) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/mahmoud20138/portfolio-optimization --- > **Skill:** Portfolio Optimization | **Domain:** trading | **Category:** risk | **Level:** advanced > **Tags:** `trading`, `risk`, `portfolio`, `markowitz`, `hrp`, `risk-parity`, `optimization` --- ## Portfolio Optimization Skill ### Overview Complete modern portfolio construction toolkit. Implements five major allocation frameworks — Markowitz MVO, Equal Risk Contribution, Black-Litterman, Hierarchical Risk Parity, and Kelly Criterion — plus robust covariance estimation, tail risk measurement, and performance attribution. ### Python Module `xtrading/skills/portfolio_optimization.py` ### Stack - **scipy.optimize.minimize** — SLSQP constrained optimisation for MVO and ERC - **scipy.cluster.hierarchy** — Ward linkage clustering for HRP - **scipy.spatial.distance** — Condensed distance matrix for HRP - **numpy** — Matrix algebra, eigenvalue decomposition - **pandas** — Returns DataFrames, weight Series --- ## 1. Covariance Estimator — Robust Matrix Estimation ```python import pandas as pd from xtrading.skills.portfolio_optimization import CovarianceEstimator # returns: DataFrame of daily returns (T × N) returns = pd.DataFrame(...) # columns = asset names # Standard sample covariance (noisy for small T) cov_sample = CovarianceEstimator.sample(returns) # Ledoit-Wolf analytical shrinkage (recommended for N > 10 or T < 3N) cov_lw = CovarianceEstimator.ledoit_wolf(returns) # Exponentially weighted (upweights recent data, halflife = 60 days) cov_ewm = CovarianceEstimator.exponential(returns, halflife=60) # Constant correlation shrinkage target cov_cc = CovarianceEstimator.constant_correlation(returns) # Guarantee positive definiteness (clip negative eigenvalues) cov_pd = CovarianceEstimator.ensure_positive_definite(cov_lw, epsilon=1e-8) ``` ### When to Use Which Estimator | Method | Best For | Limitation | |--------|----------|------------| | `sample` | Large T (T >> 5N) | Noisy; singular if T < N | | `ledoit_wolf` | General use, N ≤ 50 | Shrinks toward identity | | `exponential` | Regime-aware, recent data | Requires tuning halflife | | `constant_correlation` | Stable correlation structure | Assumes constant ρ | --- ## 2. MeanVarianceOptimiser — Markowitz Efficient Frontier ```python from xtrading.skills.portfolio_optimization import MeanVarianceOptimiser opt = MeanVarianceOptimiser( returns=returns, # daily returns DataFrame cov_method="ledoit_wolf", # covariance estimator risk_free_rate=0.05, # annualised risk-free rate allow_short=False, # long-only (set True for long/short) ) # Maximum Sharpe Ratio (tangency) portfolio max_s = opt.max_sharpe() # max_s.method → "Max Sharpe" # max_s.weights → {"EURUSD": 0.32, "XAUUSD": 0.28, ...} # max_s.expected_return → 0.1842 (18.4% annualised) # max_s.expected_volatility → 0.0921 # max_s.sharpe_ratio → 1.457 # max_s.diversification_ratio → 1.23 # max_s.effective_n → 3.8 (1 / HHI) # Global Minimum Variance portfolio min_v = opt.min_variance() # Target a specific annual return (min variance for that return) port_10 = opt.target_return(target=0.10) # 10% annualised return # Maximum Diversification portfolio max_d = opt.max_diversification() # Full efficient frontier frontier = opt.efficient_frontier(n_points=50) # DataFrame: columns = ["return", "volatility", "sharpe"] # Rebalancing trades from current allocation current_w = {"EURUSD": 0.50, "XAUUSD": 0.30, "GBPUSD": 0.20} trades_df = max_s.rebalance_trades(current_w, portfolio_value=100_000) # asset current_weight target_weight delta_weight trade_value action # EURUSD 0.50 0.32 -0.18 -18000.0 sell # XAUUSD 0.30 0.28 -0.02 -2000.0 sell # GBPUSD 0.20 0.40 +0.20 +20000.0 buy ``` ### PortfolioWeights Fields | Field | Type | Description | |-------|------|-------------| | `method` | `str` | Optimisation method name | | `weights` | `dict[str, float]` | Asset → weight (sums to 1.0) | | `expected_return` | `float` | Annualised expected return | | `expected_volatility` | `float` | Annualised volatility | | `sharpe_ratio` | `float` | (Return − rf) / Volatility | | `diversification_ratio` | `float` | Weighted avg vol / portfolio vol | | `effective_n` | `float` | 1 / HHI (effective number of bets) | | `metadata` | `dict` | Method-specific extra data | --- ## 3. RiskParityOptimiser — Equal Risk Contribution ```python from xtrading.skills.portfolio_optimization import RiskParityOptimiser import numpy as np # Equal risk contribution (each asset = same % of portfolio variance) erc = RiskParityOptimiser( returns=returns, cov_method="ledoit_wolf", ) port = erc.optimise() # port.method → "Equal Risk Contribution (Risk Parity)" # port.metadata["risk_contributions"] → {"EURUSD": 0.25, "XAUUSD": 0.25, ...} # port.metadata["erc_convergence"] → True # Custom risk budgets (e.g. 60/40 risk allocation) budgets = np.array([0.60, 0.40]) erc_custom = RiskParityOptimiser(returns[["SPY", "TLT"]], risk_budgets=budgets) port_custom = erc_custom.optimise() ``` **Key Property**: Risk parity does NOT require return estimates. It only uses the covariance matrix, making it robust to estimation error in expected returns. --- ## 4. BlackLittermanModel — Bayesian Return Integration ```python import numpy as np import pandas as pd from xtrading.skills.portfolio_optimization import BlackLittermanModel # Market-cap weights (or any prior/benchmark weights) market_caps = pd.Series({ "EURUSD": 1_000_000, "XAUUSD": 500_000, "GBPUSD": 750_000, "USDJPY": 250_000, }) bl = BlackLittermanModel( market_caps=market_caps, returns=returns, risk_free=0.05, tau=0.05, # prior uncertainty (0.025–0.10) cov_method="ledoit_wolf", ) # Add investor views # View 1 (absolute): EURUSD will return 12% next year # View 2 (relative): XAUUSD will outperform GBPUSD by 5% P = np.array([ [1, 0, 0, 0], # View 1: long EURUSD [0, 1, -1, 0], # View 2: long XAUUSD, short GBPUSD ]) Q = np.array([0.12, 0.05]) # 12% and 5% view returns result = bl.add_views(P=P, Q=Q) # { # "prior_returns": {"EURUSD": 0.0821, "XAUUSD": 0.0654, ...}, # "posterior_returns": {"EURUSD": 0.0965, "XAUUSD": 0.0721, ...}, # "bl_weights": {"EURUSD": 0.3410, "XAUUSD": 0.2850, ...}, # "return_change": {"EURUSD": +0.0144, "XAUUSD": +0.0067, ...}, # "n_views": 2 # } ``` **Black-Litterman Formula**: ``` π = δ · Σ · w_mkt (equilibrium returns) Ω = τ · P · Σ · P' (view uncertainty, diagonal) posterior μ = [(τΣ)⁻¹ + P'Ω⁻¹P]⁻¹ · [(τΣ)⁻¹π + P'Ω⁻¹Q] ``` --- ## 5. HRPOptimiser — Hierarchical Risk Parity ```python from xtrading.skills.portfolio_optimization import HRPOptimiser hrp = HRPOptimiser(returns=returns) port = hrp.optimise() # port.method → "Hierarchical Risk Parity (HRP)" # port.weights → {"EURUSD": 0.2841, "XAUUSD": 0.3102, ...} # port.metadata["cluster_order"] → ["XAUUSD", "EURUSD", "USDJPY", "GBPUSD"] ``` ### HRP Algorithm Steps ``` Step 1: Distance matrix d_ij = √(0.5 × (1 − ρ_ij)) Step 2: Ward clustering Hierarchical linkage on correlation distance Step 3: Quasi-diagonalise Order assets so similar assets are adjacent Step 4: Recursive bisection Allocate weight proportionally to inverse variance: α = 1 − Var(left_cluster) / [Var(left) + Var(right)] left_weights *= α right_weights *= (1 − α) ``` **HRP Advantages over MVO**: - No matrix inversion → stable for large N - Respects correlation structure → less concentrated - No return estimates needed - Out-of-sample outperforms MVO (Lopez de Prado 2016) --- ## 6. KellyCriterion — Optimal Position Sizing ```python from xtrading.skills.portfolio_optimization import KellyCriterion # Discrete Kelly (binary bet / single trade) k = KellyCriterion.discrete( win_probability=0.60, win_payoff=1.5, # 1.5R on win loss_payoff=1.0, # 1.0R on loss ) # { # "full_kelly": 0.2667, # "half_kelly": 0.1333, # "quarter_kelly": 0.0667, # "kelly_pct": 26.67, # "edge": 0.4000, # "recommendation": "MODERATE BET" # } # Continuous Kelly (Gaussian strategy returns) k2 = KellyCriterion.continuous( mu=0.20, # 20% annualised expected return sigma=0.15, # 15% annualised volatility risk_free=0.05, ) # { # "full_kelly": 6.667, # leverage ratio (use fractional!) # "half_kelly": 3.333, # "growth_rate_full": 0.450, # "growth_rate_half": 0.431, # "sharpe_ratio": 1.00, # "recommendation": "AGGRESSIVE" # } # Multi-asset Kelly (full Kelly portfolio) import pandas as pd, numpy as np mu = pd.Series({"A": 0.12, "B": 0.08, "C": 0.15}) cov = pd.DataFrame([[0.04, 0.01, 0.02], [0.01, 0.02, 0.01], [0.02, 0.01, 0.06]], index=mu.index, columns=mu.index) k3 = KellyCriterion.multi_asset(mu, cov, risk_free=0.05) # { # "full_kelly_weights": {"A": 3.2, "B": 1.4, "C": 2.8}, # leveraged # "normalised_kelly_weights":{"A": 0.43, "B": 0.19, "C": 0.38}, # long-only # "leverage_ratio": 7.4 # } ``` ### Kelly Sizing Rules | Full Kelly | Half Kelly | Recommended Use | |------------|------------|-----------------| | < 0.02 | < 0.01 | No edge — skip trade | | 0.02–0.10 | 0.01–0.05 | Small bet (conservative) | | 0.10–0.20 | 0.05–0.10 | Moderate bet | | > 0.20 | > 0.10 | Strong edge — still use half-Kelly | **Rule**: Always trade **half-Kelly or less** in practice. Full Kelly maximises long-run growth but has high variance and frequent large drawdowns. --- ## 7. TailRiskEstimator — VaR and CVaR ```python import pandas as pd from xtrading.skills.portfolio_optimization import TailRiskEstimator # Historical simulation VaR (most conservative, data-driven) hist = TailRiskEstimator.historical_var( returns=portfolio_returns, # pd.Series of daily returns confidence=0.95, horizon_days=1, ) # { # "method": "historical", "confidence": 0.95, # "var": 0.0182, "var_pct": 1.82, # "cvar": 0.0251, "cvar_pct": 2.51, # "worst_return": -0.0487, "n_observations": 252 # } # Parametric (Gaussian) VaR — fast, assumes normality para = TailRiskEstimator.parametric_var( mu=0.0004, # daily mean return sigma=0.012, # daily volatility confidence=0.99, horizon_days=10, # 10-day regulatory horizon ) # Monte Carlo VaR (GBM paths, most flexible) mc = TailRiskEstimator.monte_carlo_var( mu=0.0004, sigma=0.012, confidence=0.95, horizon_days=1, n_paths=100_000, seed=42, ) # Cornish-Fisher VaR (adjusted for fat tails) cf = TailRiskEstimator.cornish_fisher_var( returns=portfolio_returns, confidence=0.95, ) # { # "method": "cornish_fisher", # "var": 0.0209, "var_pct": 2.09, # "skewness": -0.42, "excess_kurtosis": 1.85, # "z_standard": -1.6449, "z_adjusted": -1.9213, # } ``` ### VaR Method Comparison | Method | Assumption | Best For | |--------|-----------|----------| | Historical | None (empirical) | Stable regimes, ≥ 250 obs | | Parametric | Gaussian returns | Quick estimate, symmetric | | Monte Carlo | GBM dynamics | Custom paths, derivatives | | Cornish-Fisher | Non-Gaussian (skew+kurtosis) | Fat-tailed, skewed returns | **CVaR (Conditional VaR)** = Expected loss *given* that loss exceeds VaR. Always use CVaR alongside VaR — it captures tail severity, not just threshold. --- ## 8. PortfolioAnalytics — Performance Attribution ```python import pandas as pd from xtrading.skills.portfolio_optimization import PortfolioAnalytics analytics = PortfolioAnalytics( portfolio_returns=my_daily_returns, # pd.Series benchmark_returns=spy_daily_returns, # pd.Series risk_free_rate=0.05, ) # Individual statistics ret = analytics.annualised_return() # 0.1842 → 18.4% vol = analytics.annualised_volatility() # 0.0921 → 9.2% sr = analytics.sharpe_ratio() # 1.457 so = analytics.sortino_ratio() # 2.103 ir = analytics.information_ratio() # 0.823 (vs benchmark) dd = analytics.max_drawdown() # -0.082 → -8.2% beta = analytics.beta() # 0.65 alp = analytics.alpha() # 0.042 → +4.2% annualised Jensen's α # Full report in one call report = analytics.full_report() # { # "annualised_return": 0.1842, # "annualised_volatility": 0.0921, # "sharpe_ratio": 1.457, # "sortino_ratio": 2.103, # "information_ratio": 0.823, # "calmar_ratio": 2.247, # "max_drawdown": -0.082, # "max_drawdown_pct": -8.20, # "beta": 0.65, # "alpha_annualised": 0.042, # "var_95_1d": 0.0182, # "cvar_95_1d": 0.0251, # "n_days": 252, # } # Rolling Sharpe (252-day window) rolling_sr = analytics.rolling_sharpe(window=252) # pd.Series indexed by date: "rolling_sharpe" ``` --- ## Full Workflow — Multi-Model Comparison ```python import pandas as pd import numpy as np from xtrading.skills.portfolio_optimization import ( MeanVarianceOptimiser, RiskParityOptimiser, HRPOptimiser, KellyCriterion, TailRiskEstimator, PortfolioAnalytics, ) # 1. Load returns data returns = pd.read_csv("returns.csv", index_col=0, parse_dates=True) # 2. Build four portfolios mvo = MeanVarianceOptimiser(returns, cov_method="ledoit_wolf", risk_free_rate=0.05) ms = mvo.max_sharpe() mv = mvo.min_variance() erc = RiskParityOptimiser(returns).optimise() hrp = HRPOptimiser(returns).optimise() print(f"Max Sharpe: Sharpe={ms.sharpe_ratio:.3f}, Vol={ms.expected_volatility:.1%}") print(f"Min Variance: Sharpe={mv.sharpe_ratio:.3f}, Vol={mv.expected_volatility:.1%}") print(f"Risk Parity: EffN={erc.effective_n:.1f}") print(f"HRP: EffN={hrp.effective_n:.1f}") # 3. Kelly sizing for a strategy k = KellyCriterion.discrete(win_probability=0.60, win_payoff=2.0) print(f"Half-Kelly: {k['half_kelly']:.1%} per trade") # 4. Tail risk assessment port_returns = (returns * ms.to_series()).sum(axis=1) var_hist = TailRiskEstimator.historical_var(port_returns, 0.95) var_cf = TailRiskEstimator.cornish_fisher_var(port_returns, 0.95) print(f"VaR 95%: {var_hist['var_pct']:.2f}% (CF: {var_cf['var_pct']:.2f}%)") # 5. Rebalancing trades current = {"A": 0.33, "B": 0.33, "C": 0.34} rebalance = ms.rebalance_trades(current, portfolio_value=1_000_000) print(rebalance) ``` --- ## Algorithm Selection Guide | Objective | Recommended Model | Key Parameter | |-----------|------------------|---------------| | Best risk-adjusted return | `MVO.max_sharpe()` | `cov_method` | | Lowest volatility | `MVO.min_variance()` | `allow_short` | | Equal risk contribution | `RiskParityOptimiser` | `risk_budgets` | | Incorporate analyst views | `BlackLittermanModel` | `tau`, `P`, `Q` | | Correlation-robust allocation | `HRPOptimiser` | none | | Optimal bet sizing | `KellyCriterion.discrete()` | fraction of full Kelly | | Tail risk measurement | `TailRiskEstimator` | `confidence`, `horizon_days` | ## Usage Conventions 1. **Returns format** — daily decimal returns (not percent): `0.01 = 1%` 2. **Annualisation** — all outputs are annualised using `× 252` (trading days) 3. **Covariance** — input `returns` is daily; annualised internally by `× 252` 4. **Kelly fractions** — always use half-Kelly or less in live trading 5. **VaR sign** — returned as a positive loss (0.018 = 1.8% potential loss) 6. **CVaR** — always ≥ VaR; represents expected loss in the tail beyond VaR 7. **HRP** — no return estimates needed; pure covariance-based allocation 8. **Black-Litterman tau** — typical range 0.025–0.10; lower = more weight on prior --- ## Sharpe-First Portfolio Construction (Wall Street Quants) > Source: "The Sharpe Ratio Explained (by a quant trader)" by Wall Street Quants (Aug 2024) **Key quant insight:** Maximize Sharpe first, then use leverage to target desired return level. - Two negatively correlated SR=2.0 strategies combined 50/50 → SR=5.0 portfolio - Leverage preserves Sharpe: 2x leverage = 2x returns, 2x vol, same SR - The tangency portfolio (highest SR portfolio) is mathematically optimal per mean-variance theory - Benchmark: S&P 500 SR ~0.45, Buffett ~0.75, good hedge funds 2.0+ - Goal: SR 2.0+ consistently = better than 99% of investors For full Sharpe ratio deep-dive, see `risk-and-portfolio` skill. --- --- ## Strategy Allocation by Market Regime (merged from portfolio-optimization) > Pipeline: `market-regime-classifier` → `strategy-selection` → **this section** → `multi-strategy-orchestration` ### Regime → Strategy Allocation Matrix **TRENDING Regime** ``` Strategy | Allocation | Max Concurrent ICT MSS + FVG | 40% | 2 Displacement Trap Entry | 25% | 2 ORB (with-trend only) | 20% | 1 Silver Bullet / ICT 2022 | 15% | 1 Total portfolio heat: up to 4% ``` **RANGING Regime** ``` Strategy | Allocation | Max Concurrent S&D Zone Fades | 35% | 2 VWAP Mean Reversion | 30% | 2 Asian Range Fade | 20% | 1 ORB Mean Reversion (fade) | 15% | 1 Total portfolio heat: up to 3% (half-size — false breakouts common) ``` **TRANSITIONING Regime** ``` Strategy | Allocation | Max Concurrent Liquidity Trap / CRT | 40% | 2 Breaker Block Entry | 30% | 1 Counter-trend ICT models | 30% | 1 Total portfolio heat: up to 2% (highest uncertainty — smallest allocation) ``` **VOLATILE / NEWS Regime** ``` No new entries | Capital preservation | Post-news fade (30m wait) only Total portfolio heat: reduce to 1% max ``` ### Session-Based Allocation Overlay ``` Asia 00-07 UTC | Asian Range: 1.0x | All others: 0.25x London 07-12 | Breakout/Sweep: 1.0x | Mean Rev: 0.5x NY Open 13:30-15 | ORB: 1.0x | ICT: 1.0x | All valid Overlap 13:30-16 | ALL strategies: 1.0x (peak liquidity) Rule: Strategy allocation × session modifier = effective allocation ``` ### Drawdown-Based Allocation Scaling ``` 0-2% → 1.0x (full) | 2-4% → 0.75x | 4-6% → 0.50x 6-8% → 0.25x (one position max) | >10% → STOP — full review Recovery: Only step UP one level per profitable day ``` ### Rebalancing Priority - Reduce: (1) Against HTF trend → (2) Wrong regime strategy → (3) Worst R-multiple → (4) Newest → (5) Illiquid - Increase: (1) Highest win rate for regime → (2) Best liquidity for session → (3) Highest confluence score → (4) Uncorrelated --- ## Related Skills - [Risk And Portfolio](../risk-and-portfolio.md) - [Cross Asset Relationships](../cross-asset-relationships.md) - [Statistical Analysis](../statistics-timeseries.md)