Differential Sharpe Ratio

Differential Sharpe Ratio (DSR) - Risk-Adjusted Reward Component

Overview

Context

The reward function in v3.3.0 had 7 components but lacked explicit risk-adjusted metrics. Literature review (2024-2025) identified Differential Sharpe Ratio (DSR) as a key technique for online risk-adjusted learning in financial RL.

Literature Sources:

• Moody & Saffell (1998): "Learning to trade via direct reinforcement"
• arXiv 2506.04358: "A Risk-Aware RL Reward for Financial Trading"
• SAGE Journals 2025 (CPPO): Risk-adjusted PPO achieving Sharpe ~2.15

Solution: Differential Sharpe Ratio

The DSR Formula

DSR provides an incremental, online measure of risk-adjusted returns:

A_t = A_{t-1} + eta * (R_t - A_{t-1})       # EMA of returns
B_t = B_{t-1} + eta * (R_t^2 - B_{t-1})     # EMA of squared returns
DSR_t = (B_t * delta_A - 0.5 * A_t * delta_B) / (B_t - A_t^2)^1.5

Where:

• R_t = return at time t
• eta = EMA decay rate (default 0.01)
• A = exponential moving average of returns (tracks mean)
• B = exponential moving average of squared returns (tracks variance)
• delta_A = R_t - A_{t-1} (innovation in mean)
• delta_B = R_t^2 - B_{t-1} (innovation in variance)

Configuration Parameters

Implementation

State Variables

# In GPUEnvConfig dataclass (~line 496)
dsr_weight: float = 0.10             # Risk-adjusted return component
dsr_eta: float = 0.01                # EMA decay for DSR statistics

# In GPUVectorizedTradingEnv.__init__ (~line 1006)
self.dsr_A = torch.zeros(self.n_envs, dtype=self.dtype, device=self.device)
self.dsr_B = torch.zeros(self.n_envs, dtype=self.dtype, device=self.device)

Reset Logic

# In reset() method (~line 1098)
self.dsr_A[env_ids] = 0.0
self.dsr_B[env_ids] = 0.0

DSR Calculation Method

def _calculate_dsr_reward(self, returns: Tensor) -> Tensor:
    eta = self.config.dsr_eta

    # Compute deltas before update
    delta_A = returns - self.dsr_A
    delta_B = returns ** 2 - self.dsr_B

    # Update EMA statistics
    A_new = self.dsr_A + eta * delta_A
    B_new = self.dsr_B + eta * delta_B

    # Variance with stability floor
    variance = torch.clamp(B_new - A_new ** 2, min=1e-8)

    # DSR formula
    numerator = B_new * delta_A - 0.5 * A_new * delta_B
    denominator = variance ** 1.5
    dsr = numerator / (denominator + 1e-8)

    # Update state
    self.dsr_A = A_new
    self.dsr_B = B_new

    # Scale and clamp
    return torch.clamp(dsr * 10, -2.0, 2.0)

Integration in Reward Function

# In _calculate_rewards() (~line 2252)
dsr_reward = self._calculate_dsr_reward(raw_pnl)

# Combined reward (8 components as of v3.8.0)
reward = (
    self.config.direction_weight * direction_reward +
    self.config.magnitude_weight * magnitude_reward +
    self.config.pnl_weight * pnl_reward +
    self.config.stop_tp_weight * stop_tp_reward +
    self.config.exploration_weight * exploration_bonus +
    self.config.slippage_weight * slippage_penalty +
    self.config.drawdown_penalty_weight * drawdown_penalty +
    self.config.dsr_weight * dsr_reward  # NEW in v3.8.0
) * risk_adjustment

Why DSR Works

Online Estimation

Unlike batch Sharpe ratio (requires full episode), DSR:

• Updates incrementally at each step
• Provides immediate feedback during training
• No need to wait for episode completion

Risk Adjustment

DSR penalizes volatile returns:

• High mean, low variance = high DSR (good)
• High mean, high variance = moderate DSR (risky)
• Low mean = low or negative DSR (bad)

Gradient Signal

DSR provides gradient signal for:

• Reducing position size in volatile periods
• Maintaining consistent returns over erratic wins
• Avoiding large drawdowns even with occasional big wins

Failed Attempts

Key Insights

Weight Balance

At 10% weight, DSR provides meaningful signal without dominating:

• P&L (40%) remains primary objective
• DSR (10%) adds risk awareness
• Other components maintain their roles

EMA Decay Rate

The eta=0.01 default means:

• ~63% weight on last 100 observations
• Smooth signal that adapts to regime changes
• Not too reactive to individual outliers

Scaling

Raw DSR values are typically small (-0.1 to 0.1), so we:

1. Multiply by 10 to amplify signal
2. Clamp to [-2, 2] to prevent extreme values
3. Handle NaN/Inf from numerical edge cases

Related Skills

• reward-function-v330 - Base reward function design
• integrated-risk-manager - Unified risk sizing
• adaptive-predator-prey - Regime-aware dynamics