Codebase Consolidation Pattern

Codebase Consolidation Pattern

Experiment Overview

Context

Problem: Codebase grew organically with many root-level modules (24 .py files). Related functionality scattered:

• Trading: broker.py, executor.py, pdt.py, market_hours.py, profit_tracker.py, notify.py
• Training: model_version.py, online_bandit.py, online_learning.py, rl_context_gpu.py
• Risk: risk_monitor.py floating at root instead of in risk/

Solution: Move related modules to subpackages, create compatibility shims at original locations.

Verified Workflow

1. Identify Module Groups

Analyze imports and functionality to group related modules:

# Trading execution group
trading/
  - broker.py          # Alpaca API wrapper
  - executor.py        # Order execution
  - pdt.py             # Pattern Day Trader rules
  - market_hours.py    # Market session detection
  - profit_tracker.py  # P&L tracking
  - notify.py          # Notifications

# Training system group
training/
  - model_version.py   # Version protocol
  - online_bandit.py   # Thompson sampling
  - online_learning.py # Real-time updates
  - rl_context_gpu.py  # GPU context building

# Existing packages to extend
risk/
  - risk_monitor.py    # Real-time monitoring (move from root)

2. Move Files with Git (Preserve History)

# Create new package directory
mkdir alpaca_trading/trading

# Move files preserving git history
git mv alpaca_trading/broker.py alpaca_trading/trading/
git mv alpaca_trading/executor.py alpaca_trading/trading/
git mv alpaca_trading/pdt.py alpaca_trading/trading/
git mv alpaca_trading/market_hours.py alpaca_trading/trading/
git mv alpaca_trading/profit_tracker.py alpaca_trading/trading/
git mv alpaca_trading/notify.py alpaca_trading/trading/

3. Update Internal Imports

After moving, update relative imports in moved files:

# BEFORE (in executor.py at root):
from .broker import Broker, BrokerConfig
from .utils import setup_logger
from .pdt import DayTradeGuard

# AFTER (in trading/executor.py):
from .broker import Broker, BrokerConfig  # Same package, no change
from ..utils import setup_logger          # Parent package, add ..
from .pdt import DayTradeGuard            # Same package, no change

4. Create Package init.py

# alpaca_trading/trading/__init__.py
"""Trading execution and broker integration."""
from .broker import Broker, BrokerConfig
from .executor import Executor
from .pdt import DayTradeGuard
from .market_hours import (
    is_trading_hours,
    get_next_trading_window,
    detect_asset_type,
    AssetType,
    MarketSession,
)
from .profit_tracker import ProfitTracker
from .notify import (
    EmailNotifier,
    TelegramNotifier,
    from_env_email,
    from_env_telegram,
)

__all__ = [
    'Broker', 'BrokerConfig', 'Executor', 'DayTradeGuard',
    'is_trading_hours', 'get_next_trading_window', 'detect_asset_type',
    'AssetType', 'MarketSession', 'ProfitTracker',
    'EmailNotifier', 'TelegramNotifier', 'from_env_email', 'from_env_telegram',
]

5. Create Backwards Compatibility Shims

At original locations, create thin wrapper files:

# alpaca_trading/broker.py (shim)
"""Compatibility shim - broker moved to trading/broker.py"""
from .trading.broker import *  # noqa: F401,F403

# alpaca_trading/executor.py (shim)
"""Compatibility shim - executor moved to trading/executor.py"""
from .trading.executor import *  # noqa: F401,F403

6. Handle Private Functions

from module import * doesn't export private functions (starting with _). If tests use them:

# alpaca_trading/mdp.py (shim with private function)
"""Compatibility shim - mdp archived to _archive/mdp.py"""
from ._archive.mdp import *  # noqa: F401,F403
from ._archive.mdp import _tanh_clip  # noqa: F401 - private but used in tests

7. Update Extended Package init.py

When adding to existing package (e.g., risk/):

# alpaca_trading/risk/__init__.py
"""Risk management module."""
from .portfolio_risk import AdvancedRiskManager, RiskLimits, PositionRisk
from .garch import GARCHRiskManager, fit_garch_model
from .capital_manager import CapitalManager, CapitalAllocation
# NEW: Add moved module
from .risk_monitor import (
    RealTimeRiskMonitor,
    RiskAlert,
    CircuitBreakerConfig,
    create_risk_monitoring_system,
)

__all__ = [
    # ... existing exports ...
    # NEW exports
    'RealTimeRiskMonitor', 'RiskAlert', 'CircuitBreakerConfig',
    'create_risk_monitoring_system',
]

Import Patterns Summary

Failed Attempts

Checklist

• Identify logical module groups
• Create target package directories
• Use git mv to preserve history
• Update internal imports (. → .. for parent)
• Create package __init__.py with exports
• Create compatibility shims at original locations
• Handle private functions explicitly if needed
• Run tests to verify nothing broken
• Commit with descriptive message

Final Structure

alpaca_trading/
├── trading/                    # NEW: Trading execution
│   ├── __init__.py
│   ├── broker.py
│   ├── executor.py
│   ├── pdt.py
│   ├── market_hours.py
│   ├── profit_tracker.py
│   └── notify.py
├── training/                   # EXTENDED: Training system
│   ├── __init__.py
│   ├── archive.py             # existing
│   ├── gating.py              # existing
│   ├── model_version.py       # moved
│   ├── online_bandit.py       # moved
│   ├── online_learning.py     # moved
│   └── rl_context_gpu.py      # moved
├── risk/                       # EXTENDED: Risk management
│   ├── __init__.py
│   ├── portfolio_risk.py      # existing
│   ├── garch.py               # existing
│   └── risk_monitor.py        # moved from root
│
├── broker.py                   # SHIM → trading/broker.py
├── executor.py                 # SHIM → trading/executor.py
├── pdt.py                      # SHIM → trading/pdt.py
├── market_hours.py             # SHIM → trading/market_hours.py
├── profit_tracker.py           # SHIM → trading/profit_tracker.py
├── notify.py                   # SHIM → trading/notify.py
├── model_version.py            # SHIM → training/model_version.py
├── online_bandit.py            # SHIM → training/online_bandit.py
├── online_learning.py          # SHIM → training/online_learning.py
├── rl_context_gpu.py           # SHIM → training/rl_context_gpu.py
└── risk_monitor.py             # SHIM → risk/risk_monitor.py

Key Insights

• Shims are essential: Without them, all existing code breaks instantly
• Git history preserved: Use git mv, not mv + git add
• Test after each group: Move one group, test, commit, then next group
• Relative imports change: Moving deeper requires .. prefix for parent imports
• Private functions need explicit import: * doesn't export _prefixed functions

References

• Commit d0f12b5: Full consolidation implementation
• alpaca_trading/trading/__init__.py: Example package exports
• alpaca_trading/broker.py: Example shim file