Ml4t Production Readiness

Production Readiness

A validated backtest is not a production system. The gap between "model works" and "strategy can run unattended with real money" requires infrastructure that most teams skip until the first incident.

The Problem

A team deploys a model that passed validation. On day three, the data vendor has an outage. The model receives stale prices, generates signals, and the execution system submits orders on bad data. There is no kill switch, no staleness check, no alert. By the time someone notices, the strategy has lost two months of expected returns in a single session. Every production failure traces back to missing infrastructure, not bad models.

The Pattern

WRONG

# Model works in backtest - deploy directly
import pickle
from broker_api import submit_orders

model = pickle.load(open("best_model.pkl", "rb"))
data = fetch_latest_data()
signals = model.predict(data)
orders = signals_to_orders(signals)
for order in orders:
    submit_orders(order)  # No limits, no monitoring, no kill switch

CORRECT

# Five infrastructure layers before any live order
import datetime as dt

# Layer 1: DATA PIPELINE - validate freshness and integrity
latest_ts = data_store.get_latest_timestamp()
staleness = dt.datetime.now(dt.timezone.utc) - latest_ts
assert staleness < dt.timedelta(hours=2), f"Data stale: {staleness}"
assert data_store.validate_schema(), "Schema mismatch - pipeline broken"

# Layer 2: MODEL VERSIONING - reproducibility and audit trail
model_hash = compute_model_hash(model_path)
assert model_hash == registry.get_deployed_hash(), "Model hash mismatch"
assert registry.get_training_date(model_hash) > dt.date(2025, 6, 1), "Model too old"

# Layer 3: RISK LIMITS - hard limits that cannot be overridden by signals
risk_config = {
    "max_position_pct": 0.05,       # No single position > 5% of portfolio
    "max_sector_pct": 0.25,         # No sector > 25%
    "max_daily_turnover": 0.20,     # Max 20% daily turnover
    "max_drawdown_halt": 0.10,      # Halt trading at 10% drawdown
    "max_gross_leverage": 1.5,      # Hard leverage cap
}

# Layer 4: MONITORING - detect problems before they compound
# → Use ml4t-drift-detection for feature/prediction distribution shifts
# → Use ml4t-risk-metrics for rolling drawdown and exposure tracking
# Alerts fire on: data staleness, drift PSI > 0.25, drawdown > threshold

# Layer 5: KILL SWITCH - automated and manual halt capability
# → Use ml4t-kill-switch for implementation
# Kill switch triggers: max drawdown, data staleness, model drift, manual override
# Kill switch action: flatten all positions, cancel open orders, alert team

Five Readiness Layers

Paper Trading Gate

No strategy goes live without minimum 4 weeks of paper trading covering at least one rebalance cycle. Paper trading validates order generation, fill assumptions, data pipeline reliability, and monitoring/alerting behavior.

Guardrails

• If you cannot explain exactly which model version is running, do not trade live
• If the kill switch has not been tested (triggered and verified), it does not work
• If data fallback has not been tested by simulating a source outage, assume it will fail
• If drawdown limits exist only in code you can override, they are not real limits
• If there is no on-call rotation or incident playbook, the first production issue will be chaotic

Production Implementation

ml4t-live provides the infrastructure for safe live deployment:

import asyncio
from ml4t.backtest import Strategy  # Same Strategy class as backtest
from ml4t.live import AlpacaDataFeed, AlpacaBroker, LiveEngine, LiveRiskConfig, SafeBroker

risk = LiveRiskConfig(execution_mode="shadow", max_drawdown_pct=0.10)
broker = SafeBroker(AlpacaBroker(api_key, secret_key), risk)  # pre-trade limits and kill switch
feed = AlpacaDataFeed(api_key, secret_key, symbols=["SPY"], experimental=True)
engine = LiveEngine(MyStrategy(), broker, feed)  # run() is a coroutine
asyncio.run(engine.run())

Checklist

• Data pipeline validates freshness (staleness < threshold) and schema on every run
• Fallback data source tested by simulating primary source failure
• Deployed model hash matches training registry entry
• Feature pipeline is identical between training and production (no train/serve skew)
• Hard risk limits set; sector and leverage caps enforced outside LiveRiskConfig
• Kill-switch drill: it latches, close_all_positions() flattens, your pager fires
• Monitoring dashboards operational with alerts for drift, drawdown, staleness
• Paper trading completed for minimum 4 weeks with no execution anomalies
• Incident playbook and governance sign-off completed before go-live