# Ml4t Monitoring Alerting

> Real-time monitoring and alerting for live trading systems. Use when building observability for data pipelines, model drift, or execution quality.

- Skill: `ml4t/ml4t-monitoring-alerting` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ml4t/ml4t-monitoring-alerting`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ml4t/ml4t-monitoring-alerting/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ml4t (https://skillmd.com/u/ml4t)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ml4t/ml4t-monitoring-alerting

---

# Monitoring and Alerting

Checking performance at end-of-day is too late. A stuck data feed, a rejected order, or a flash crash can cause irreversible losses in minutes. Automated real-time monitoring catches problems when they are still small.

## The Problem

A strategy runs in production. The data feed silently stalls at 10:15 AM - the strategy stops generating signals but nobody notices until 4 PM. By then, the portfolio has drifted and missed the day's best opportunities. Worse: a position limit was breached because a partial fill was not tracked, and the strategy doubled down. End-of-day review catches the problem 6 hours too late.

## The Pattern

### WRONG
```python
# End-of-day check - too late for anything but damage assessment
def daily_report():
    pnl = portfolio.value() - portfolio.start_of_day_value()
    if pnl < -10_000:
        send_email("Bad day", f"Lost ${abs(pnl):,.0f}")
schedule.every().day.at("16:30").do(daily_report)  # 6+ hours after it started
```

### CORRECT
```python
import time
from dataclasses import dataclass
from datetime import UTC, datetime

@dataclass
class AlertThresholds:
    max_drawdown: float = 0.05         # 5% intraday drawdown
    max_daily_loss: float = 0.02       # 2% daily loss
    max_position_pct: float = 0.10     # 10% in single name
    data_stale_seconds: int = 120      # 2 minutes without new data
    fill_rate_floor: float = 0.80      # 80% of orders must fill

def monitor_loop(portfolio, data_feed, thresholds: AlertThresholds, debounce=3):
    """Continuous monitoring with immediate alerting."""
    streak = {}

    def check(name, breached, message):
        # One bad sample is noise; `debounce` of them in a row is an incident.
        streak[name] = streak.get(name, 0) + 1 if breached else 0
        if streak[name] == debounce:
            alert(name, message)

    while True:
        now = datetime.now(UTC)
        last = data_feed.last_timestamp()
        if last.tzinfo is None:      # naive: subtracting from `now` raises
            last = last.replace(tzinfo=UTC)
        # .seconds is the sub-day part, so a 24h stall would read as fresh
        stale = (now - last).total_seconds()
        check("DATA_STALE", stale > thresholds.data_stale_seconds, f"Stale {stale:.0f}s")

        dd = portfolio.current_drawdown()
        check("DRAWDOWN", dd > thresholds.max_drawdown, f"Intraday DD: {dd:.1%}")
        loss = -portfolio.daily_return()  # the threshold was defined and never read
        check("DAILY_LOSS", loss > thresholds.max_daily_loss, f"Daily loss: {loss:.1%}")

        for sym, weight in portfolio.weights().items():
            check(f"CONCENTRATION:{sym}", abs(weight) > thresholds.max_position_pct,
                  f"{sym}: {weight:.1%}")

        fill_rate = portfolio.fill_rate(window_minutes=30)
        check("LOW_FILLS", fill_rate < thresholds.fill_rate_floor, f"Fills {fill_rate:.0%}")
        time.sleep(10)  # every 10s, so a 3-sample debounce alerts after 30s
```

## What to Monitor

Five categories: **P&L** (intraday drawdown > 5%, daily loss > 2%), **Positions** (single-name > 10%, gross leverage > 2x), **Execution** (fill rate < 80%, slippage > 2x model), **Data** (bar age > 2x interval, stale prices), **System** (latency > 500ms, memory > 80%).

## Alert Escalation

Four levels: **INFO** (log only), **WARNING** (Slack), **CRITICAL** (page on-call), **HALT** (trigger kill switch). Route by name from `check` above; the debounce there is what keeps a single bad sample off the pager.

## Guardrails

- Monitor loop must run independently from the trading process - if trading crashes, monitoring must still work
- Data staleness check must use wall clock, not data timestamps (which stop updating when the feed dies)
- Test the alerting path end-to-end: trigger a fake alert and verify it reaches the right person

## Production Implementation

`ml4t-live` provides monitoring hooks integrated with the trading engine:

```python
from ml4t.live import SafeBroker, LiveRiskConfig

config = LiveRiskConfig(
    execution_mode="shadow",   # required: "shadow", "paper" or "live"
    max_drawdown_pct=0.05,     # positive fraction below the high-water mark
    max_daily_loss=5_000.0,
    max_data_staleness_seconds=120.0,
)
broker = SafeBroker(inner_broker, config)
# A breach raises RiskLimitError in the TRADING process and latches the switch.
# The monitor above runs separately and never sees that exception - the trading
# process has to record the incident somewhere the monitor reads.
```

## Checklist

- [ ] Monitoring loop runs independently from trading process
- [ ] Data staleness detected within 2x expected bar interval
- [ ] Drawdown and daily loss alerts configured and tested
- [ ] Position concentration limits enforced
- [ ] Fill rate tracked with 30-minute rolling window
- [ ] Alert escalation tested end-to-end; all thresholds documented before go-live