Daily Calibration Report

Daily Calibration Report Skill

Purpose

Automated daily health check of all model components. This is your early warning system for:

• Model degradation
• Regime shifts
• Parameter drift
• Unexpected losses

When to Use

• Run automatically every day at market close or 6am
• Manually when debugging performance issues
• After deploying model changes
• When something "feels off"

Prerequisites

• measurement-infrastructure collecting data
• calibration-analysis for metrics computation
• At least 24 hours of production data

Report Sections

1. Executive Summary

╔════════════════════════════════════════════════════════════════════╗
║             DAILY CALIBRATION REPORT - 2024-01-15                  ║
╠════════════════════════════════════════════════════════════════════╣
║  Overall Status: WARNING                                           ║
║                                                                    ║
║  PnL:           $347.82  (+12% vs 7d avg)                         ║
║  Sharpe (1d):   2.1                                               ║
║  Fill Rate:     847 fills                                         ║
║  Uptime:        99.7%                                             ║
║                                                                    ║
║  Issues:                                                           ║
║  - Adverse selection model IR dropped to 0.92 (threshold: 1.0)    ║
║  - Cascade regime PnL: -$45 (investigate)                         ║
╚════════════════════════════════════════════════════════════════════╝

2. PnL Attribution

Breaks down daily PnL into: spread capture, adverse selection, inventory cost, fees. Compared against 7-day rolling average. Hourly PnL histogram for time-of-day analysis.

3. Model Calibration

Per-model metrics: Brier Score, Information Ratio, 7-day IR trend. Models tracked: Fill Prediction (1s, 10s), Adverse Selection, Direction (1s), Regime Detection. Includes calibration curve visualization (predicted vs realized).

4. Signal Health

Mutual information per signal vs 7 days ago. Flags signals with >15% MI decay for investigation. Tracks lead-lag R-squared for cross-exchange signals.

5. Regime Distribution

Time fraction, PnL, PnL/hour, fill count, and AS rate per regime (Quiet/Trending/Volatile/Cascade). Cascade losses are highlighted for investigation.

6. Conditional Calibration Issues

Sliced calibration by: volatility regime, funding state, time of day. Flags conditions where model IR drops below 1.0 — these are where the model adds noise.

7. Actionable Items

Prioritized list generated from all sections:

• HIGH: Model IR < 1.0, daily loss exceeding limit, AS > 50% of spread capture
• MEDIUM: Cascade losses, signal decay > 15%, conditional calibration issues
• LOW: Elevated inventory cost, minor parameter drift

See references/implementation.md for the DailyReportGenerator struct, generate() pipeline, action item generation, and status determination logic.

Alert Thresholds

See references/alerting.md for the AlertManager implementation and alert routing (Slack/Email/PagerDuty).

Dashboard Integration

Export metrics to Grafana/Prometheus for real-time monitoring. Key panels: PnL Waterfall, Model IR Trend, Signal MI Heatmap, Regime Timeline, Calibration Curve.

See references/dashboard.md for export_metrics() code and panel definitions.

Operational Checklist

• Report job scheduled (cron, systemd timer, etc.)
• Slack webhook configured
• Email alerts configured
• PagerDuty integration for critical alerts
• Dashboard panels created
• Historical reports archived
• Weekly review meeting scheduled

Dependencies

• Requires: measurement-infrastructure, calibration-analysis, all model components
• Enables: Automated monitoring, early warning, performance tracking

Common Mistakes

1. Not running daily: Problems compound if undetected
2. Ignoring warnings: "It's probably fine" leads to losses
3. Too many alerts: Alert fatigue defeats the purpose
4. No historical comparison: Need 7d context for anomaly detection
5. Missing conditional analysis: Aggregate metrics hide regime-specific issues

Supporting Files

• references/implementation.md — DailyReportGenerator, generate() pipeline, action items, status logic
• references/alerting.md — AlertManager, Slack/Email/PagerDuty routing
• references/dashboard.md — Grafana export, metric panels

Converted and distributed by TomeVault — claim your Tome and manage your conversions.