Amey-Thakur Model deployment
A model in production is software with an extra failure mode: it can be healthy and wrong. Deployment therefore adds statistical gates (shadow, canary against metrics) on top of ordinary release engineering.
Method
- Choose the serving pattern by decision latency. Batch scoring (nightly scores into a table) when decisions tolerate staleness: cheapest, testable like any pipeline (see data-pipeline-design). Online service (REST/gRPC) for request-time decisions, with p99 latency and throughput budgets set before building. Edge/in-process for offline-capable or ultra-low-latency paths, paying in update latency and fleet fragmentation (see mobile-release-strategy).
- Package the whole prediction function. Model weights plus preprocessing, feature transformations, and postprocessing ship as one versioned artifact from the registry (see experiment-tracking); a model served without its exact pipeline is training/serving skew by construction (see feature-engineering). Containerize with pinned dependencies; validate the artifact answers golden requests correctly in CI before any environment sees it.
- Shadow before serving. Run the candidate on live traffic without acting on outputs: compare latency, error rate, prediction distribution vs incumbent, and (where labels arrive fast) accuracy. Shadow catches the engineering skew and distribution surprises offline evaluation cannot (see model-evaluation boundaries), at zero user risk.
- Roll out with statistical gates. Canary a small traffic slice; gates are business guardrails and prediction-health metrics (see drift-monitoring), not just HTTP 200s; where the model's value claim matters, the canary is a real experiment (see ab-test-design). Expand on evidence; halt criteria numeric and pre-agreed (the mobile-release-strategy discipline, applied to models).
- Keep rollback one switch away. Previous model version stays warm and loadable; rollback is a config flip, tested in drills, not a retraining project. Model versions are immutable and addressable; "we retrained over it" means you have no rollback (see rollback-strategy).
- Instrument the model-specific telemetry. Log inputs (sampled, PII-scrubbed), predictions, scores, model version per request; dashboards on prediction distribution, feature nulls, latency by version. This telemetry is what drift-monitoring and incident debugging consume; without it the model is a black box that pages you in business metrics.
Boundaries
- Retraining cadence and triggers are the drift-monitoring contract; deployment provides the safe path each retrain travels, not the schedule.
- Multi-armed bandits and online learning change the statistical machinery entirely; this skill covers the fixed-model release loop.
- LLM serving adds token streaming, caching, and cost tiers on top (see llm-cost-latency, inference-serving-optimization); the gating and rollback discipline transfers unchanged.