Mlops And Deployment

MLOps and Deployment

Machine learning foundations end where a trained, validated model sits on disk. MLOps begins there. It is the operational discipline of taking that model into production and keeping it reliable: packaging and serving it, versioning everything that produced it, automating the path from commit to deployment, watching it degrade against a shifting world, and closing the loop with retraining. A model that scores 0.94 on a held-out test set is a scientific result; a model that still scores 0.94 on live traffic six months later is an engineering achievement. This skill covers the second thing.

Agent affinity: breiman (model packaging, serving strategy, evaluation gates), nightingale (pipeline routing, deployment orchestration, monitoring synthesis)

Concept IDs: data-science-drift-detection, data-hypothesis-testing, data-data-quality, data-sampling-methods

The Production ML Lifecycle

Where the ML workflow stops at "deploy," MLOps treats deployment as the midpoint of a loop, not the finish line.

The distinguishing property of ML systems is that the code is not the only thing that changes -- the data changes underneath a static model. This is why MLOps is not "DevOps for models": you version and monitor data and models as first-class artifacts, not just source.

Model Packaging and Serving

The Artifact Boundary

A deployable model is more than its weights. Package the entire inference function:

• Model weights / parameters (the serialized estimator, e.g. joblib, ONNX, SavedModel, TorchScript).
• Preprocessing -- the exact fitted transformers (scalers, encoders, vocabularies). A model shipped without its scaler is a silent failure waiting to happen.
• A signature -- input schema (names, dtypes, shapes) and output schema, so consumers and monitors can validate.
• Environment -- pinned dependency versions, ideally a container image, so the runtime matches training.

Prefer an interchange format (ONNX, PMML) or a container over pickling a raw Python object: pickled objects are brittle across library versions and are a code-execution risk if the source is untrusted.

Serving Patterns

Batch is cheaper and simpler; reach for online serving only when a fresh prediction must exist at request time for an input you could not have precomputed.

Online Endpoints (REST)

An online model is a service. Design it like one:

• Contract: stable request/response schema; validate inputs against the model signature and reject malformed payloads (a 4xx beats a garbage prediction).
• Health and readiness: liveness for the process, readiness that confirms the model artifact loaded.
• Latency budget: the model is one hop; feature lookups, network, and serialization all count. Measure p99, not the mean.
• Scaling: stateless replicas behind a load balancer; autoscale on throughput. Warm the model at startup so the first request is not a cold-start outlier.
• Batching: micro-batch concurrent requests for throughput on GPU-bound models, trading a little latency.

Feature Stores

The hardest online-serving bug is training/serving skew: the features computed offline for training differ subtly from those computed online at inference. A feature store solves this by computing each feature once and serving it to both paths.

• Offline store (columnar warehouse) provides point-in-time-correct features for training -- crucially, the feature as it was known at label time, never leaking the future.
• Online store (low-latency key-value) serves the same feature definitions at inference in milliseconds.
• Shared definitions guarantee the same transformation logic feeds training and serving, eliminating skew and enabling feature reuse across models.

Reproducibility

If you cannot rebuild a model bit-for-bit from recorded inputs, you cannot debug it, audit it, or trust its retraining.

Version and Pin Everything

A model artifact should carry lineage: given a version, you can name the exact code SHA, data version, and config that produced it.

Deterministic Pipelines

• Seed every stochastic step (splits, initialization, shuffling, augmentation) and record the seed.
• Beware nondeterminism you do not control -- unordered parallel reductions, GPU kernel nondeterminism, dict/set ordering. Pin what you can; document what you cannot.
• Make the pipeline idempotent: re-running a stage on the same inputs yields the same outputs, so partial failures re-run safely.

Experiment Tracking and Model Registries

Experiment Tracking

During development you will train hundreds of candidates. Track each run's parameters, metrics, code version, data version, and artifacts (MLflow, Weights & Biases, or equivalent) so you can compare runs, reproduce the winner, and answer "why did we ship this one?" months later. An untracked experiment is a result you cannot defend.

Model Registry

The registry is the governed inventory of models that are candidates for or are in production. It provides:

• Versioning -- immutable, numbered model versions with their metrics and lineage.
• Stage transitions -- None -> Staging -> Production -> Archived, with an explicit, often gated, promotion step.
• A single source of truth -- serving infrastructure pulls "the Production version of model X," decoupling deployment from any one training run.
• Governance -- who promoted what, when, and against which evaluation. This is the audit trail regulators and incident reviews ask for.

CI/CD for ML

Classic CI/CD ships code. ML adds two more moving parts -- data and models -- so the discipline extends to CI/CD/CT (continuous training).

• Continuous Integration extends beyond unit tests to data validation (schema, ranges, distributions) and model validation (does the retrained candidate beat the current production model on a frozen evaluation set and on fairness/slice metrics?).
• Continuous Delivery deploys not just a service but a training pipeline; the deliverable is a repeatable pipeline that can produce a model, not only the model.
• Continuous Training automatically retrains and re-validates when triggered (see Retraining Triggers), producing a new registry candidate.

Gate promotion on evaluation, never on "it trained." A candidate must clear: no regression on the reference test set, acceptable performance on protected slices, and passing data-validation checks. Only then does it become eligible for release. Test the pipeline, not just the model -- most production failures are plumbing (a schema change, a null feature), not modeling.

Monitoring in Production

A deployed model degrades silently: it keeps returning confident predictions while the world drifts away from its training distribution. Monitor three layers.

1. Operational health (it is a service): latency, throughput, error rate, saturation. Necessary but not sufficient -- a model can be perfectly healthy operationally while making increasingly wrong predictions.

2. Data and model quality:

Distinguish drift types because the fix differs: data drift may only need recalibration or retraining on recent data; concept drift means the learned relationship is stale and the model must relearn. Concept drift can be sudden (a policy change), gradual, incremental, or recurring (seasonality) -- and each calls for a different retraining cadence.

3. The label-latency problem: ground truth often arrives late (a loan default surfaces months later) or never. When you cannot measure accuracy in real time, use proxy signals -- input drift, prediction drift, and business KPIs -- as leading indicators, and reserve delayed labels for confirmation.

Retraining Triggers and Pipelines

Retraining is the loop that keeps a model current. Choose a trigger policy:

A retraining pipeline is the same reproducible pipeline used originally, re-run on fresh data: assemble and validate new data, retrain, evaluate the candidate against the current production model on a frozen set, and -- only if it wins and passes checks -- register it and hand it to the release stage. Automate the candidate; keep a human (or a hard gate) on promotion. Guard against feedback loops: when a model's own predictions influence the data it later trains on (a ranker teaches users what to click), naive retraining amplifies its own biases. Log the decisions the model influenced so you can correct for them.

Progressive Deployment and Rollback

Never flip 100% of traffic to a new model version at once. Release progressively so a bad model is contained.

Shadow answers correctness, A/B answers value. Shadow deployment catches operational and sanity failures before any user is affected; A/B testing measures whether the offline metric gain translates to a real outcome (offline AUC gains routinely fail to move the business metric).

Rollback is the non-negotiable safety net. Because the previous model is an immutable, registered artifact, rollback is a version pointer change, not a rebuild: repoint the serving layer at the last-known-good Production version. Keep the prior version deployable and warm; define, in advance, the metric thresholds that trigger an automatic rollback. A deployment strategy without a tested rollback path is not a deployment strategy.

Common Mistakes

Cross-References

• breiman agent: Model selection and evaluation -- the quality gate a retrained candidate must clear before promotion.
• nightingale agent: Department routing and synthesis -- orchestrating the deploy/monitor/retrain loop across specialists.
• machine-learning-foundations skill: The training-side counterpart; produces the validated model this skill takes to production. Its final workflow stage ("deployment") is this skill's entire subject.
• experimental-design-ds skill: The statistical backbone of A/B tests and shadow evaluations -- randomization, sample size, and causal reading of a rollout.
• ethics-governance skill: Fairness monitoring, model cards, and the accountability trail that model registries and gated promotion make enforceable.
• data-wrangling skill: The reproducible transformation chain that feature stores and retraining pipelines depend on.

References

• Sculley, D., et al. (2015). "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems (NeurIPS), 28.
• Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction." IEEE International Conference on Big Data, 1123-1132.
• Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly Media.
• Gama, J., Zliobaite, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). "A Survey on Concept Drift Adaptation." ACM Computing Surveys, 46(4), 1-37.
• Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2022). "Challenges in Deploying Machine Learning: A Survey of Case Studies." ACM Computing Surveys, 55(6), 1-29.