cloudthinker-ai ML Model Deployment Checklist
Phase 1: Model Validation
- Validate model before deployment
- Model performance metrics meet acceptance criteria
- Evaluated on held-out test set (not training data)
- Tested on edge cases and failure modes
- Bias and fairness metrics reviewed
- Model size and inference latency within requirements
- Comparison with current production model (if exists)
- Model card/documentation completed
- Get model review sign-off from stakeholders
Model Performance Summary
| Metric | Training | Validation | Test | Production Baseline | Threshold |
|---|---|---|---|---|---|
| Primary metric | > | ||||
| Secondary metric | > | ||||
| Latency (P50) | N/A | N/A | ms | ms | < ms |
| Latency (P99) | N/A | N/A | ms | ms | < ms |
| Model size | MB | N/A | N/A | MB | < MB |
Phase 2: Infrastructure Setup
- Prepare serving infrastructure
- Compute resources provisioned (GPU/CPU, memory)
- Auto-scaling configured (min/max replicas)
- Load balancer and health checks configured
- Model artifact stored in versioned registry
- Feature store connected (if real-time features needed)
- Caching layer configured (if applicable)
- Network security (API authentication, rate limiting)
- Validate infrastructure handles expected load
Resource Configuration
| Resource | Specification | Auto-Scale Min | Auto-Scale Max | Cost/hour |
|---|---|---|---|---|
| Compute | $ | |||
| GPU | $ | |||
| Memory | GB | |||
| Storage | GB | N/A | N/A | $ |
Phase 3: Serving Configuration
- Configure model serving
- Model serialization format (ONNX, TensorFlow SavedModel, PyTorch, pickle)
- Input validation and preprocessing pipeline
- Output post-processing and formatting
- Batching configuration (dynamic batching for throughput)
- Timeout configuration for inference requests
- Fallback behavior on model errors
- API versioning for model versions
- Test serving endpoint thoroughly
API Contract
| Field | Request | Response | Required | Validation |
|---|---|---|---|---|
| Input features | N/A | Type, range checks | ||
| Predictions | N/A | |||
| Confidence scores | N/A | |||
| Model version | N/A | |||
| Request ID | UUID format |
Phase 4: Monitoring Setup
- Configure model monitoring
- Prediction latency (P50, P95, P99)
- Prediction throughput (requests/sec)
- Error rate and error types
- Input data distribution (feature drift detection)
- Output distribution (prediction drift detection)
- Model performance degradation (if ground truth available)
- Resource utilization (CPU, GPU, memory)
- Feature store freshness (if applicable)
- Set up alerts for anomalies
Monitoring Alert Configuration
| Alert | Condition | Severity | Action |
|---|---|---|---|
| Latency spike | P95 > threshold | Warning | Investigate |
| Error rate | > 1% | Critical | Page on-call |
| Data drift | Distribution shift > threshold | Warning | Investigate model |
| Prediction drift | Output shift > threshold | Warning | Retrain pipeline |
| Resource saturation | CPU/GPU > 90% | Warning | Scale up |
Phase 5: Deployment Strategy
- Select deployment strategy
- Shadow deployment (run alongside, compare outputs)
- Canary deployment (gradual traffic shift)
- A/B test (controlled experiment with metrics)
- Blue-green (instant switch with rollback)
- Execute deployment
- Deploy new model version
- Route initial traffic percentage
- Monitor metrics during rollout
- Gradually increase traffic if healthy
- Full rollout or rollback decision
Rollout Plan
| Phase | Traffic % | Duration | Success Criteria | Rollback Trigger |
|---|---|---|---|---|
| Shadow | 0% (duplicate) | 24 hours | Output matches expectations | N/A |
| Canary | 5% | 4 hours | Error rate < 0.5%, latency OK | Error > 2% |
| Expand | 25% | 12 hours | Metrics stable | Performance degradation |
| Expand | 50% | 24 hours | Business metrics OK | Business impact |
| Full | 100% | - | Stable for 72 hours | Any regression |
Phase 6: Post-Deployment
- Post-deployment validation
- Verify model version serving correctly
- Validate end-to-end prediction quality
- Confirm monitoring and alerting working
- Document deployment in model registry
- Archive previous model version (keep for rollback)
- Schedule model retraining pipeline
- Set up data pipeline for ground truth collection
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
- Model Card: Performance, limitations, and intended use
- Deployment Runbook: Step-by-step deployment procedures
- Monitoring Dashboard: Model health and drift metrics
- Rollback Procedure: Steps to revert to previous version
- Retraining Schedule: Automated retraining cadence
Action Items
- Validate model meets performance thresholds
- Provision and test serving infrastructure
- Configure monitoring and drift detection
- Execute phased deployment with monitoring
- Validate post-deployment metrics
- Set up automated retraining pipeline
- Document model version and deployment details