arbazkhan971 Activate When
/godmode:mlops, "deploy model", "model serving"- "model drift", "retrain", "A/B test models"
- Trained model ready for production deployment
Workflow
1. Model Readiness
Model: <name and version>
Source: EXP-<ID>
Checklist:
[ ] Evaluation complete (test metrics documented)
[ ] Bias/fairness check passed
[ ] Artifacts saved (weights, config, preprocessor)
[ ] Input/output schema documented
[ ] Latency benchmarked (< target p99 ms)
[ ] Size acceptable (< N MB)
IF latency p99 > 100ms: apply optimization. IF model size > 500MB: consider distillation/pruning.
2. Serving Infrastructure
Options:
TF Serving: TensorFlow models, gRPC/REST
Triton: multi-framework, ONNX/TensorRT
SageMaker: managed AWS, auto-scaling
FastAPI/Ray Serve: custom, flexible
# Check for serving frameworks
pip list | grep -iE "fastapi|ray|triton|sagemaker"
ls model_repository/ serve/ 2>/dev/null
3. Inference Optimization
| Optimization | Latency | Size | Accuracy |
| Baseline FP32 | <ms> | <MB> | <val> |
| FP16 quant | <ms> | <MB> | <val> |
| INT8 quant | <ms> | <MB> | <val> |
| ONNX | <ms> | <MB> | <val> |
| Distillation | <ms> | <MB> | <val> |
IF accuracy drop > 1% from quantization: use FP16 only. IF latency target not met: try TensorRT or distillation.
Batching: static (fixed workload), dynamic (variable traffic, max_queue_delay_ms), adaptive (auto-tune).
4. Model Versioning
| Version | Metric | Status | Traffic |
| v3.1 | F1=0.891 | CHAMPION | 90% |
| v3.2 | F1=0.903 | CANARY | 10% |
| v3.0 | F1=0.879 | ARCHIVED | 0% |
Lifecycle: STAGED->CANARY->CHAMPION->ARCHIVED
5. A/B Testing
Champion: v<N> Challenger: v<N>
Split: <champion%>/<challenger%>
Routing: random|user-hash|feature-flag
Duration: <minimum days>
Sample size: <minimum per variant>
Success: primary metric >= <threshold> improvement
Guardrails: latency p99, error rate, business KPIs
IF p-value > 0.05 after min samples: no winner. IF guardrail regresses > 2%: stop test, revert.
6. Drift Detection
Feature drift (PSI):
< 0.1: no drift
0.1-0.2: moderate — monitor closely
> 0.2: significant — trigger retraining
Performance:
< 2% drop: normal variance
2-5% drop: warning — schedule review
> 5% drop: alert — trigger retraining
7. Retraining
Trigger: scheduled|drift-based|performance-based
Frequency: daily|weekly|monthly
Data window: last N days
Auto_deploy: false (requires A/B or human gate)
Cooldown: minimum time between retraining runs
8. Monitoring Dashboard
Requests/sec: <current> (avg/peak)
Latency p50/p95/p99: <ms>/<ms>/<ms>
Error rate: <pct>
Primary metric (7d rolling): <val>
Drift status: NONE|LOW|MODERATE|HIGH
Hard Rules
- NEVER deploy without readiness checklist complete.
- NEVER 100% traffic to new model on day one. Start shadow->5% canary->ramp.
- ALWAYS keep previous champion ready for rollback.
- NEVER auto-promote without validation gate.
- Rollback must take < 5 minutes.
TSV Logging
Append .godmode/mlops-results.tsv:
timestamp model version action latency_p99 status
Keep/Discard
KEEP if: metrics improve AND no guardrail regression
AND pipeline runs end-to-end.
DISCARD if: metrics regress OR pipeline fails.
Revert and log reason.
Stop Conditions
STOP when FIRST of:
- Model stable at 100% for 24h
- Drift monitoring configured
- Rollback tested < 5 min
Autonomous Operation
On failure: git reset --hard HEAD~1. Never pause.
Error Recovery
| Failure | Action |
|---|---|
| OOM during training | Resume checkpoint, reduce batch |
| Performance degrades | Check drift, trigger retrain |
| A/B no difference | Verify sample size, document null |