Using Ml Production

Using ML Production

Overview

This meta-skill routes you to the right production deployment skill based on your concern. Load this when you need to move ML models to production but aren't sure which specific aspect to address.

Core Principle: Production concerns fall into four categories. Identify the concern first, then route to the appropriate skill. Tools and infrastructure choices are implementation details, not routing criteria.

When to Use

Load this skill when:

• Deploying ML models to production
• Optimizing model inference (speed, size, cost)
• Setting up MLOps workflows (tracking, automation, CI/CD)
• Monitoring or debugging production models
• User mentions: "production", "deploy", "serve model", "MLOps", "monitoring", "optimize inference"

Don't use for: Training optimization (use training-optimization), model architecture selection (use neural-architectures), PyTorch infrastructure (use pytorch-engineering)

How to Access Reference Sheets

IMPORTANT: All reference sheets are located in the SAME DIRECTORY as this SKILL.md file.

When this skill is loaded from: skills/using-ml-production/SKILL.md

Reference sheets like quantization-for-inference.md are at: skills/using-ml-production/quantization-for-inference.md

NOT at: skills/quantization-for-inference.md ← WRONG PATH

When you see a link like [quantization-for-inference.md](quantization-for-inference.md), read the file from the same directory as this SKILL.md.

Routing by Concern

Category 1: Model Optimization

Symptoms: "Model too slow", "inference latency high", "model too large", "need to optimize for edge", "reduce model size", "speed up inference"

When to route here:

• Model itself is the bottleneck (not infrastructure)
• Need to reduce model size or increase inference speed
• Deploying to resource-constrained hardware (edge, mobile)
• Cost optimization through model efficiency

Routes to:

• quantization-for-inference.md - Reduce precision (INT8/INT4), speed up inference
• model-compression-techniques.md - Pruning, distillation, architecture optimization
• hardware-optimization-strategies.md - GPU/CPU/edge tuning, batch sizing

Key question to ask: "Is the MODEL the bottleneck, or is it infrastructure/serving?"

Category 2: Serving Infrastructure

Symptoms: "How to serve model", "need API endpoint", "deploy to production", "containerize model", "scale serving", "load balancing", "traffic management"

When to route here:

• Need to expose model as API or service
• Questions about serving patterns (REST, gRPC, batch)
• Deployment strategies (gradual rollout, A/B testing)
• Scaling concerns (traffic, replicas, autoscaling)

Routes to:

• model-serving-patterns.md - FastAPI, TorchServe, gRPC, ONNX, batching, containerization
• deployment-strategies.md - A/B testing, canary, shadow mode, rollback procedures
• scaling-and-load-balancing.md - Horizontal scaling, autoscaling, load balancing, cost optimization

Key distinction:

• Serving patterns = HOW to expose model (API, container, batching)
• Deployment strategies = HOW to roll out safely (gradual, testing, rollback)
• Scaling = HOW to handle traffic (replicas, autoscaling, balancing)

Category 3: MLOps Tooling

Symptoms: "Track experiments", "version models", "automate deployment", "reproducibility", "CI/CD for ML", "feature store", "model registry", "experiment management"

When to route here:

• Need workflow/process improvements
• Want to track experiments or version models
• Need to automate training-to-deployment pipeline
• Team collaboration and reproducibility concerns

Routes to:

• experiment-tracking-and-versioning.md - MLflow, Weights & Biases, model registries, reproducibility, lineage
• mlops-pipeline-automation.md - CI/CD for ML, feature stores, data validation, automated retraining, orchestration

Key distinction:

• Experiment tracking = Research/development phase (track runs, version models)
• Pipeline automation = Production phase (automate workflows, CI/CD)

Multi-concern: Queries like "track experiments AND automate deployment" → route to BOTH skills

Category 4: Observability

Symptoms: "Monitor production", "model degrading", "detect drift", "production debugging", "alert on failures", "model not working in prod", "performance issues in production"

When to route here:

• Model already deployed, need to monitor or debug
• Detecting production issues (drift, errors, degradation)
• Setting up alerts and dashboards
• Root cause analysis for production failures

Routes to:

• production-monitoring-and-alerting.md - Metrics, drift detection, dashboards, alerts, SLAs
• production-debugging-techniques.md - Error analysis, profiling, rollback procedures, post-mortems

Key distinction:

• Monitoring = Proactive (set up metrics, alerts, detect issues early)
• Debugging = Reactive (diagnose and fix existing issues)

"Performance" ambiguity:

• If "performance" = speed/latency → might be Category 1 (optimization) or Category 2 (serving/scaling)
• If "performance" = accuracy degradation → Category 4 (observability - drift detection)
• Ask clarifying question: "By performance, do you mean inference speed or model accuracy?"

Routing Decision Tree

User query → Identify primary concern

Is model THE problem (size/speed)?
  YES → Category 1: Model Optimization
  NO → Continue

Is it about HOW to expose/deploy model?
  YES → Category 2: Serving Infrastructure
  NO → Continue

Is it about workflow/process/automation?
  YES → Category 3: MLOps Tooling
  NO → Continue

Is it about monitoring/debugging in production?
  YES → Category 4: Observability
  NO → Ask clarifying question

Ambiguous? → Ask ONE question to clarify concern category

Clarification Questions for Ambiguous Queries

Query: "My model is too slow"

Ask: "Is this inference latency (how fast predictions are), or training time?"

• Training → Route to training-optimization (wrong pack)
• Inference → Follow-up: "Have you profiled to find bottlenecks?"
  • Model is bottleneck → Category 1 (optimization)
  • Infrastructure/batching issue → Category 2 (serving)

Query: "I need to deploy my model"

Ask: "What's your deployment target - cloud server, edge device, or batch processing?"

• Cloud/server → Category 2 (serving-patterns, then maybe deployment-strategies if gradual rollout needed)
• Edge/mobile → Category 1 (optimization first for size/speed) + Category 2 (serving)
• Batch → Category 2 (serving-patterns - batch processing)

Query: "My model isn't performing well in production"

Ask: "By performance, do you mean inference speed or prediction accuracy?"

• Speed → Category 1 (optimization) or Category 2 (serving/scaling)
• Accuracy → Category 4 (observability - drift detection, monitoring)

Query: "Set up MLOps for my team"

Ask: "What's the current pain point - experiment tracking, automated deployment, or both?"

• Tracking/versioning → Category 3 (experiment-tracking-and-versioning)
• Automation/CI/CD → Category 3 (mlops-pipeline-automation)
• Both → Route to BOTH skills

Multi-Concern Scenarios

Some queries span multiple categories. Route to ALL relevant skills in logical order:

Principle: Route in execution order (what needs to happen first).

Relationship with Other Packs

With llm-specialist

ml-production covers: General serving, quantization, deployment, monitoring (universal patterns)

llm-specialist covers: LLM-specific optimization (KV cache, prompt caching, speculative decoding, token streaming)

When to use both:

• "Deploy LLM to production" → llm-specialist (for inference-optimization) + ml-production (for serving, monitoring)
• "Quantize LLM" → llm-specialist (LLM-specific quantization patterns) OR ml-production (general quantization)

Rule of thumb: LLM-specific optimization stays in llm-specialist. General production patterns use ml-production.

With training-optimization

Clear boundary:

• training-optimization = Training phase (convergence, hyperparameters, training speed)
• ml-production = Inference phase (deployment, serving, monitoring)

"Too slow" disambiguation:

• Training slow → training-optimization
• Inference slow → ml-production

With pytorch-engineering

pytorch-engineering covers: Foundation (distributed training, profiling, memory management)

ml-production covers: Production-specific (serving APIs, deployment patterns, MLOps)

When to use both:

• "Profile production inference" → pytorch-engineering (profiling techniques) + ml-production (production context)
• "Optimize serving performance" → ml-production (serving patterns) + pytorch-engineering (if need low-level profiling)

Common Routing Mistakes

Common Rationalizations (Don't Do These)

Red Flags Checklist

If you catch yourself thinking ANY of these, STOP and clarify:

• "I'll guess optimization vs serving" → ASK which is the bottleneck
• "Performance probably means speed" → ASK speed or accuracy
• "Deploy = serving-patterns only" → Consider deployment-strategies and monitoring too
• "They mentioned [tool], route based on tool" → Route by CONCERN, not tool
• "MLOps = one skill" → Could span experiment tracking AND automation
• "Skip question to save time" → Clarifying prevents wrong routing

When in doubt: Ask ONE clarifying question. 10 seconds of clarification prevents minutes of wrong-skill loading.

Routing Summary Table

Integration Examples

Example 1: Full Production Pipeline

Query: "I trained a model, now I need to put it in production"

Routing:

1. Ask: "What's your deployment target and are there performance concerns?"
2. If "cloud deployment, model is fast enough":
   • model-serving-patterns.md (expose as API)
   • deployment-strategies.md (if gradual rollout needed)
   • production-monitoring-and-alerting.md (set up observability)
3. If "edge device, model too large":
   • quantization-for-inference.md (reduce size first)
   • model-serving-patterns.md (edge deployment pattern)
   • production-monitoring-and-alerting.md (if possible on edge)

Example 2: Optimization Decision

Query: "My inference is slow"

Routing:

1. Ask: "Have you profiled to find the bottleneck - is it the model or serving infrastructure?"
2. If "not profiled yet":
   • production-debugging-techniques.md (profile first to diagnose)
   • Then route based on findings
3. If "model is bottleneck":
   • hardware-optimization-strategies.md (check if hardware tuning helps)
   • If not enough → quantization-for-inference.md or model-compression-techniques.md
4. If "infrastructure/batching is bottleneck":
   • model-serving-patterns.md (batching strategies)
   • scaling-and-load-balancing.md (if traffic-related)

Example 3: MLOps Maturity

Query: "We need better ML workflows"

Routing:

1. Ask: "What's the current pain point - can't reproduce experiments, manual deployment, or both?"
2. If "can't reproduce, need to track experiments":
   • experiment-tracking-and-versioning.md
3. If "manual deployment is slow":
   • mlops-pipeline-automation.md
4. If "both reproducibility and automation":
   • experiment-tracking-and-versioning.md (establish tracking first)
   • mlops-pipeline-automation.md (then automate workflow)

When NOT to Use ml-production Skills

Skip ml-production when:

• Still designing/training model → Use neural-architectures, training-optimization
• PyTorch infrastructure issues → Use pytorch-engineering
• LLM-specific optimization only → Use llm-specialist (unless also need serving)
• Classical ML deployment → ml-production still applies but consider if gradient boosting/sklearn instead

Red flag: If model isn't trained yet, probably don't need ml-production. Finish training first.

Success Criteria

You've routed correctly when:

• ✅ Identified concern category (optimization, serving, MLOps, observability)
• ✅ Asked clarifying question for ambiguous queries
• ✅ Routed to appropriate skill(s) in logical order
• ✅ Didn't let tool choices (Kubernetes, TorchServe) dictate routing
• ✅ Recognized multi-concern scenarios and routed to multiple skills

ML Production Specialist Skills Catalog

After routing, load the appropriate specialist skill for detailed guidance:

1. quantization-for-inference.md - INT8/INT4 quantization, post-training quantization, quantization-aware training, precision reduction for inference speed
2. model-compression-techniques.md - Pruning (structured/unstructured), knowledge distillation, architecture optimization, model size reduction
3. hardware-optimization-strategies.md - GPU/CPU/edge tuning, batch sizing, memory optimization, hardware-specific acceleration (TensorRT, ONNX Runtime)
4. model-serving-patterns.md - FastAPI, TorchServe, gRPC, ONNX, batching strategies, containerization (Docker), REST/gRPC APIs
5. deployment-strategies.md - A/B testing, canary deployment, shadow mode, gradual rollout, rollback procedures, blue-green deployment
6. scaling-and-load-balancing.md - Horizontal scaling, autoscaling, load balancing, traffic management, cost optimization, replica management
7. experiment-tracking-and-versioning.md - MLflow, Weights & Biases, model registries, experiment reproducibility, model lineage, versioning
8. mlops-pipeline-automation.md - CI/CD for ML, feature stores, data validation, automated retraining, orchestration (Airflow, Kubeflow)
9. production-monitoring-and-alerting.md - Metrics tracking, drift detection, dashboards, alerting, SLAs, proactive monitoring
10. production-debugging-techniques.md - Error analysis, production profiling, rollback procedures, post-mortems, root cause analysis

References

• See design doc: docs/plans/2025-10-30-ml-production-pack-design.md
• Primary router: yzmir/ai-engineering-expert/using-ai-engineering
• Related packs: llm-specialist/using-llm-specialist, training-optimization/using-training-optimization