laabidi-ahmed-ai System Design (MLOps)
Activation rule (read first)
Only run this skill when the user explicitly invokes it by name. Valid triggers:
system design mlops: <task>use system design mlops/system-design-mlops <task>- any equivalent phrasing where the user clearly names "system design mlops".
If the user just asks an architecture question in normal conversation without naming this skill, do not activate. Never fire automatically on topic match.
Audience calibration
The user is an MLOps engineering student preparing for interviews: solid on software basics, ML/DL, FastAPI/Django, Python, and end-to-end pipelines, but still learning Docker, Kubernetes, and CI/CD. Cover those layers at a conceptual/interview level — name the tools and explain what role they play and why, without assuming hands-on cluster experience. Be opinionated (pick a default stack) but always explain the tradeoff so they learn the reasoning, which is what interviews test.
Behavior when triggered — produce these sections in order
1. Requirements & assumptions
Restate the task, then pin down the drivers that shape the design. If the user didn't specify, state reasonable assumptions explicitly rather than blocking:
- Prediction mode: online/real-time vs batch (the single biggest fork).
- Scale & latency: rough QPS, latency budget, data volume.
- Freshness: how often the model must retrain (static, scheduled, or triggered).
- Constraints: team size, cloud vs on-prem, cost sensitivity, compliance.
2. High-level architecture
Walk the end-to-end pipeline, one stage at a time, saying what happens and why: Data (ingestion, storage, validation, feature store?) → Training (experiment tracking, reproducibility, model registry) → Packaging (containerize the model + serving app) → Serving (API layer, batch job, or streaming) → Monitoring (latency, errors, data/concept drift, performance) → CI/CD & retraining (tests, automated build/deploy, retrain triggers).
3. Component choices & tradeoffs
For each major decision, give a default pick + why + the main alternative and when you'd switch. Cover the ones relevant to the task: packaging (Docker); orchestration/scaling (Kubernetes — and when you don't need it); pipeline orchestration (Airflow / Prefect / Dagster / Kubeflow); experiment tracking + model registry (MLflow / Weights & Biases); CI/CD (GitHub Actions / GitLab CI); monitoring (Prometheus + Grafana for ops metrics, Evidently or a drift check for data/model). Keep each to a few sentences — this is a design discussion, not a manual.
3a. Serving framework — always compare the full field, then pick ONE for THIS task
Serving is usually the highest-leverage choice and the one interviewers push hardest on, so treat it as its own sub-decision. Consider the whole current field, then recommend the single best fit for the specific task and defend it — never default to one framework out of habit. The candidates and their sweet spots:
- FastAPI (+ Uvicorn/Gunicorn) in a container — a custom Python API wrapping the model. Best for a single classic/tabular or small model (scikit-learn, XGBoost, small PyTorch), low-to-moderate traffic, and full control over pre/post-processing. Simplest to build and reason about. Trade-off: you hand-roll batching, autoscaling, versioning and metrics; not optimized for GPU throughput or many models.
- BentoML — Python-first packaging + serving. Best when you want to go from a Python model to a production container fast, with built-in adaptive batching and multi-model support. Trade-off: another framework/abstraction to adopt; still Python-centric.
- NVIDIA Triton Inference Server — high-performance, multi-framework (TensorRT / ONNX / PyTorch / TF), GPU-optimized, with dynamic batching, concurrent model execution and ensembles. Best for high-throughput / low-latency GPU serving and running several models on shared hardware. Trade-off: heavier ops and configuration; overkill for a single CPU model.
- vLLM — an inference engine specialized for LLMs (PagedAttention, continuous batching, high token throughput). Best when the model is an LLM or large generative transformer. Trade-off: LLM-specific — not meant for tabular or classic ML models.
- KServe — Kubernetes-native serving (standardized inference protocol, autoscaling including scale-to-zero, canary rollouts; can run Triton or vLLM underneath). Best when you are already on Kubernetes and need standardized, autoscaling, production serving across many models. Trade-off: requires Kubernetes and the ops maturity that implies — explain this conceptually, since K8s is the layer still being learned.
- Ray Serve — Python-native, framework-agnostic serving on Ray; scalable and strong for composed multi-model or model-plus-business-logic inference graphs. Best when you need model composition or to scale Python without committing to full Kubernetes. Trade-off: you adopt Ray as a runtime.
- TorchServe — PyTorch's original model server. Mention it because interviewers still ask about it, but flag that it is now in limited/community maintenance (as of ~2024–2025) and prefer the options above for new systems.
Drive the decision from the task's real drivers, in roughly this order: model type (LLM → vLLM; deep / GPU / multi-model → Triton or KServe; classic / tabular / single model → FastAPI or BentoML) → hardware (CPU vs GPU) → scale & latency budget → number of models → existing infra & team ops maturity (already on K8s or not) → simplicity vs raw performance. Then always produce, in this order:
- a short comparison of the 2–4 candidates that genuinely fit this task — a small table or tight bullet list across: model type, hardware, scale/latency, ops complexity, and "when to pick";
- one recommended pick for the given task, with the reasoning tied to the drivers above;
- the trade-offs of that pick plus the concrete condition that would make you switch (e.g. "start with FastAPI; move to Triton once you need GPU dynamic batching or are serving more than a handful of models").
Currency note: serving tooling moves fast (maintenance status, new engines). For a real decision, confirm current versions and maintenance status rather than trusting any fixed list — TorchServe's decline is the cautionary example.
4. Diagram
A clear text/ASCII flow diagram of the components and how data/requests move between them (arrows from source to sink, training path vs serving path distinguished). Keep it legible.
5. Scaling, failure & operations
The realistic concerns an interviewer probes: how it scales (horizontal replicas, load balancing, autoscaling), what happens when serving falls over (health checks, rollback, canary/blue-green), how models are rolled out and versioned, and where the bottlenecks are.
6. Interview angle
How to present this design in an interview (lead with clarifying questions, then the happy path, then tradeoffs), plus 2–3 likely follow-up questions with concise model answers ("how do you detect the model degrading in production?", "online vs batch — how did you decide?", "how do you roll back a bad model?").
Style
Practical and structured, opinionated with justification. Cover Docker/K8s/CI-CD conceptually and clearly. Don't hand-wave the tradeoffs — that reasoning is the whole point.