# System Design Mlops > Designs an end-to-end MLOps system for a task you give it — clarifies requirements, walks the pipeline (data → training → serving → monitoring → CI/CD), justifies each component choice with tradeoffs and alternatives, draws the flow, and frames it as an interview answer. ONLY trigger on explicit invocation by name — e.g. "system design mlops: ", "use system design mlops", or "/system-design-mlops ". Do NOT activate for general questions unless the user names this skill. - Skill: `laabidi-ahmed-ai/system-design-mlops` (Agent Skill) - Install (CLI): `npx skillmds@latest add laabidi-ahmed-ai/system-design-mlops` - Raw SKILL.md: https://api.skillmd.com/api/skills/laabidi-ahmed-ai/system-design-mlops/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Web & Frontend - Author: laabidi-ahmed-ai (https://skillmd.com/u/laabidi-ahmed-ai) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/laabidi-ahmed-ai/system-design-mlops --- # System Design (MLOps) ## Activation rule (read first) Only run this skill when the user **explicitly invokes it by name**. Valid triggers: - `system design mlops: ` - `use system design mlops` - `/system-design-mlops ` - any equivalent phrasing where the user clearly names "system design mlops". If the user just asks an architecture question in normal conversation **without** naming this skill, do **not** activate. Never fire automatically on topic match. ## Audience calibration The user is an **MLOps engineering student** preparing for interviews: solid on software basics, ML/DL, FastAPI/Django, Python, and end-to-end pipelines, but **still learning Docker, Kubernetes, and CI/CD**. Cover those layers at a conceptual/interview level — name the tools and explain *what role they play and why*, without assuming hands-on cluster experience. Be opinionated (pick a default stack) but always explain the tradeoff so they learn the reasoning, which is what interviews test. ## Behavior when triggered — produce these sections in order ### 1. Requirements & assumptions Restate the task, then pin down the drivers that shape the design. If the user didn't specify, **state reasonable assumptions explicitly** rather than blocking: - Prediction mode: **online/real-time vs batch** (the single biggest fork). - Scale & latency: rough QPS, latency budget, data volume. - Freshness: how often the model must retrain (static, scheduled, or triggered). - Constraints: team size, cloud vs on-prem, cost sensitivity, compliance. ### 2. High-level architecture Walk the end-to-end pipeline, one stage at a time, saying what happens and why: **Data** (ingestion, storage, validation, feature store?) → **Training** (experiment tracking, reproducibility, model registry) → **Packaging** (containerize the model + serving app) → **Serving** (API layer, batch job, or streaming) → **Monitoring** (latency, errors, data/concept drift, performance) → **CI/CD & retraining** (tests, automated build/deploy, retrain triggers). ### 3. Component choices & tradeoffs For each major decision, give a **default pick + why + the main alternative and when you'd switch**. Cover the ones relevant to the task: packaging (Docker); orchestration/scaling (Kubernetes — and when you *don't* need it); pipeline orchestration (Airflow / Prefect / Dagster / Kubeflow); experiment tracking + model registry (MLflow / Weights & Biases); CI/CD (GitHub Actions / GitLab CI); monitoring (Prometheus + Grafana for ops metrics, Evidently or a drift check for data/model). Keep each to a few sentences — this is a design discussion, not a manual. ### 3a. Serving framework — always compare the full field, then pick ONE for THIS task Serving is usually the highest-leverage choice and the one interviewers push hardest on, so treat it as its own sub-decision. **Consider the whole current field, then recommend the single best fit for the specific task and defend it — never default to one framework out of habit.** The candidates and their sweet spots: - **FastAPI (+ Uvicorn/Gunicorn) in a container** — a custom Python API wrapping the model. Best for a *single* classic/tabular or small model (scikit-learn, XGBoost, small PyTorch), low-to-moderate traffic, and full control over pre/post-processing. Simplest to build and reason about. Trade-off: you hand-roll batching, autoscaling, versioning and metrics; not optimized for GPU throughput or many models. - **BentoML** — Python-first packaging + serving. Best when you want to go from a Python model to a production container fast, with built-in adaptive batching and multi-model support. Trade-off: another framework/abstraction to adopt; still Python-centric. - **NVIDIA Triton Inference Server** — high-performance, multi-framework (TensorRT / ONNX / PyTorch / TF), GPU-optimized, with dynamic batching, concurrent model execution and ensembles. Best for high-throughput / low-latency GPU serving and running several models on shared hardware. Trade-off: heavier ops and configuration; overkill for a single CPU model. - **vLLM** — an inference *engine* specialized for LLMs (PagedAttention, continuous batching, high token throughput). Best when the model is an LLM or large generative transformer. Trade-off: LLM-specific — not meant for tabular or classic ML models. - **KServe** — Kubernetes-native serving (standardized inference protocol, autoscaling including scale-to-zero, canary rollouts; can run Triton or vLLM underneath). Best when you are *already on Kubernetes* and need standardized, autoscaling, production serving across many models. Trade-off: requires Kubernetes and the ops maturity that implies — explain this conceptually, since K8s is the layer still being learned. - **Ray Serve** — Python-native, framework-agnostic serving on Ray; scalable and strong for *composed* multi-model or model-plus-business-logic inference graphs. Best when you need model composition or to scale Python without committing to full Kubernetes. Trade-off: you adopt Ray as a runtime. - **TorchServe** — PyTorch's original model server. Mention it because interviewers still ask about it, **but flag that it is now in limited/community maintenance (as of ~2024–2025)** and prefer the options above for new systems. Drive the decision from the task's real drivers, in roughly this order: **model type** (LLM → vLLM; deep / GPU / multi-model → Triton or KServe; classic / tabular / single model → FastAPI or BentoML) → **hardware** (CPU vs GPU) → **scale & latency budget** → **number of models** → **existing infra & team ops maturity** (already on K8s or not) → **simplicity vs raw performance**. Then always produce, in this order: 1. a short **comparison** of the 2–4 candidates that genuinely fit this task — a small table or tight bullet list across: model type, hardware, scale/latency, ops complexity, and "when to pick"; 2. **one recommended pick** for the given task, with the reasoning tied to the drivers above; 3. the **trade-offs of that pick** plus the concrete condition that would make you switch (e.g. "start with FastAPI; move to Triton once you need GPU dynamic batching or are serving more than a handful of models"). **Currency note:** serving tooling moves fast (maintenance status, new engines). For a real decision, confirm current versions and maintenance status rather than trusting any fixed list — TorchServe's decline is the cautionary example. ### 4. Diagram A clear text/ASCII flow diagram of the components and how data/requests move between them (arrows from source to sink, training path vs serving path distinguished). Keep it legible. ### 5. Scaling, failure & operations The realistic concerns an interviewer probes: how it scales (horizontal replicas, load balancing, autoscaling), what happens when serving falls over (health checks, rollback, canary/blue-green), how models are rolled out and versioned, and where the bottlenecks are. ### 6. Interview angle How to *present* this design in an interview (lead with clarifying questions, then the happy path, then tradeoffs), plus 2–3 likely follow-up questions with concise model answers ("how do you detect the model degrading in production?", "online vs batch — how did you decide?", "how do you roll back a bad model?"). ## Style Practical and structured, opinionated with justification. Cover Docker/K8s/CI-CD conceptually and clearly. Don't hand-wave the tradeoffs — that reasoning is the whole point.