ML Cloud Deployment
Overview
Use this skill for deploying ML workloads to managed platforms, Kubernetes, serverless systems, GPU/TPU providers, and lakehouse environments. Start from workload requirements: training or inference, batch or online, latency SLO, throughput, model size, data gravity, compliance, region, hardware, team expertise, and budget.
Platform Selection
| Requirement |
Strong choices |
| AWS-native managed lifecycle |
SageMaker Studio, Training, Processing, Pipelines, Model Registry, Endpoints, Feature Store, Clarify, Model Monitor |
| GCP-native managed lifecycle |
Vertex AI Training, Pipelines, Endpoints, Feature Store, Model Monitoring, AutoML, Matching Engine, TPUs |
| Azure-native managed lifecycle |
Azure ML workspaces, compute clusters, pipelines, registries, managed online/batch endpoints, AutoML, Responsible ML |
| Lakehouse/Spark-centric ML |
Databricks on AWS/Azure/GCP with MLflow, Delta, Feature Store, Workflows |
| Kubernetes control |
EKS/GKE/AKS with Kubeflow, KServe, Seldon, Ray, Triton, custom operators |
| Serverless or fast GPU apps |
Modal, Replicate, Cloud Run with GPU where available, Lambda for small CPU inference |
| Flexible GPU rental |
Lambda Labs, RunPod, self-managed cloud GPU VMs |
| Ray-native scale-out |
Anyscale or Ray clusters on Kubernetes/cloud VMs |
Prefer managed services when governance, observability, and team velocity matter more than runtime customization. Prefer Kubernetes or VMs when custom networking, specialized runtimes, or cost/performance tuning dominate.
AWS Patterns
SageMaker handles managed training jobs, feature stores, and real-time or serverless endpoints. The AWS CLI (aws sagemaker and aws sagemaker-runtime) is used to orchestrate these systems programmatically.
AWS SageMaker Endpoint Variant Configuration (endpoint_config.json)
{
"EndpointConfigName": "production-llm-classifier-v1-cfg",
"ProductionVariants": [
{
"VariantName": "AllTraffic",
"ModelName": "llm-classifier-model-v1",
"InitialInstanceCount": 2,
"InstanceType": "ml.g5.2xlarge",
"InitialVariantWeight": 1.0,
"VolumeSizeInGB": 50,
"ManagedInstanceScaling": {
"MinInstanceCount": 2,
"MaxInstanceCount": 10,
"Status": "ENABLED"
},
"RoutingConfig": {
"RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS"
}
}
]
}
AWS CLI SageMaker Lifecycle Management
# 1. Setup Domain and default IAM Role/VPC settings
aws sagemaker create-domain \
--domain-name ProdMLDomain \
--auth-mode IAM \
--default-user-settings ExecutionRole=arn:aws:iam::123456789012:role/SageMakerExecutionRole \
--subnet-ids subnet-1a2b3c,subnet-4d5e6f \
--vpc-id vpc-0abc123 \
--app-network-access-type PublicInternetOnly
# 2. Provision and start/stop an interactive notebook instance
aws sagemaker create-notebook-instance \
--notebook-instance-name dev-notebook-instance \
--instance-type ml.t3.medium \
--role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole
aws sagemaker start-notebook-instance --notebook-instance-name dev-notebook-instance
aws sagemaker stop-notebook-instance --notebook-instance-name dev-notebook-instance
# 3. Submit a managed GPU training job
aws sagemaker create-training-job \
--training-job-name custom-pytorch-train-job \
--algorithm-specification TrainingImage=763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.12.0-gpu-py38-cu113-ubuntu20.04,TrainingInputMode=File \
--role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
--input-data-config '[{"ChannelName": "training", "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": "s3://my-ml-bucket/train-data/", "S3DataDistributionType": "FullyReplicated"}}}]' \
--output-data-config S3OutputPath=s3://my-ml-bucket/checkpoints/ \
--resource-config InstanceType=ml.g5.2xlarge,InstanceCount=1,VolumeSizeInGB=50 \
--stopping-condition MaxRuntimeInSeconds=86400
# 4. Register a model in the SageMaker Model Registry
aws sagemaker create-model-package-group \
--model-package-group-name credit-scoring-group \
--model-package-group-description "Model registry group for credit scoring neural nets"
aws sagemaker create-model \
--model-name credit-scoring-v1 \
--primary-container Image=763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.12.0-gpu-py38,ModelDataUrl=s3://my-ml-bucket/checkpoints/custom-pytorch-train-job/output/model.tar.gz \
--execution-role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole
# 5. Create Endpoint Configuration and Deploy Endpoint
aws sagemaker create-endpoint-config \
--endpoint-config-name credit-scoring-v1-cfg \
--production-variants '[{"VariantName": "AllTraffic", "ModelName": "credit-scoring-v1", "InitialInstanceCount": 2, "InstanceType": "ml.m5.xlarge", "InitialVariantWeight": 1.0}]'
aws sagemaker create-endpoint \
--endpoint-name credit-scoring-prod-endpoint \
--endpoint-config-name credit-scoring-v1-cfg
# 6. Invoke/Test Endpoint via CLI
aws sagemaker-runtime invoke-endpoint \
--endpoint-name credit-scoring-prod-endpoint \
--content-type application/json \
--body '{"features": [0.25, 1.4, 0.9]}' \
response_output.json
GCP Patterns
Vertex AI manages training pipelines, hyperparameter tuning sweeps, custom jobs, the model registry, and predictions via gcloud ai.
GCP Vertex AI TPU CustomJob Specification (tpu_job.yaml)
displayName: vertex-ai-tpu-training-job
studySpec:
metrics:
- metricId: val_loss
goal: MINIMIZE
parameters:
- parameterId: learning_rate
doubleValueSpec:
minValue: 1e-5
maxValue: 1e-3
scaleType: UNIT_LOG_SCALE
trialJobSpec:
workerPoolSpecs:
- machineSpec:
machineType: n1-standard-8
replicaCount: 1
containerSpec:
imageUri: gcr.io/my-project/ml-trainer:latest
- machineSpec:
machineType: cloud-tpu-v4-podslice
acceleratorType: TPU_V4
acceleratorCount: 4
replicaCount: 1
containerSpec:
imageUri: gcr.io/my-project/ml-tpu-trainer:latest
args: [
"--data_path", "gs://my-bucket/dataset-v1",
"--epochs", "10"
]
GCP Vertex AI CLI Operations
# 1. Create a user-managed Jupyter Workbench instance
gcloud notebooks instances create dev-workbench-instance \
--location=us-central1-a \
--vm-image-project=deeplearning-platform-release \
--vm-image-family=common-gpu \
--machine-type=n1-standard-4
# 2. Submit containerized custom training job
gcloud ai custom-jobs create \
--region=us-central1 \
--display-name=gpu-pytorch-training-run \
--worker-pool-spec=replica-count=1,machine-type=n1-standard-8,accelerator-type=nvidia-tesla-t4,accelerator-count=1,container-image-uri=gcr.io/my-gcp-project/pytorch-train-custom:v1 \
--args="--epochs=20,--data-path=gs://my-bucket/training-gold"
# 3. Upload model artifact to Model Registry
gcloud ai models upload \
--region=us-central1 \
--display-name=iris-classifier-v1 \
--container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/pytorch-cpu.1-11:latest \
--artifact-uri=gs://my-bucket/models/iris-classifier/
# 4. Provision endpoint and deploy model with 100% traffic allocation
# Note: replace <ENDPOINT-ID> and <MODEL-ID> with their UUIDs
gcloud ai endpoints create \
--region=us-central1 \
--display-name=production-iris-endpoint
gcloud ai endpoints deploy-model <ENDPOINT-ID> \
--region=us-central1 \
--model=<MODEL-ID> \
--display-name=iris-v1-blue-deployment \
--machine-type=n1-standard-4 \
--min-replica-count=2 \
--max-replica-count=10 \
--traffic-split=0=100
# 5. Predict via Vertex AI Endpoint CLI
gcloud ai endpoints predict <ENDPOINT-ID> \
--region=us-central1 \
--json-request=sample_payload.json
Azure Patterns
Azure AI Foundry / Azure Machine Learning Workspace orchestrates the entire ML lifecycle. Workspace, computes, environments, data assets, training jobs, models, and endpoints can be managed natively using the Azure CLI ml extension. For Azure ML code asset registration from CI, ADF orchestration, ADF WebActivity networking, result.version propagation, pointer blobs, or private storage firewall handling, load ml-azureml-adf-automation.
Azure ML v2 CLI Online Endpoint & Deployment YAML (azure_deploy.yaml)
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: credit-risk-endpoint
auth_mode: key
---
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: credit-risk-blue
endpoint_name: credit-risk-endpoint
model: azureml:credit-risk-model:1
code_configuration:
code: ./src
scoring_script: score.py
environment: azureml:credit-risk-env:1
instance_type: Standard_DS3_v2
instance_count: 2
request_settings:
request_timeout_ms: 3000
max_concurrent_requests_per_instance: 10
scale_settings:
type: default
Azure AI Foundry / Azure ML Workspace CLI Management (az ml)
# Add/update the Machine Learning CLI extension
az extension add -n ml -y
az login
# 1. Create a Resource Group and Azure ML Workspace (Azure AI Foundry Hub)
az group create --name my-ml-rg --location eastus
az ml workspace create --name my-ml-workspace --resource-group my-ml-rg --location eastus
# 2. Provision Computes (Instance, Training Cluster, CPU cluster)
az ml compute create --name dev-compute-ci --type ComputeInstance --size Standard_DS3_v2 -g my-ml-rg -w my-ml-workspace
az ml compute create --name gpu-train-cluster --type AmlCompute --size Standard_NC6s_v3 --min-instances 0 --max-instances 4 -g my-ml-rg -w my-ml-workspace
az ml compute create --name cpu-proc-cluster --type AmlCompute --size Standard_DS12_v2 --min-instances 0 --max-instances 8 -g my-ml-rg -w my-ml-workspace
# 3. Create a URI Data Asset
az ml data create --file data.yaml -g my-ml-rg -w my-ml-workspace
# 4. Create an Environment definition
az ml environment create --file env.yaml -g my-ml-rg -w my-ml-workspace
# 5. Submit, Monitor, and Stream a Command Job
az ml job create --file job.yaml -g my-ml-rg -w my-ml-workspace --web --stream
# View, list and monitor jobs
az ml job list -g my-ml-rg -w my-ml-workspace --output table
az ml job stream --name <job-id> -g my-ml-rg -w my-ml-workspace
az ml job download --name <job-id> --download-path ./outputs -g my-ml-rg -w my-ml-workspace
# 6. Register a Model
az ml model create --file model.yaml -g my-ml-rg -w my-ml-workspace
# 7. Create Endpoint, Deploy, Test, and Cleanup
az ml online-endpoint create --file endpoint.yaml -g my-ml-rg -w my-ml-workspace
az ml online-deployment create --file deployment.yaml --all-traffic -g my-ml-rg -w my-ml-workspace
# List and monitor endpoints:
az ml online-endpoint show --name credit-risk-endpoint -g my-ml-rg -w my-ml-workspace
# Test endpoint with a sample payload file:
az ml online-endpoint invoke --name credit-risk-endpoint --request-file ./sample_request.json -g my-ml-rg -w my-ml-workspace
# Delete endpoint:
az ml online-endpoint delete --name credit-risk-endpoint --yes -g my-ml-rg -w my-ml-workspace
Local & Hybrid Providers (MLflow & Kubernetes/KServe)
For local development or custom multi-cloud Kubernetes clusters, MLflow and KServe provide standardization.
MLflow Local CLI Management
# 1. Start a local tracking and model registry server
mlflow server \
--host 127.0.0.1 \
--port 5000 \
--backend-store-uri sqlite:///mlflow.db \
--default-artifact-root ./mlflow_artifacts_store
# 2. Run a packaged local MLflow Project with runtime parameters
mlflow run . -P alpha=0.1 -P l1_ratio=0.2 --experiment-name iris-training
# 3. Serve a registered model locally
mlflow models serve \
--model-uri models:/iris-classifier/Production \
--port 5001 \
--no-conda
# 4. Invoke the local serving endpoint
curl -X POST -H "Content-Type: application/json" \
-d '{"dataframe_split": {"columns": ["sepal_len", "sepal_wid"], "data": [[5.1, 3.5]]}}' \
http://127.0.0.1:5001/invocations
Kubernetes Native Serving CLI (KServe)
# kserve-inference.yaml
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "sklearn-iris"
namespace: "ml-serving"
spec:
predictor:
model:
modelFormat:
name: sklearn
storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: "500m"
memory: 1Gi
# 1. Apply inference manifests via kubectl
kubectl apply -f kserve-inference.yaml -n ml-serving
# 2. Monitor status and readiness of inference service
kubectl get inferenceservice sklearn-iris -n ml-serving
# 3. Port forward or direct payload query through Istio ingress gateway
SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n ml-serving -o jsonpath='{.status.url}' | cut -d/ -f3)
INGRESS_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
curl -v -H "Host: ${SERVICE_HOSTNAME}" \
http://${INGRESS_IP}:${INGRESS_PORT}/v1/models/sklearn-iris:predict \
-d @sample_payload.json
Serverless GPU serving (Modal Pattern)
For serverless deployments on isolated GPUs with zero-cold-start optimization, use Modal.
import modal
app = modal.App("serverless-image-classifier")
# Pre-packaged CUDA image with dependencies
image = (
modal.Image.debian_slim()
.pip_install("torch", "torchvision", "transformers")
)
@app.cls(gpu="A10G", image=image)
class ImageClassifier:
@modal.enter()
def load_model(self):
import torch
from transformers import ViTForImageClassification, ViTImageProcessor
self.device = "cuda" if torch.cuda.is_available() else "cpu"
self.processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224")
self.model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224").to(self.device)
@modal.method()
def predict(self, image_bytes: bytes):
from PIL import Image
import io
image = Image.open(io.BytesIO(image_bytes))
inputs = self.processor(images=image, return_tensors="pt").to(self.device)
outputs = self.model(**inputs)
logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item()
return self.model.config.id2label[predicted_class_idx]
Hardware Selection
Match hardware to bottleneck:
- CPU: classical ML, preprocessing, small models, latency-insensitive batch jobs.
- T4/L4/A10-class GPUs: cost-effective inference, fine-tuning smaller models, notebooks.
- A100/H100/H200/B200-class GPUs: large model training, high-throughput inference, memory-heavy workloads.
- TPUs: supported dense tensor workloads at scale, especially JAX/TensorFlow ecosystems.
- Inferentia/Trainium: AWS cost/performance if model operators and framework support are mature.
Estimate memory before launching: model weights, optimizer states, gradients, activations, KV cache, batch size, sequence length, and runtime overhead. For inference, benchmark p50/p95/p99 latency under realistic concurrency and payload sizes.
Endpoint Design
Define request schema, response schema, timeout, max payload, auth, rate limits, autoscaling policy, health checks, logging, and rollback. Use canary or blue/green releases for production. For GPU endpoints, tune dynamic batching and concurrency carefully; too much concurrency can increase tail latency or cause OOM. For multi-region deployments, replicate artifacts, keep model versions consistent, and plan data residency and failover.
Security and Compliance
Use private networking when models access sensitive data. Store secrets in cloud secret managers. Use least-privilege IAM/service accounts/managed identities. Encrypt data and artifacts. Log access and deployment events. Scan containers. Avoid baking credentials into images. For public endpoints, apply auth, input validation, abuse controls, rate limits, and prompt-injection defenses for LLM/RAG systems.
Cost Optimization
Use spot/preemptible instances for checkpointed training and stateless batch jobs. Use autoscaling endpoints with scale-to-zero when acceptable. Prefer batch inference for non-real-time workloads. Quantize or distill models before scaling replicas. Right-size GPU memory, not just GPU count. Use reserved/committed capacity only after workload shape is stable. Track cost per run, per model version, and per 1,000 predictions.
Deployment Readiness Checklist
- Model artifact, preprocessing, tokenizer, and config are versioned together.
- Input schema is validated and compatible with training features.
- Evaluation report includes relevant slices and latency/cost benchmarks.
- Endpoint has health checks, autoscaling, logs, metrics, alerts, and rollback.
- Security controls cover identity, secrets, network, encryption, and audit.
- Monitoring captures drift, data quality, prediction distribution, outcomes, and incidents.
Sources
1---
2name: ml-cloud-deployment
3description: |
4 This skill should be used when the user asks to deploy, scale, or cost-optimize ML workloads on cloud platforms. PROACTIVELY activate for: (1) AWS SageMaker Studio, Training, Processing, Pipelines, Endpoints, Model Monitor, Feature Store, Clarify, Ground Truth, EC2 GPUs, EKS, Lambda, Inferentia, Trainium, (2) GCP Vertex AI Training, Pipelines, Endpoints, Feature Store, Model Monitoring, AutoML, Matching Engine, TPU, GKE, Cloud Run, (3) Azure ML workspaces, pipelines, managed endpoints, AutoML, Responsible ML, AKS/ACI, (4) Databricks, Modal, Replicate, RunPod, Lambda Labs, Anyscale. Provides: cloud ML architecture, autoscaling, hardware, security, and cost guidance.
5metadata:
6 source: https://skillmd.com/skills/josiahsiegel/ml-cloud-deployment
7---
8
9# ML Cloud Deployment
10
11## Overview
12
13Use this skill for deploying ML workloads to managed platforms, Kubernetes, serverless systems, GPU/TPU providers, and lakehouse environments. Start from workload requirements: training or inference, batch or online, latency SLO, throughput, model size, data gravity, compliance, region, hardware, team expertise, and budget.
14
15## Platform Selection
16
17| Requirement | Strong choices |
18|---|---|
19| AWS-native managed lifecycle | SageMaker Studio, Training, Processing, Pipelines, Model Registry, Endpoints, Feature Store, Clarify, Model Monitor |
20| GCP-native managed lifecycle | Vertex AI Training, Pipelines, Endpoints, Feature Store, Model Monitoring, AutoML, Matching Engine, TPUs |
21| Azure-native managed lifecycle | Azure ML workspaces, compute clusters, pipelines, registries, managed online/batch endpoints, AutoML, Responsible ML |
22| Lakehouse/Spark-centric ML | Databricks on AWS/Azure/GCP with MLflow, Delta, Feature Store, Workflows |
23| Kubernetes control | EKS/GKE/AKS with Kubeflow, KServe, Seldon, Ray, Triton, custom operators |
24| Serverless or fast GPU apps | Modal, Replicate, Cloud Run with GPU where available, Lambda for small CPU inference |
25| Flexible GPU rental | Lambda Labs, RunPod, self-managed cloud GPU VMs |
26| Ray-native scale-out | Anyscale or Ray clusters on Kubernetes/cloud VMs |
27
28Prefer managed services when governance, observability, and team velocity matter more than runtime customization. Prefer Kubernetes or VMs when custom networking, specialized runtimes, or cost/performance tuning dominate.
29
30## AWS Patterns
31
32SageMaker handles managed training jobs, feature stores, and real-time or serverless endpoints. The AWS CLI (`aws sagemaker` and `aws sagemaker-runtime`) is used to orchestrate these systems programmatically.
33
34### AWS SageMaker Endpoint Variant Configuration (`endpoint_config.json`)
35```json
36{
37 "EndpointConfigName": "production-llm-classifier-v1-cfg",
38 "ProductionVariants": [
39 {
40 "VariantName": "AllTraffic",
41 "ModelName": "llm-classifier-model-v1",
42 "InitialInstanceCount": 2,
43 "InstanceType": "ml.g5.2xlarge",
44 "InitialVariantWeight": 1.0,
45 "VolumeSizeInGB": 50,
46 "ManagedInstanceScaling": {
47 "MinInstanceCount": 2,
48 "MaxInstanceCount": 10,
49 "Status": "ENABLED"
50 },
51 "RoutingConfig": {
52 "RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS"
53 }
54 }
55 ]
56}
57```
58
59### AWS CLI SageMaker Lifecycle Management
60
61```bash
62# 1. Setup Domain and default IAM Role/VPC settings
63aws sagemaker create-domain \
64 --domain-name ProdMLDomain \
65 --auth-mode IAM \
66 --default-user-settings ExecutionRole=arn:aws:iam::123456789012:role/SageMakerExecutionRole \
67 --subnet-ids subnet-1a2b3c,subnet-4d5e6f \
68 --vpc-id vpc-0abc123 \
69 --app-network-access-type PublicInternetOnly
70
71# 2. Provision and start/stop an interactive notebook instance
72aws sagemaker create-notebook-instance \
73 --notebook-instance-name dev-notebook-instance \
74 --instance-type ml.t3.medium \
75 --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole
76
77aws sagemaker start-notebook-instance --notebook-instance-name dev-notebook-instance
78aws sagemaker stop-notebook-instance --notebook-instance-name dev-notebook-instance
79
80# 3. Submit a managed GPU training job
81aws sagemaker create-training-job \
82 --training-job-name custom-pytorch-train-job \
83 --algorithm-specification TrainingImage=763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.12.0-gpu-py38-cu113-ubuntu20.04,TrainingInputMode=File \
84 --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
85 --input-data-config '[{"ChannelName": "training", "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": "s3://my-ml-bucket/train-data/", "S3DataDistributionType": "FullyReplicated"}}}]' \
86 --output-data-config S3OutputPath=s3://my-ml-bucket/checkpoints/ \
87 --resource-config InstanceType=ml.g5.2xlarge,InstanceCount=1,VolumeSizeInGB=50 \
88 --stopping-condition MaxRuntimeInSeconds=86400
89
90# 4. Register a model in the SageMaker Model Registry
91aws sagemaker create-model-package-group \
92 --model-package-group-name credit-scoring-group \
93 --model-package-group-description "Model registry group for credit scoring neural nets"
94
95aws sagemaker create-model \
96 --model-name credit-scoring-v1 \
97 --primary-container Image=763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.12.0-gpu-py38,ModelDataUrl=s3://my-ml-bucket/checkpoints/custom-pytorch-train-job/output/model.tar.gz \
98 --execution-role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole
99
100# 5. Create Endpoint Configuration and Deploy Endpoint
101aws sagemaker create-endpoint-config \
102 --endpoint-config-name credit-scoring-v1-cfg \
103 --production-variants '[{"VariantName": "AllTraffic", "ModelName": "credit-scoring-v1", "InitialInstanceCount": 2, "InstanceType": "ml.m5.xlarge", "InitialVariantWeight": 1.0}]'
104
105aws sagemaker create-endpoint \
106 --endpoint-name credit-scoring-prod-endpoint \
107 --endpoint-config-name credit-scoring-v1-cfg
108
109# 6. Invoke/Test Endpoint via CLI
110aws sagemaker-runtime invoke-endpoint \
111 --endpoint-name credit-scoring-prod-endpoint \
112 --content-type application/json \
113 --body '{"features": [0.25, 1.4, 0.9]}' \
114 response_output.json
115```
116
117## GCP Patterns
118
119Vertex AI manages training pipelines, hyperparameter tuning sweeps, custom jobs, the model registry, and predictions via `gcloud ai`.
120
121### GCP Vertex AI TPU CustomJob Specification (`tpu_job.yaml`)
122```yaml
123displayName: vertex-ai-tpu-training-job
124studySpec:
125 metrics:
126 - metricId: val_loss
127 goal: MINIMIZE
128 parameters:
129 - parameterId: learning_rate
130 doubleValueSpec:
131 minValue: 1e-5
132 maxValue: 1e-3
133 scaleType: UNIT_LOG_SCALE
134trialJobSpec:
135 workerPoolSpecs:
136 - machineSpec:
137 machineType: n1-standard-8
138 replicaCount: 1
139 containerSpec:
140 imageUri: gcr.io/my-project/ml-trainer:latest
141 - machineSpec:
142 machineType: cloud-tpu-v4-podslice
143 acceleratorType: TPU_V4
144 acceleratorCount: 4
145 replicaCount: 1
146 containerSpec:
147 imageUri: gcr.io/my-project/ml-tpu-trainer:latest
148 args: [
149 "--data_path", "gs://my-bucket/dataset-v1",
150 "--epochs", "10"
151 ]
152```
153
154### GCP Vertex AI CLI Operations
155
156```bash
157# 1. Create a user-managed Jupyter Workbench instance
158gcloud notebooks instances create dev-workbench-instance \
159 --location=us-central1-a \
160 --vm-image-project=deeplearning-platform-release \
161 --vm-image-family=common-gpu \
162 --machine-type=n1-standard-4
163
164# 2. Submit containerized custom training job
165gcloud ai custom-jobs create \
166 --region=us-central1 \
167 --display-name=gpu-pytorch-training-run \
168 --worker-pool-spec=replica-count=1,machine-type=n1-standard-8,accelerator-type=nvidia-tesla-t4,accelerator-count=1,container-image-uri=gcr.io/my-gcp-project/pytorch-train-custom:v1 \
169 --args="--epochs=20,--data-path=gs://my-bucket/training-gold"
170
171# 3. Upload model artifact to Model Registry
172gcloud ai models upload \
173 --region=us-central1 \
174 --display-name=iris-classifier-v1 \
175 --container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/pytorch-cpu.1-11:latest \
176 --artifact-uri=gs://my-bucket/models/iris-classifier/
177
178# 4. Provision endpoint and deploy model with 100% traffic allocation
179# Note: replace <ENDPOINT-ID> and <MODEL-ID> with their UUIDs
180gcloud ai endpoints create \
181 --region=us-central1 \
182 --display-name=production-iris-endpoint
183
184gcloud ai endpoints deploy-model <ENDPOINT-ID> \
185 --region=us-central1 \
186 --model=<MODEL-ID> \
187 --display-name=iris-v1-blue-deployment \
188 --machine-type=n1-standard-4 \
189 --min-replica-count=2 \
190 --max-replica-count=10 \
191 --traffic-split=0=100
192
193# 5. Predict via Vertex AI Endpoint CLI
194gcloud ai endpoints predict <ENDPOINT-ID> \
195 --region=us-central1 \
196 --json-request=sample_payload.json
197```
198
199## Azure Patterns
200
201Azure AI Foundry / Azure Machine Learning Workspace orchestrates the entire ML lifecycle. Workspace, computes, environments, data assets, training jobs, models, and endpoints can be managed natively using the Azure CLI `ml` extension. For Azure ML code asset registration from CI, ADF orchestration, ADF WebActivity networking, `result.version` propagation, pointer blobs, or private storage firewall handling, load `ml-azureml-adf-automation`.
202
203### Azure ML v2 CLI Online Endpoint & Deployment YAML (`azure_deploy.yaml`)
204```yaml
205$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
206name: credit-risk-endpoint
207auth_mode: key
208---
209$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
210name: credit-risk-blue
211endpoint_name: credit-risk-endpoint
212model: azureml:credit-risk-model:1
213code_configuration:
214 code: ./src
215 scoring_script: score.py
216environment: azureml:credit-risk-env:1
217instance_type: Standard_DS3_v2
218instance_count: 2
219request_settings:
220 request_timeout_ms: 3000
221 max_concurrent_requests_per_instance: 10
222scale_settings:
223 type: default
224```
225
226### Azure AI Foundry / Azure ML Workspace CLI Management (`az ml`)
227
228```bash
229# Add/update the Machine Learning CLI extension
230az extension add -n ml -y
231az login
232
233# 1. Create a Resource Group and Azure ML Workspace (Azure AI Foundry Hub)
234az group create --name my-ml-rg --location eastus
235az ml workspace create --name my-ml-workspace --resource-group my-ml-rg --location eastus
236
237# 2. Provision Computes (Instance, Training Cluster, CPU cluster)
238az ml compute create --name dev-compute-ci --type ComputeInstance --size Standard_DS3_v2 -g my-ml-rg -w my-ml-workspace
239az ml compute create --name gpu-train-cluster --type AmlCompute --size Standard_NC6s_v3 --min-instances 0 --max-instances 4 -g my-ml-rg -w my-ml-workspace
240az ml compute create --name cpu-proc-cluster --type AmlCompute --size Standard_DS12_v2 --min-instances 0 --max-instances 8 -g my-ml-rg -w my-ml-workspace
241
242# 3. Create a URI Data Asset
243az ml data create --file data.yaml -g my-ml-rg -w my-ml-workspace
244
245# 4. Create an Environment definition
246az ml environment create --file env.yaml -g my-ml-rg -w my-ml-workspace
247
248# 5. Submit, Monitor, and Stream a Command Job
249az ml job create --file job.yaml -g my-ml-rg -w my-ml-workspace --web --stream
250
251# View, list and monitor jobs
252az ml job list -g my-ml-rg -w my-ml-workspace --output table
253az ml job stream --name <job-id> -g my-ml-rg -w my-ml-workspace
254az ml job download --name <job-id> --download-path ./outputs -g my-ml-rg -w my-ml-workspace
255
256# 6. Register a Model
257az ml model create --file model.yaml -g my-ml-rg -w my-ml-workspace
258
259# 7. Create Endpoint, Deploy, Test, and Cleanup
260az ml online-endpoint create --file endpoint.yaml -g my-ml-rg -w my-ml-workspace
261az ml online-deployment create --file deployment.yaml --all-traffic -g my-ml-rg -w my-ml-workspace
262
263# List and monitor endpoints:
264az ml online-endpoint show --name credit-risk-endpoint -g my-ml-rg -w my-ml-workspace
265
266# Test endpoint with a sample payload file:
267az ml online-endpoint invoke --name credit-risk-endpoint --request-file ./sample_request.json -g my-ml-rg -w my-ml-workspace
268
269# Delete endpoint:
270az ml online-endpoint delete --name credit-risk-endpoint --yes -g my-ml-rg -w my-ml-workspace
271```
272
273## Local & Hybrid Providers (MLflow & Kubernetes/KServe)
274
275For local development or custom multi-cloud Kubernetes clusters, MLflow and KServe provide standardization.
276
277### MLflow Local CLI Management
278
279```bash
280# 1. Start a local tracking and model registry server
281mlflow server \
282 --host 127.0.0.1 \
283 --port 5000 \
284 --backend-store-uri sqlite:///mlflow.db \
285 --default-artifact-root ./mlflow_artifacts_store
286
287# 2. Run a packaged local MLflow Project with runtime parameters
288mlflow run . -P alpha=0.1 -P l1_ratio=0.2 --experiment-name iris-training
289
290# 3. Serve a registered model locally
291mlflow models serve \
292 --model-uri models:/iris-classifier/Production \
293 --port 5001 \
294 --no-conda
295
296# 4. Invoke the local serving endpoint
297curl -X POST -H "Content-Type: application/json" \
298 -d '{"dataframe_split": {"columns": ["sepal_len", "sepal_wid"], "data": [[5.1, 3.5]]}}' \
299 http://127.0.0.1:5001/invocations
300```
301
302### Kubernetes Native Serving CLI (KServe)
303
304```yaml
305# kserve-inference.yaml
306apiVersion: "serving.kserve.io/v1beta1"
307kind: "InferenceService"
308metadata:
309 name: "sklearn-iris"
310 namespace: "ml-serving"
311spec:
312 predictor:
313 model:
314 modelFormat:
315 name: sklearn
316 storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
317 resources:
318 limits:
319 cpu: "1"
320 memory: 2Gi
321 requests:
322 cpu: "500m"
323 memory: 1Gi
324```
325
326```bash
327# 1. Apply inference manifests via kubectl
328kubectl apply -f kserve-inference.yaml -n ml-serving
329
330# 2. Monitor status and readiness of inference service
331kubectl get inferenceservice sklearn-iris -n ml-serving
332
333# 3. Port forward or direct payload query through Istio ingress gateway
334SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n ml-serving -o jsonpath='{.status.url}' | cut -d/ -f3)
335INGRESS_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
336INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
337
338curl -v -H "Host: ${SERVICE_HOSTNAME}" \
339 http://${INGRESS_IP}:${INGRESS_PORT}/v1/models/sklearn-iris:predict \
340 -d @sample_payload.json
341```
342
343
344## Serverless GPU serving (Modal Pattern)
345
346For serverless deployments on isolated GPUs with zero-cold-start optimization, use Modal.
347
348```python
349import modal
350
351app = modal.App("serverless-image-classifier")
352
353# Pre-packaged CUDA image with dependencies
354image = (
355 modal.Image.debian_slim()
356 .pip_install("torch", "torchvision", "transformers")
357)
358
359@app.cls(gpu="A10G", image=image)
360class ImageClassifier:
361 @modal.enter()
362 def load_model(self):
363 import torch
364 from transformers import ViTForImageClassification, ViTImageProcessor
365 self.device = "cuda" if torch.cuda.is_available() else "cpu"
366 self.processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224")
367 self.model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224").to(self.device)
368
369 @modal.method()
370 def predict(self, image_bytes: bytes):
371 from PIL import Image
372 import io
373 image = Image.open(io.BytesIO(image_bytes))
374 inputs = self.processor(images=image, return_tensors="pt").to(self.device)
375 outputs = self.model(**inputs)
376 logits = outputs.logits
377 predicted_class_idx = logits.argmax(-1).item()
378 return self.model.config.id2label[predicted_class_idx]
379```
380
381## Hardware Selection
382
383Match hardware to bottleneck:
384- CPU: classical ML, preprocessing, small models, latency-insensitive batch jobs.
385- T4/L4/A10-class GPUs: cost-effective inference, fine-tuning smaller models, notebooks.
386- A100/H100/H200/B200-class GPUs: large model training, high-throughput inference, memory-heavy workloads.
387- TPUs: supported dense tensor workloads at scale, especially JAX/TensorFlow ecosystems.
388- Inferentia/Trainium: AWS cost/performance if model operators and framework support are mature.
389
390Estimate memory before launching: model weights, optimizer states, gradients, activations, KV cache, batch size, sequence length, and runtime overhead. For inference, benchmark p50/p95/p99 latency under realistic concurrency and payload sizes.
391
392## Endpoint Design
393
394Define request schema, response schema, timeout, max payload, auth, rate limits, autoscaling policy, health checks, logging, and rollback. Use canary or blue/green releases for production. For GPU endpoints, tune dynamic batching and concurrency carefully; too much concurrency can increase tail latency or cause OOM. For multi-region deployments, replicate artifacts, keep model versions consistent, and plan data residency and failover.
395
396## Security and Compliance
397
398Use private networking when models access sensitive data. Store secrets in cloud secret managers. Use least-privilege IAM/service accounts/managed identities. Encrypt data and artifacts. Log access and deployment events. Scan containers. Avoid baking credentials into images. For public endpoints, apply auth, input validation, abuse controls, rate limits, and prompt-injection defenses for LLM/RAG systems.
399
400## Cost Optimization
401
402Use spot/preemptible instances for checkpointed training and stateless batch jobs. Use autoscaling endpoints with scale-to-zero when acceptable. Prefer batch inference for non-real-time workloads. Quantize or distill models before scaling replicas. Right-size GPU memory, not just GPU count. Use reserved/committed capacity only after workload shape is stable. Track cost per run, per model version, and per 1,000 predictions.
403
404## Deployment Readiness Checklist
405
406- Model artifact, preprocessing, tokenizer, and config are versioned together.
407- Input schema is validated and compatible with training features.
408- Evaluation report includes relevant slices and latency/cost benchmarks.
409- Endpoint has health checks, autoscaling, logs, metrics, alerts, and rollback.
410- Security controls cover identity, secrets, network, encryption, and audit.
411- Monitoring captures drift, data quality, prediction distribution, outcomes, and incidents.
412
413## Sources
414
415- AWS SageMaker documentation: https://docs.aws.amazon.com/sagemaker/
416- Google Vertex AI documentation: https://cloud.google.com/vertex-ai/docs
417- Azure Machine Learning documentation: https://learn.microsoft.com/azure/machine-learning/
418- Databricks Machine Learning documentation: https://docs.databricks.com/machine-learning/
419- Ray and Anyscale documentation: https://docs.ray.io/ and https://docs.anyscale.com/